Lago
3e7e5d0b32
- httpx singleton: replace per-call AsyncClient with module-level singleton using keepalive pool (configurable via env vars). Close client on shutdown. - Scraper pagination: multi-page fetching (default 2 pages/60 ads, configurable via SCRAPE_MAX_PAGES). Cursor-based early-stop to skip stale pages. Tracks ads_cursor per keyword for cross-cycle deduplication. - Notification retry queue: persistent notification_queue table with exponential backoff (max 5 attempts → dead). Failed Telegram notifications are enqueued instead of silently dropped. Queue processed at start of each scheduler cycle (50 items max limit). /stats endpoint reports pending/dead counts.
4.3 KiB
4.3 KiB
Phase 1 — Reliability & Completeness Improvements
Scope
This phase addresses the three highest-impact reliability gaps identified in the code review: incomplete ad coverage due to page-limited scraping, inefficient HTTP client usage, and silent notification loss. After this phase, the worker will:
- Capture a larger window of ads per scrape cycle (no longer limited to 30 newest)
- Reuse TCP/TLS connections for willhaben API calls instead of creating one per request
- Retry failed Telegram notifications instead of dropping them permanently
Architecture
┌──────────────────────────────────────────────────────┐
│ worker container │
│ │
│ ┌───────────┐ │
│ │ scraper.py│ ← SINGLETON AsyncClient │
│ │ │ (connection pool, keepalive) │
│ │ │ │
│ │ fetch_ads() │
│ │ ├─ page 1: rows=30 & published_after=<cursor> │
│ │ ├─ page 2: rows=30 & offset=30 │
│ │ └─ ... until no new ads or max_pages reached │
│ └───────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ ┌──────────────────┐ │
│ │ notifier.py │──►│ notification_queue│ │
│ │ │ │ table (new) │ │
│ │ notify_new() │ │ │ │
│ │ notify_drop()│ │ - ad_id │ │
│ │ │ │ - telegram_id │ │
│ │ if success: │ │ - attempts (0→5) │ │
│ │ log_notify │ │ - last_error │ │
│ │ if fail: │ │ - status │ │
│ │ enqueue! │ └────────┬─────────┘ │
│ └──────────────┘ │ │
│ ▼ │
│ scheduler retries │
│ pending items each cycle │
└──────────────────────────────────────────────────────┘
Tasks
| Task | File | Description |
|---|---|---|
| Pagination in willhaben scraper | task-scraper-pagination.md | Implement cursor-based or offset pagination to fetch more than 30 ads per cycle, tracking the last seen timestamp to avoid duplicates across cycles. |
| httpx singleton with connection pool | task-httpx-singleton.md | Replace per-call AsyncClient creation with a module-level singleton using keepalive connections and configurable limits. |
| Retry queue for failed notifications | task-notification-retry-queue.md | Add a notification_queue table to persist failed Telegram sends with exponential backoff retries (up to 5 attempts). |
General Acceptance Criteria
- A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30 — configurable via
SCRAPE_MAX_PAGES, default is 2 pages (60 ads) - Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
- HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
- Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
- After 5 failed retries the notification is marked as
deadand logged — not silently dropped - The scheduler processes queued notifications at the start of each cycle before scraping new keywords