Lago
3e7e5d0b32
- httpx singleton: replace per-call AsyncClient with module-level singleton using keepalive pool (configurable via env vars). Close client on shutdown. - Scraper pagination: multi-page fetching (default 2 pages/60 ads, configurable via SCRAPE_MAX_PAGES). Cursor-based early-stop to skip stale pages. Tracks ads_cursor per keyword for cross-cycle deduplication. - Notification retry queue: persistent notification_queue table with exponential backoff (max 5 attempts → dead). Failed Telegram notifications are enqueued instead of silently dropped. Queue processed at start of each scheduler cycle (50 items max limit). /stats endpoint reports pending/dead counts.
61 lines
4.3 KiB
Markdown
61 lines
4.3 KiB
Markdown
# Phase 1 — Reliability & Completeness Improvements
|
||
|
||
## Scope
|
||
|
||
This phase addresses the **three highest-impact reliability gaps** identified in the code review: incomplete ad coverage due to page-limited scraping, inefficient HTTP client usage, and silent notification loss. After this phase, the worker will:
|
||
|
||
- Capture a larger window of ads per scrape cycle (no longer limited to 30 newest)
|
||
- Reuse TCP/TLS connections for willhaben API calls instead of creating one per request
|
||
- Retry failed Telegram notifications instead of dropping them permanently
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────┐
|
||
│ worker container │
|
||
│ │
|
||
│ ┌───────────┐ │
|
||
│ │ scraper.py│ ← SINGLETON AsyncClient │
|
||
│ │ │ (connection pool, keepalive) │
|
||
│ │ │ │
|
||
│ │ fetch_ads() │
|
||
│ │ ├─ page 1: rows=30 & published_after=<cursor> │
|
||
│ │ ├─ page 2: rows=30 & offset=30 │
|
||
│ │ └─ ... until no new ads or max_pages reached │
|
||
│ └───────────┘ │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌──────────────┐ ┌──────────────────┐ │
|
||
│ │ notifier.py │──►│ notification_queue│ │
|
||
│ │ │ │ table (new) │ │
|
||
│ │ notify_new() │ │ │ │
|
||
│ │ notify_drop()│ │ - ad_id │ │
|
||
│ │ │ │ - telegram_id │ │
|
||
│ │ if success: │ │ - attempts (0→5) │ │
|
||
│ │ log_notify │ │ - last_error │ │
|
||
│ │ if fail: │ │ - status │ │
|
||
│ │ enqueue! │ └────────┬─────────┘ │
|
||
│ └──────────────┘ │ │
|
||
│ ▼ │
|
||
│ scheduler retries │
|
||
│ pending items each cycle │
|
||
└──────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Tasks
|
||
|
||
| Task | File | Description |
|
||
|------|------|-------------|
|
||
| Pagination in willhaben scraper | [task-scraper-pagination.md](./task-scraper-pagination.md) | Implement cursor-based or offset pagination to fetch more than 30 ads per cycle, tracking the last seen timestamp to avoid duplicates across cycles. |
|
||
| httpx singleton with connection pool | [task-httpx-singleton.md](./task-httpx-singleton.md) | Replace per-call AsyncClient creation with a module-level singleton using keepalive connections and configurable limits. |
|
||
| Retry queue for failed notifications | [task-notification-retry-queue.md](./task-notification-retry-queue.md) | Add a `notification_queue` table to persist failed Telegram sends with exponential backoff retries (up to 5 attempts). |
|
||
|
||
## General Acceptance Criteria
|
||
|
||
- [x] A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30 — *configurable via `SCRAPE_MAX_PAGES`, default is 2 pages (60 ads)*
|
||
- [x] Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
|
||
- [x] HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
|
||
- [x] Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
|
||
- [x] After 5 failed retries the notification is marked as `dead` and logged — not silently dropped
|
||
- [x] The scheduler processes queued notifications at the start of each cycle before scraping new keywords
|