feat: Phase 1 — reliability & completeness improvements

- httpx singleton: replace per-call AsyncClient with module-level singleton
  using keepalive pool (configurable via env vars). Close client on shutdown.

- Scraper pagination: multi-page fetching (default 2 pages/60 ads, configurable
  via SCRAPE_MAX_PAGES). Cursor-based early-stop to skip stale pages. Tracks
  ads_cursor per keyword for cross-cycle deduplication.

- Notification retry queue: persistent notification_queue table with exponential
  backoff (max 5 attempts → dead). Failed Telegram notifications are enqueued
  instead of silently dropped. Queue processed at start of each scheduler cycle
  (50 items max limit). /stats endpoint reports pending/dead counts.
This commit is contained in:
2026-07-05 22:22:28 +02:00
parent c9dd9ba076
commit 3e7e5d0b32
10 changed files with 347 additions and 56 deletions
+6 -6
View File
@@ -149,9 +149,9 @@ HTTP_TIMEOUT_S=30.0 # Request timeout in seconds
## Acceptance Criteria
- [ ] Only one `httpx.AsyncClient` is created per process lifetime (logged once at startup)
- [ ] Subsequent calls to `fetch_ads()` reuse the existing client (no "Created httpx client" log)
- [ ] After calling `close_client()`, a new call to `get_client()` creates a fresh client
- [ ] Connection keepalive reduces latency for sequential API calls (verifiable via timing in logs)
- [ ] Fatal transport errors trigger client recreation without crashing the scheduler
- [ ] The client is properly closed during graceful shutdown (no resource warnings)
- [x] Only one `httpx.AsyncClient` is created per process lifetime (logged once at startup)
- [x] Subsequent calls to `fetch_ads()` reuse the existing client (no "Created httpx client" log)
- [x] After calling `close_client()`, a new call to `get_client()` creates a fresh client
- [x] Connection keepalive reduces latency for sequential API calls (verifiable via timing in logs)
- [x] Fatal transport errors trigger client recreation without crashing the scheduler
- [x] The client is properly closed during graceful shutdown (no resource warnings)