Lago/willhaben-tracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Phase 1 — reliability & completeness improvements

Browse Source 
- httpx singleton: replace per-call AsyncClient with module-level singleton
  using keepalive pool (configurable via env vars). Close client on shutdown.

- Scraper pagination: multi-page fetching (default 2 pages/60 ads, configurable
  via SCRAPE_MAX_PAGES). Cursor-based early-stop to skip stale pages. Tracks
  ads_cursor per keyword for cross-cycle deduplication.

- Notification retry queue: persistent notification_queue table with exponential
  backoff (max 5 attempts → dead). Failed Telegram notifications are enqueued
  instead of silently dropped. Queue processed at start of each scheduler cycle
  (50 items max limit). /stats endpoint reports pending/dead counts.
This commit is contained in:
Lago
2026-07-05 22:22:28 +02:00
parent c9dd9ba076
commit 3e7e5d0b32
10 changed files with 347 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-1/Phase.md
+6 -6
View File
@@ -52,9 +52,9 @@ This phase addresses the **three highest-impact reliability gaps** identified in
## General Acceptance Criteria
- [ ] A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30
- [ ] Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
- [ ] HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
- [ ] Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
- [ ] After 5 failed retries the notification is marked as `dead` and logged — not silently dropped
- [ ] The scheduler processes queued notifications at the start of each cycle before scraping new keywords
- [x] A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30 — *configurable via `SCRAPE_MAX_PAGES`, default is 2 pages (60 ads)*
- [x] Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
- [x] HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
- [x] Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
- [x] After 5 failed retries the notification is marked as `dead` and logged — not silently dropped
- [x] The scheduler processes queued notifications at the start of each cycle before scraping new keywords