Lago/willhaben-tracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Phase 1 — reliability & completeness improvements

Browse Source 
- httpx singleton: replace per-call AsyncClient with module-level singleton
  using keepalive pool (configurable via env vars). Close client on shutdown.

- Scraper pagination: multi-page fetching (default 2 pages/60 ads, configurable
  via SCRAPE_MAX_PAGES). Cursor-based early-stop to skip stale pages. Tracks
  ads_cursor per keyword for cross-cycle deduplication.

- Notification retry queue: persistent notification_queue table with exponential
  backoff (max 5 attempts → dead). Failed Telegram notifications are enqueued
  instead of silently dropped. Queue processed at start of each scheduler cycle
  (50 items max limit). /stats endpoint reports pending/dead counts.
This commit is contained in:
Lago
2026-07-05 22:22:28 +02:00
parent c9dd9ba076
commit 3e7e5d0b32
10 changed files with 347 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-1/task-scraper-pagination.md
+7 -7
View File
@@ -180,10 +180,10 @@ COMMENT ON COLUMN keywords.ads_cursor IS
## Acceptance Criteria
- [ ] A scrape cycle fetches at least 2 pages (60 ads) by default for active keywords
- [ ] The `max_pages` limit is configurable via `SCRAPE_MAX_PAGES` environment variable
- [ ] Early stop detection works: if page N has no newer ads than the cursor, pagination stops without fetching remaining pages
- [ ] Ads are not re-notified across cycles (cursor prevents duplicates)
- [ ] A 1-second delay between page fetches is logged and respected
- [ ] `total_hits` from willhaben API is still returned for logging/stats purposes
- [ ] No regression in single-page behavior when max_pages=1
- [x] A scrape cycle fetches at least 2 pages (60 ads) by default for active keywords
- [x] The `max_pages` limit is configurable via `SCRAPE_MAX_PAGES` environment variable
- [x] Early stop detection works: if page N has no newer ads than the cursor, pagination stops without fetching remaining pages
- [x] Ads are not re-notified across cycles (cursor prevents duplicates)
- [x] A 1-second delay between page fetches is logged and respected
- [x] `total_hits` from willhaben API is still returned for logging/stats purposes
- [x] No regression in single-page behavior when max_pages=1