feat: Phase 1 — reliability & completeness improvements
- httpx singleton: replace per-call AsyncClient with module-level singleton using keepalive pool (configurable via env vars). Close client on shutdown. - Scraper pagination: multi-page fetching (default 2 pages/60 ads, configurable via SCRAPE_MAX_PAGES). Cursor-based early-stop to skip stale pages. Tracks ads_cursor per keyword for cross-cycle deduplication. - Notification retry queue: persistent notification_queue table with exponential backoff (max 5 attempts → dead). Failed Telegram notifications are enqueued instead of silently dropped. Queue processed at start of each scheduler cycle (50 items max limit). /stats endpoint reports pending/dead counts.
This commit is contained in:
@@ -292,10 +292,10 @@ async def run_scheduler() -> None:
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] When `_send_message()` raises `TelegramError`, the notification is INSERTed into `notification_queue` with status='pending'
|
||||
- [ ] On the next scheduler cycle, pending items are attempted (respecting backoff)
|
||||
- [ ] After 5 failed attempts, the notification status becomes 'dead' and a warning is logged
|
||||
- [ ] The queue processes at most 50 items per cycle to avoid blocking the scheduler
|
||||
- [ ] Duplicate enqueue prevention works: calling `_enqueue_retry` twice for the same ad+user creates only one queue entry
|
||||
- [ ] Successful retries update `log_notifications` table (same as direct notifications)
|
||||
- [ ] The `/health` endpoint or logs can show the current count of pending/dead items
|
||||
- [x] When `_send_message()` raises `TelegramError`, the notification is INSERTed into `notification_queue` with status='pending'
|
||||
- [x] On the next scheduler cycle, pending items are attempted (respecting backoff)
|
||||
- [x] After 5 failed attempts, the notification status becomes 'dead' and a warning is logged
|
||||
- [x] The queue processes at most 50 items per cycle to avoid blocking the scheduler
|
||||
- [x] Duplicate enqueue prevention works: calling `_enqueue_retry` twice for the same ad+user creates only one queue entry
|
||||
- [x] Successful retries update `log_notifications` table (same as direct notifications)
|
||||
- [x] The `/health` endpoint or logs can show the current count of pending/dead items
|
||||
|
||||
Reference in New Issue
Block a user