docs(plan): add Phase 1, 2, 3 implementation specs

2026-07-05 08:51:38 -04:00
parent f540cbe7ef
commit c9dd9ba076
12 changed files with 3041 additions and 0 deletions
@@ -0,0 +1,60 @@
+# Phase 1 — Reliability & Completeness Improvements
+
+## Scope
+
+This phase addresses the **three highest-impact reliability gaps** identified in the code review: incomplete ad coverage due to page-limited scraping, inefficient HTTP client usage, and silent notification loss. After this phase, the worker will:
+
+- Capture a larger window of ads per scrape cycle (no longer limited to 30 newest)
+- Reuse TCP/TLS connections for willhaben API calls instead of creating one per request
+- Retry failed Telegram notifications instead of dropping them permanently
+
+## Architecture
+
+```
+┌──────────────────────────────────────────────────────┐
+│  worker container                                     │
+│                                                      │
+│  ┌───────────┐                                       │
+│  │ scraper.py│  ← SINGLETON AsyncClient              │
+│  │           │    (connection pool, keepalive)        │
+│  │           │                                       │
+│  │ fetch_ads()                                        │
+│  │ ├─ page 1: rows=30 & published_after=<cursor>     │
+│  │ ├─ page 2: rows=30 & offset=30                    │
+│  │ └─ ... until no new ads or max_pages reached      │
+│  └───────────┘                                       │
+│       │                                               │
+│       ▼                                               │
+│  ┌──────────────┐   ┌──────────────────┐             │
+│  │ notifier.py  │──►│ notification_queue│            │
+│  │              │   │ table (new)       │            │
+│  │ notify_new() │   │                   │            │
+│  │ notify_drop()│   │ - ad_id           │            │
+│  │              │   │ - telegram_id     │            │
+│  │ if success:  │   │ - attempts (0→5) │            │
+│  │   log_notify │   │ - last_error      │            │
+│  │ if fail:     │   │ - status          │            │
+│  │   enqueue!   │   └────────┬─────────┘             │
+│  └──────────────┘           │                        │
+│                             ▼                        │
+│                   scheduler retries                  │
+│                   pending items each cycle            │
+└──────────────────────────────────────────────────────┘
+```
+
+## Tasks
+
+| Task | File | Description |
+|------|------|-------------|
+| Pagination in willhaben scraper | [task-scraper-pagination.md](./task-scraper-pagination.md) | Implement cursor-based or offset pagination to fetch more than 30 ads per cycle, tracking the last seen timestamp to avoid duplicates across cycles. |
+| httpx singleton with connection pool | [task-httpx-singleton.md](./task-httpx-singleton.md) | Replace per-call AsyncClient creation with a module-level singleton using keepalive connections and configurable limits. |
+| Retry queue for failed notifications | [task-notification-retry-queue.md](./task-notification-retry-queue.md) | Add a `notification_queue` table to persist failed Telegram sends with exponential backoff retries (up to 5 attempts). |
+
+## General Acceptance Criteria
+
+- [ ] A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30
+- [ ] Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
+- [ ] HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
+- [ ] Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
+- [ ] After 5 failed retries the notification is marked as `dead` and logged — not silently dropped
+- [ ] The scheduler processes queued notifications at the start of each cycle before scraping new keywords