willhaben-tracker/docs/phase-1/Phase.md

Phase 1 — Reliability & Completeness Improvements

Scope

This phase addresses the three highest-impact reliability gaps identified in the code review: incomplete ad coverage due to page-limited scraping, inefficient HTTP client usage, and silent notification loss. After this phase, the worker will:

• Capture a larger window of ads per scrape cycle (no longer limited to 30 newest)
• Reuse TCP/TLS connections for willhaben API calls instead of creating one per request
• Retry failed Telegram notifications instead of dropping them permanently

Architecture

┌──────────────────────────────────────────────────────┐
│  worker container                                     │
│                                                      │
│  ┌───────────┐                                       │
│  │ scraper.py│  ← SINGLETON AsyncClient              │
│  │           │    (connection pool, keepalive)        │
│  │           │                                       │
│  │ fetch_ads()                                        │
│  │ ├─ page 1: rows=30 & published_after=<cursor>     │
│  │ ├─ page 2: rows=30 & offset=30                    │
│  │ └─ ... until no new ads or max_pages reached      │
│  └───────────┘                                       │
│       │                                               │
│       ▼                                               │
│  ┌──────────────┐   ┌──────────────────┐             │
│  │ notifier.py  │──►│ notification_queue│            │
│  │              │   │ table (new)       │            │
│  │ notify_new() │   │                   │            │
│  │ notify_drop()│   │ - ad_id           │            │
│  │              │   │ - telegram_id     │            │
│  │ if success:  │   │ - attempts (0→5) │            │
│  │   log_notify │   │ - last_error      │            │
│  │ if fail:     │   │ - status          │            │
│  │   enqueue!   │   └────────┬─────────┘             │
│  └──────────────┘           │                        │
│                             ▼                        │
│                   scheduler retries                  │
│                   pending items each cycle            │
└──────────────────────────────────────────────────────┘

Tasks

Task	File	Description
Pagination in willhaben scraper	task-scraper-pagination.md	Implement cursor-based or offset pagination to fetch more than 30 ads per cycle, tracking the last seen timestamp to avoid duplicates across cycles.
httpx singleton with connection pool	task-httpx-singleton.md	Replace per-call AsyncClient creation with a module-level singleton using keepalive connections and configurable limits.
Retry queue for failed notifications	task-notification-retry-queue.md	Add a notification_queue table to persist failed Telegram sends with exponential backoff retries (up to 5 attempts).

General Acceptance Criteria

• A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30
• Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
• HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
• Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
• After 5 failed retries the notification is marked as dead and logged — not silently dropped
• The scheduler processes queued notifications at the start of each cycle before scraping new keywords