# Phase 1 — Reliability & Completeness Improvements

## Scope

This phase addresses the **three highest-impact reliability gaps** identified in the code review: incomplete ad coverage due to page-limited scraping, inefficient HTTP client usage, and silent notification loss. After this phase, the worker will:

- Capture a larger window of ads per scrape cycle (no longer limited to 30 newest)
- Reuse TCP/TLS connections for willhaben API calls instead of creating one per request
- Retry failed Telegram notifications instead of dropping them permanently

## Architecture

```
┌──────────────────────────────────────────────────────┐
│  worker container                                     │
│                                                      │
│  ┌───────────┐                                       │
│  │ scraper.py│  ← SINGLETON AsyncClient              │
│  │           │    (connection pool, keepalive)        │
│  │           │                                       │
│  │ fetch_ads()                                        │
│  │ ├─ page 1: rows=30 & published_after=<cursor>     │
│  │ ├─ page 2: rows=30 & offset=30                    │
│  │ └─ ... until no new ads or max_pages reached      │
│  └───────────┘                                       │
│       │                                               │
│       ▼                                               │
│  ┌──────────────┐   ┌──────────────────┐             │
│  │ notifier.py  │──►│ notification_queue│            │
│  │              │   │ table (new)       │            │
│  │ notify_new() │   │                   │            │
│  │ notify_drop()│   │ - ad_id           │            │
│  │              │   │ - telegram_id     │            │
│  │ if success:  │   │ - attempts (0→5) │            │
│  │   log_notify │   │ - last_error      │            │
│  │ if fail:     │   │ - status          │            │
│  │   enqueue!   │   └────────┬─────────┘             │
│  └──────────────┘           │                        │
│                             ▼                        │
│                   scheduler retries                  │
│                   pending items each cycle            │
└──────────────────────────────────────────────────────┘
```

## Tasks

| Task | File | Description |
|------|------|-------------|
| Pagination in willhaben scraper | [task-scraper-pagination.md](./task-scraper-pagination.md) | Implement cursor-based or offset pagination to fetch more than 30 ads per cycle, tracking the last seen timestamp to avoid duplicates across cycles. |
| httpx singleton with connection pool | [task-httpx-singleton.md](./task-httpx-singleton.md) | Replace per-call AsyncClient creation with a module-level singleton using keepalive connections and configurable limits. |
| Retry queue for failed notifications | [task-notification-retry-queue.md](./task-notification-retry-queue.md) | Add a `notification_queue` table to persist failed Telegram sends with exponential backoff retries (up to 5 attempts). |

## General Acceptance Criteria

- [ ] A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30
- [ ] Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
- [ ] HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
- [ ] Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
- [ ] After 5 failed retries the notification is marked as `dead` and logged — not silently dropped
- [ ] The scheduler processes queued notifications at the start of each cycle before scraping new keywords