Lago/willhaben-tracker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(plan): add Phase 1, 2, 3 implementation specs

Browse Source
This commit is contained in:
hermes
2026-07-05 08:51:38 -04:00
parent f540cbe7ef
commit c9dd9ba076
12 changed files with 3041 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-1/Phase.md
+60
View File
@@ -0,0 +1,60 @@
# Phase 1 — Reliability & Completeness Improvements
## Scope
This phase addresses the **three highest-impact reliability gaps** identified in the code review: incomplete ad coverage due to page-limited scraping, inefficient HTTP client usage, and silent notification loss. After this phase, the worker will:
- Capture a larger window of ads per scrape cycle (no longer limited to 30 newest)
- Reuse TCP/TLS connections for willhaben API calls instead of creating one per request
- Retry failed Telegram notifications instead of dropping them permanently
## Architecture
```
┌──────────────────────────────────────────────────────┐
│ worker container │
│ │
│ ┌───────────┐ │
│ │ scraper.py│ ← SINGLETON AsyncClient │
│ │ │ (connection pool, keepalive) │
│ │ │ │
│ │ fetch_ads() │
│ │ ├─ page 1: rows=30 & published_after=<cursor> │
│ │ ├─ page 2: rows=30 & offset=30 │
│ │ └─ ... until no new ads or max_pages reached │
│ └───────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ ┌──────────────────┐ │
│ │ notifier.py │──►│ notification_queue│ │
│ │ │ │ table (new) │ │
│ │ notify_new() │ │ │ │
│ │ notify_drop()│ │ - ad_id │ │
│ │ │ │ - telegram_id │ │
│ │ if success: │ │ - attempts (0→5) │ │
│ │ log_notify │ │ - last_error │ │
│ │ if fail: │ │ - status │ │
│ │ enqueue! │ └────────┬─────────┘ │
│ └──────────────┘ │ │
│ ▼ │
│ scheduler retries │
│ pending items each cycle │
└──────────────────────────────────────────────────────┘
```
## Tasks
| Task | File | Description |
|------|------|-------------|
| Pagination in willhaben scraper | [task-scraper-pagination.md](./task-scraper-pagination.md) | Implement cursor-based or offset pagination to fetch more than 30 ads per cycle, tracking the last seen timestamp to avoid duplicates across cycles. |
| httpx singleton with connection pool | [task-httpx-singleton.md](./task-httpx-singleton.md) | Replace per-call AsyncClient creation with a module-level singleton using keepalive connections and configurable limits. |
| Retry queue for failed notifications | [task-notification-retry-queue.md](./task-notification-retry-queue.md) | Add a `notification_queue` table to persist failed Telegram sends with exponential backoff retries (up to 5 attempts). |
## General Acceptance Criteria
- [ ] A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30
- [ ] Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
- [ ] HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
- [ ] Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
- [ ] After 5 failed retries the notification is marked as `dead` and logged — not silently dropped
- [ ] The scheduler processes queued notifications at the start of each cycle before scraping new keywords
docs/phase-1/task-httpx-singleton.md
+157
View File
@@ -0,0 +1,157 @@
# Task: httpx singleton with connection pool
## Description
The current `scraper.fetch_ads()` creates a **new** `httpx.AsyncClient` on every call:
```python
async def fetch_ads(keyword: str):
async with httpx.AsyncClient(timeout=30.0) as client:
resp = await client.get(_API_URL, ...)
```
This means each scrape cycle incurs the full cost of TCP handshake + TLS negotiation (≈100-300ms per call on a cold connection). For keywords scraped every 5 minutes with multiple pages, this overhead adds up to **seconds of unnecessary latency per cycle**.
This task replaces the per-call client with a module-level singleton that reuses connections via keepalive.
## Architecture
```
Current:
Cycle 1: create AsyncClient → fetch → close → ~300ms overhead
Cycle 2: create AsyncClient → fetch → close → ~300ms overhead
Cycle N: ... (repeated forever)
Target:
Module load: create AsyncClient (singleton, keepalive pool)
Cycle 1: use client → fetch → ~50ms (warm connection)
Cycle 2: use client → fetch → ~50ms (warm connection)
Cycle N: ...
Shutdown: close client gracefully
```
### Key design decisions
- **Module-level singleton** (`_client = None`, lazy init). Simpler than dependency injection and works with the existing async context.
- **Keepalive connections**: Default `max_keepalive_connections=5` handles concurrent keyword scrapes efficiently.
- **Client recreation on error**: If the client is closed or encounters a fatal transport error, it's recreated on the next call. This prevents stale connection issues.
## Implementation Details
### 1. Add singleton getter to `scraper.py`
```python
import os
_client: httpx.AsyncClient | None = None
async def get_client() -> httpx.AsyncClient:
"""Return a shared AsyncClient with keepalive connection pool."""
global _client
if _client is None or _client.is_closed:
max_conns = int(os.getenv("HTTP_MAX_CONNECTIONS", "10"))
max_keepalive = int(os.getenv("HTTP_KEEPALIVE_CONNECTIONS", "5"))
_client = httpx.AsyncClient(
timeout=float(os.getenv("HTTP_TIMEOUT_S", "30.0")),
limits=httpx.Limits(
max_connections=max_conns,
max_keepalive_connections=max_keepalive,
keepalive_expiry=60, # seconds
),
)
logger.info(
"Created httpx client: max_conns=%d, keepalive=%d",
max_conns, max_keepalive,
)
return _client
async def close_client() -> None:
"""Close the shared AsyncClient. Call during shutdown."""
global _client
if _client and not _client.is_closed:
await _client.aclose()
logger.info("Closed httpx client")
_client = None
```
### 2. Update `fetch_ads()` to use the singleton
**Replace:**
```python
async def fetch_ads(keyword: str):
params = {...}
async with httpx.AsyncClient(timeout=30.0) as client:
for attempt in range(1, 4):
try:
resp = await client.get(_API_URL, headers=_HEADERS, params=params)
```
**With:**
```python
async def fetch_ads(keyword: str):
params = {...}
client = await get_client()
for attempt in range(1, 4):
try:
resp = await client.get(_API_URL, headers=_HEADERS, params=params)
resp.raise_for_status()
data = resp.json()
break
except httpx.ConnectError as exc:
# Transport error — recreate client on next attempt
logger.warning("Transport error on attempt %d: %s", attempt, exc)
await close_client() # force recreation
if attempt < 3:
await asyncio.sleep(2 ** attempt)
continue
raise
except Exception as exc:
logger.warning("fetch_ads attempt %d failed: %s", attempt, exc)
if attempt < 3:
await asyncio.sleep(2 ** attempt)
continue
raise
# ... rest unchanged (extract ads_raw, total_hits)
```
### 3. Call `close_client()` during shutdown in `main.py`
Add to the cleanup function (from Phase 0 task-graceful-shutdown):
```python
async def cleanup(app: Application) -> None:
logger.info("Shutting down...")
# ... existing cleanup steps ...
# Close HTTP client
from scraper import close_client
await close_client()
# ... rest of cleanup (close DB pool, etc.)
```
### 4. Update `.env.example` with new config options
```bash
# HTTP Client Configuration
HTTP_MAX_CONNECTIONS=10 # Max concurrent connections to willhaben API
HTTP_KEEPALIVE_CONNECTIONS=5 # Connections kept alive in the pool
HTTP_TIMEOUT_S=30.0 # Request timeout in seconds
```
## Acceptance Criteria
- [ ] Only one `httpx.AsyncClient` is created per process lifetime (logged once at startup)
- [ ] Subsequent calls to `fetch_ads()` reuse the existing client (no "Created httpx client" log)
- [ ] After calling `close_client()`, a new call to `get_client()` creates a fresh client
- [ ] Connection keepalive reduces latency for sequential API calls (verifiable via timing in logs)
- [ ] Fatal transport errors trigger client recreation without crashing the scheduler
- [ ] The client is properly closed during graceful shutdown (no resource warnings)
docs/phase-1/task-notification-retry-queue.md
+301
View File
@@ -0,0 +1,301 @@
# Task: Retry queue for failed notifications
## Description
The current `notifier.notify_new()` and `notify_drop()` call `_send_message()` directly. If the Telegram API returns an error (rate limiting, network hiccup, user deleted the bot), the notification is **silently logged** and never retried:
```python
async def _send_message(bot: Bot, chat_id: int, message_text):
try:
await bot.send_message(chat_id=chat_id, text=message_text)
except TelegramError as e:
logger.warning("Failed to send notification ...") # ← notification LOST forever
```
This task introduces a **persistent retry queue** backed by the `notification_queue` table. Failed notifications are stored with an attempt counter and retried on subsequent scheduler cycles with exponential backoff. After 5 failed attempts, they're marked as `dead` and logged — not silently dropped.
## Architecture
```
┌───────────────────────────────────────┐
│ notification_queue table (new) │
│ │
│ id uuid PK │
│ ad_id uuid │
│ telegram_id text │
│ message_text text │
│ type enum('new','drop') │
│ attempts int DEFAULT 0 │
│ max_attempts int DEFAULT 5 │
│ last_error text │
│ status enum │
│ ('pending','sent', │
│ 'failed','dead') │
│ created_at timestamptz │
│ updated_at timestamptz │
│ │
│ INDEX: status, attempts (composite) │
└───────────┬───────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ notifier.py flow: │
│ │
│ send_notification(): │
│ try: │
│ await bot.send_message(...) │
│ → log_notify() (as before) │
│ except TelegramError as e: │
│ INSERT INTO notification_queue │
│ (ad_id, telegram_id, ...) │
│ VALUES (...) │
└───────────┬───────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ scheduler.py / main.py: │
│ At the START of each cycle: │
│ │
│ for item in pending_queue: │
│ if attempts < max_attempts: │
│ backoff = 2^attempts minutes │
│ if now >= updated_at + backoff:│
│ try send again │
│ success → mark 'sent' │
│ fail → increment count │
│ elif attempts >= max_attempts: │
│ mark as 'dead' │
│ log warning │
└───────────────────────────────────────┘
Exponential backoff schedule:
Attempt 1 → wait 1 min (2^0)
Attempt 2 → wait 2 min (2^1)
Attempt 3 → wait 4 min (2^2)
Attempt 4 → wait 8 min (2^3)
Attempt 5 → wait 16 min (2^4)
Total worst case: ~31 min before giving up
```
### Key design decisions
- **Store full message text** in the queue table so we can retry without re-rendering. This is important because ad data might change or be removed from willhaben by the time we retry.
- **Process at start of scheduler cycle** — ensures queued items are attempted before new scraping starts, prioritizing user notifications over fresh data collection.
- **Backoff based on `updated_at`**, not wall-clock from first attempt. Each retry resets the backoff timer. This handles edge cases where a failure was transient but then another transient follows.
## Implementation Details
### 1. Add migration for `notification_queue` table
In `worker/src/migrations/02-notification-queue.sql`:
```sql
-- Notification retry queue table
CREATE TYPE notification_type AS ENUM ('new', 'drop');
CREATE TYPE notification_status AS ENUM ('pending', 'sent', 'failed', 'dead');
CREATE TABLE IF NOT EXISTS notification_queue (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
ad_id uuid NOT NULL REFERENCES ads(id) ON DELETE SET NULL,
telegram_id text NOT NULL,
message_text text NOT NULL,
type notification_type NOT NULL,
attempts int NOT NULL DEFAULT 0,
max_attempts int NOT NULL DEFAULT 5,
last_error text,
status notification_status NOT NULL DEFAULT 'pending',
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
-- Index for efficient queue polling
CREATE INDEX IF NOT EXISTS idx_notif_queue_poll
ON notification_queue(status, attempts, updated_at)
WHERE status IN ('pending', 'failed');
COMMENT ON TABLE notification_queue IS
'Persistent retry queue for failed Telegram notifications';
```
### 2. Update `notifier.py` — enqueue on failure
**Current:**
```python
async def _send_message(bot: Bot, chat_id: int, message_text):
try:
await bot.send_message(chat_id=chat_id, text=message_text)
except TelegramError as e:
logger.warning(
"Failed to send notification to %d: %s", chat_id, str(e)[:30]
)
```
**After:**
```python
async def _send_message(
bot: Bot,
chat_id: int,
message_text: str,
ad_id: uuid.UUID | None = None,
notif_type: str = "new",
):
try:
await bot.send_message(chat_id=chat_id, text=message_text)
except TelegramError as e:
error_msg = str(e)[:300] # cap length
logger.warning("Telegram send failed for %s: %s", chat_id, error_msg)
if ad_id:
await _enqueue_retry(
ad_id=ad_id,
telegram_id=str(chat_id),
message_text=message_text,
notif_type=notif_type,
error_msg=error_msg,
)
async def _enqueue_retry(
ad_id: uuid.UUID,
telegram_id: str,
message_text: str,
notif_type: str,
error_msg: str,
) -> None:
"""Store a failed notification for later retry."""
from db import get_pool
pool = await get_pool()
# Check if already queued (avoid duplicates for same ad+user)
existing = await pool.fetchval(
"""SELECT id FROM notification_queue
WHERE ad_id = $1 AND telegram_id = $2 AND status IN ('pending', 'failed')""",
ad_id, telegram_id,
)
if existing:
logger.info("Already queued: ad=%s user=%s", ad_id[:8], telegram_id)
return
await pool.execute(
"""INSERT INTO notification_queue
(ad_id, telegram_id, message_text, type, last_error, status)
VALUES ($1, $2, $3, $4, $5, 'pending')""",
ad_id, telegram_id, message_text, notif_type, error_msg,
)
logger.info("Queued for retry: ad=%s user=%s", ad_id[:8], telegram_id)
```
### 3. Add queue processing to scheduler in `main.py`
At the start of each scheduler cycle (before keyword scraping):
```python
async def process_notification_queue() -> int:
"""Process pending notifications from the retry queue."""
from db import get_pool
pool = await get_pool()
# Get items eligible for retry (backoff respected)
rows = await pool.fetch("""
SELECT id, ad_id, telegram_id, message_text, type, attempts,
max_attempts, last_error, updated_at
FROM notification_queue
WHERE status IN ('pending', 'failed')
AND updated_at + ($1 || ' minutes')::interval <= now()
ORDER BY attempts ASC, updated_at ASC
""", "2^attempts" if pool.is_pg else 0)
# Actually use a computed backoff in Python since PG expressions are tricky:
rows = await pool.fetch("""
SELECT id, ad_id, telegram_id, message_text, type, attempts,
max_attempts, last_error, updated_at
FROM notification_queue
WHERE status IN ('pending', 'failed')
ORDER BY attempts ASC, updated_at ASC
LIMIT 50
""")
processed = 0
for row in rows:
backoff_min = min(2 ** row["attempts"], 60) # cap at 60 min
retry_after = row["updated_at"] + timedelta(minutes=backoff_min)
if datetime.now(tz=timezone.utc) < retry_after:
continue # not yet eligible
try:
from bot import get_application_bot
bot = get_application_bot()
await bot.send_message(
chat_id=int(row["telegram_id"]),
text=row["message_text"]
)
await pool.execute(
"UPDATE notification_queue SET status = 'sent', updated_at = now() WHERE id = $1",
row["id"],
)
await log_notify(pool, row["ad_id"], int(row["telegram_id"]), row["type"])
processed += 1
except TelegramError as e:
new_attempts = row["attempts"] + 1
if new_attempts >= row["max_attempts"]:
await pool.execute(
"""UPDATE notification_queue
SET status = 'dead', attempts = $2, last_error = $3, updated_at = now()
WHERE id = $1""",
row["id"], new_attempts, str(e)[:300],
)
logger.error(
"Notification DEAD after %d attempts: ad=%s user=%s err=%s",
new_attempts, row["ad_id"][:8], row["telegram_id"], e,
)
else:
await pool.execute(
"""UPDATE notification_queue
SET status = 'failed', attempts = $2, last_error = $3, updated_at = now()
WHERE id = $1""",
row["id"], new_attempts, str(e)[:300],
)
return processed
```
### 4. Call `process_notification_queue()` in the scheduler loop
In `main.py`, before iterating keywords:
```python
async def run_scheduler() -> None:
while True:
try:
record_scheduler_run() # healthcheck
# Process pending notifications FIRST
processed = await process_notification_queue()
if processed:
logger.info("Retried %d queued notifications", processed)
# ... existing keyword iteration ...
```
## Acceptance Criteria
- [ ] When `_send_message()` raises `TelegramError`, the notification is INSERTed into `notification_queue` with status='pending'
- [ ] On the next scheduler cycle, pending items are attempted (respecting backoff)
- [ ] After 5 failed attempts, the notification status becomes 'dead' and a warning is logged
- [ ] The queue processes at most 50 items per cycle to avoid blocking the scheduler
- [ ] Duplicate enqueue prevention works: calling `_enqueue_retry` twice for the same ad+user creates only one queue entry
- [ ] Successful retries update `log_notifications` table (same as direct notifications)
- [ ] The `/health` endpoint or logs can show the current count of pending/dead items
docs/phase-1/task-scraper-pagination.md
+189
View File
@@ -0,0 +1,189 @@
# Task: Pagination in willhaben scraper
## Description
The current `scraper.fetch_ads()` only fetches page 1 (30 ads, sorted newest first). For popular keywords, this means many new listings are missed between scrape cycles — especially when the cycle interval is ≥5 minutes.
This task implements **cursor-based pagination** that:
- Fetches multiple pages per cycle (configurable, default: 2 pages = 60 ads)
- Tracks a cursor timestamp to avoid re-processing already-seen ads from the previous cycle
- Respects API rate limits by adding delays between page fetches
## Architecture
```
Current flow (page 1 only):
scheduler → fetch_ads("keyword")
→ GET .../ad-search?rows=30&sort=1
→ process 30 ads → done
Target flow (paginated with cursor):
scheduler → fetch_ads("keyword", last_seen_cursor)
├─ GET ?rows=30&offset=0 → process batch, track latest timestamp
├─ sleep 1s (politeness)
├─ GET ?rows=30&offset=30 → process batch, stop if duplicates detected
└─ ... until max_pages or no new ads
After processing: update last_seen_cursor for this keyword
Database tracking:
keywords table adds: last_seen_cursor timestamptz
(or use existing last_scraped_at as cursor — simpler)
```
### Key design decisions
- **Use `last_scraped_at` as cursor** instead of adding a new column. After each cycle, the earliest ad processed becomes the cursor for the next cycle. Only ads newer than this are candidates for notification.
- *Tradeoff*: If an ad was posted exactly between cycles, it could be missed if it appears on page 2+. Mitigated by processing at least 2 pages and keeping intervals short.
- **`max_pages` config via env var** (`SCRAPE_MAX_PAGES=2`). Default is conservative (2) to balance coverage vs API load. Users with expensive keywords can increase per-keyword later.
- **Stop early on duplicate detection**: If page N has the same `PUBLISHED_String` as page N-1's last ad, stop — we've exhausted newer results.
## Implementation Details
### 1. Update `keywords` table schema
Add a cursor column (or reuse `last_scraped_at`). Recommendation: **reuse** since it already exists and is indexed:
```sql
-- No new column needed. Use last_scraped_at as the cursor.
-- Ads published after last_scraped_at are "new" for this cycle.
```
If we want a dedicated, more precise cursor (in case last_scraped_at is set before processing completes):
```sql
ALTER TABLE keywords ADD COLUMN IF NOT EXISTS ads_cursor timestamptz;
COMMENT ON COLUMN keywords.ads_cursor IS
'Timestamp of the oldest ad processed in the last cycle. Used for pagination.';
```
### 2. Update `scraper.py` — add pagination support
```python
_MAX_PAGES = int(os.getenv("SCRAPE_MAX_PAGES", "2"))
_PAGE_DELAY_S = float(os.getenv("SCRAPE_PAGE_DELAY_S", "1.0"))
async def fetch_ads(
keyword: str,
cursor_at: datetime | None = None,
max_pages: int | None = None,
) -> tuple[list[dict[str, Any]], int]:
"""Fetch ads with pagination, deduping by cursor timestamp."""
pages = max_pages or _MAX_PAGES
all_ads_raw: list[dict[str, Any]] = []
total_hits: int = 0
client = await get_client() # from task-httpx-singleton
for page in range(pages):
params = {
"keyword": keyword,
"rows": 30,
"sort": 1, # newest first
"offset": page * 30,
}
try:
resp = await fetch_with_retry(client, _API_URL, params)
data = resp.json()
total_hits = int(data.get("rowsFound", 0))
except Exception as exc:
logger.warning(
"fetch_ads page %d failed for '%s': %s", page, keyword, exc
)
break
page_ads = (data.get("advertSummaryList") or {}).get("advertSummary", [])
if not page_ads:
logger.info("No more ads on page %d for '%s'", page, keyword)
break
# Check early stop: if oldest ad on this page is at or before cursor
oldest_published = _get_oldest_published(page_ads)
if cursor_at and oldest_published and oldest_published <= cursor_at:
logger.info(
"Early stop at page %d for '%s' — reached cursor",
page, keyword
)
break
# Filter out already-seen ads within this batch
new_batch = [
ad for ad in page_ads
if not cursor_at or _get_published(ad) is None or _get_published(ad) > cursor_at
]
all_ads_raw.extend(new_batch)
# Politeness delay between pages (not after last page)
if page < pages - 1 and new_batch:
await asyncio.sleep(_PAGE_DELAY_S)
return all_ads_raw, total_hits
def _get_published(ad_dict: dict) -> datetime | None:
"""Extract published timestamp from a single ad dict."""
attrs = _parse_attributes(ad_dict)
raw = attrs.get("PUBLISHED_String") or attrs.get("CHANGED_String")
if not raw:
return None
try:
return datetime.fromisoformat(raw.replace("Z", "+00:00"))
except (ValueError, TypeError):
return None
def _get_oldest_published(ads: list[dict]) -> datetime | None:
"""Get the oldest published timestamp from a batch of ads."""
timestamps = [_get_published(ad) for ad in ads]
timestamps = [t for t in timestamps if t is not None]
return min(timestamps) if timestamps else None
```
### 3. Update `main.py` scheduler to pass cursor and update it
In the scheduler loop, before calling `fetch_ads`:
```python
# Get current cursor (last_scraped_at or ads_cursor)
cursor = row["ads_cursor"] or row["last_scraped_at"]
ads_raw, total_hits = await fetch_ads(keyword, cursor_at=cursor)
# ... process ads ...
# Update cursor to the oldest new ad processed
if new_timestamps:
oldest_new = min(new_timestamps)
await pool.execute(
"UPDATE keywords SET last_scraped_at = now(), ads_cursor = $1 WHERE id = $2",
oldest_new, kw_id,
)
else:
await pool.execute(
"UPDATE keywords SET last_scraped_at = now() WHERE id = $2",
kw_id,
)
```
### 4. Add `ads_cursor` to migration file
In `worker/src/migrations/01-schema.sql`, add after the keywords table:
```sql
ALTER TABLE keywords ADD COLUMN IF NOT EXISTS ads_cursor timestamptz;
COMMENT ON COLUMN keywords.ads_cursor IS
'Timestamp of oldest ad processed in last cycle, for pagination cursor';
```
## Acceptance Criteria
- [ ] A scrape cycle fetches at least 2 pages (60 ads) by default for active keywords
- [ ] The `max_pages` limit is configurable via `SCRAPE_MAX_PAGES` environment variable
- [ ] Early stop detection works: if page N has no newer ads than the cursor, pagination stops without fetching remaining pages
- [ ] Ads are not re-notified across cycles (cursor prevents duplicates)
- [ ] A 1-second delay between page fetches is logged and respected
- [ ] `total_hits` from willhaben API is still returned for logging/stats purposes
- [ ] No regression in single-page behavior when max_pages=1
docs/phase-2/Phase.md
+82
View File
@@ -0,0 +1,82 @@
# Phase 2 — User Experience & Advanced Filtering
## Scope
This phase introduces **user-facing features** that significantly improve the experience of keyword tracking. Currently, every matching ad triggers an instant notification regardless of price, location, or time of day — leading to noise for popular keywords.
After this phase:
- Users can configure price ranges and postcodes per keyword
- Notifications respect mute hours (no alerts at 3 AM)
- Users opt into digest mode (bundled summaries instead of individual pings)
## Architecture
```
┌──────────────────────────────────────────────┐
│ User Interaction Layer │
│ │
│ Telegram Bot Commands: │
│ /set_price_min <kw> <€> │
│ /set_price_max <kw> <€> │
│ /set_postcode <kw> <list> │
│ /mute_hours <start>-<end> │
│ /digest on|off │
│ │
└──────────┬───────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ Database Schema (extended) │
│ │
│ keywords table: │
│ + price_min int │
│ + price_max int │
│ + allowed_postcodes text[] │
│ │
│ user_settings table (new): │
│ telegram_id text PK │
│ mute_start time │
│ mute_end time │
│ digest_mode bool DEFAULT false │
│ digest_interval int DEFAULT 60 │
└──────────┬───────────────────────────────────┘
│
▼
┌──────────────────────────────────────────────┐
│ Notification Pipeline (modified) │
│ │
│ For each new ad: │
│ ├─ filter by price_min/max? → skip │
│ ├─ filter by allowed_postcodes? → skip │
│ ├─ user in mute hours? │
│ │ digest_on → buffer to digest_table │
│ │ digest_off→ skip notification │
│ └─ normal → send now │
│ │
│ Digest scheduler (separate task): │
│ every digest_interval: │
│ collect buffered notifications per user │
│ format as summary message │
│ send single message │
│ clear buffer │
└──────────────────────────────────────────────┘
```
## Tasks
| Task | File | Description |
|------|------|-------------|
| Price range filters per keyword | [task-price-filters.md](./task-price-filters.md) | Add `price_min` and `price_max` columns to the keywords table; filter ads during processing based on these thresholds. Bot commands to set/unset. |
| Location / postcode filters per keyword | [task-postcode-filters.md](./task-postcode-filters.md) | Add `allowed_postcodes` text[] column to keywords; only notify if an ad's location matches any allowed postcode. |
| Mute hours per user | [task-mute-hours.md](./task-mute-hours.md) | Create `user_settings` table with configurable mute window (start/end time in UTC); suppress notifications during this window. |
| Digest / summary notifications | [task-digest-notifications.md](./task-digest-notifications.md) | Buffer notifications for users with digest mode enabled; send a bundled summary at configured intervals instead of individual alerts. |
## General Acceptance Criteria
- [ ] Users can set price min/max on any keyword and only receive notifications within that range
- [ ] Postcode filtering works — ads outside allowed postcodes are silently skipped (not counted as new)
- [ ] Mute hours suppress all notifications to a user during the configured window, regardless of keyword
- [ ] Digest mode buffers individual alerts and sends one summary message at the configured interval
- [ ] All filters combine correctly: an ad is only notified if it passes price + postcode checks AND the user is not muted (or digest mode active)
- [ ] The bot provides clear feedback when a filter setting is changed ("Keyword X: price range set to €100–€500")
- [ ] Admin can view all keyword filters and user settings via `/keywords` command output
docs/phase-2/task-digest-notifications.md
+355
View File
@@ -0,0 +1,355 @@
# Task: Digest / summary notifications
## Description
For popular keywords that generate many matches per cycle, users may receive 10–20 individual notifications in quick succession. This task introduces **digest mode** — instead of immediate alerts, notifications are buffered and sent as a single summary message at configurable intervals.
Digest mode is complementary to mute hours: during mute hours, all messages are suppressed; with digest mode ON, messages are collected and sent as a batch at the configured interval (default: every 60 minutes).
## Architecture
```
┌───────────────────────────────────────┐
│ user_settings table │
│ digest_mode bool DEFAULT false │
│ digest_interval int DEFAULT 60 │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ Notification Pipeline (modified) │
│ │
│ For each new ad: │
│ if user.digest_mode == false: │
│ → send immediately (current) │
│ elif in mute hours: │
│ → discard (already handled) │
│ else: │
│ → insert into digest_buffer │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ digest_buffer table (new) │
│ │
│ id uuid PK │
│ telegram_id text │
│ ad_id uuid REFERENCES ads │
│ keyword text │
│ title text │
│ price int │
│ url text │
│ created_at timestamptz │
│ │
│ INDEX: telegram_id, created_at │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ Digest Scheduler (separate task) │
│ │
│ Runs every digest_interval per user │
│ ├─ SELECT all buffered items │
│ ├─ GROUP BY telegram_id │
│ ├─ Format summary message │
│ └─ DELETE buffered items │
│ │
│ Summary format: │
│ 📋 Digest — 5 new ads (14:30 UTC) │
│ │
│ 🔑 "rtx 3090" (3 ads): │
│ • RTX 3090 Ti - €750 [link] │
│ • ASUS RTX 3090 - €680 [link] │
│ • MSI RTX 3090 Gaming X - €720 │
│ │
│ 🔑 "gtx 1660" (2 ads): │
│ • GTX 1660 Super - €120 [link] │
│ • EVGA GTX 1660 - €95 [link] │
└───────────────────────────────────────┘
```
### Key design decisions
- **Separate buffer table** instead of in-memory only. Survives restarts, visible via pgAdmin for debugging.
- **Per-user interval**: Each user configures their own digest frequency (default 60 min). Implemented with a single scheduler task that checks all users' intervals on each cycle.
- **Group by keyword** in the summary message. Makes it easy to scan relevant categories without digging through unrelated listings.
## Implementation Details
### 1. Add migration
In `worker/src/migrations/04-user-settings.sql`:
```sql
-- digest_buffer table for accumulating notifications
CREATE TABLE IF NOT EXISTS digest_buffer (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
telegram_id text NOT NULL,
ad_id uuid REFERENCES ads(id) ON DELETE CASCADE,
keyword text NOT NULL,
title text NOT NULL,
price int, -- in cents
url text,
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_digest_buffer
ON digest_buffer(telegram_id, created_at DESC);
COMMENT ON TABLE digest_buffer IS
'Buffer for digest-mode notifications. Flushed to Telegram at intervals.';
```
### 2. Buffer notifications instead of sending immediately
In `main.py` scheduler loop, after all filters pass and mute check passes:
```python
# Check if user has digest mode enabled
settings = await pool.fetchrow(
"SELECT digest_mode FROM user_settings WHERE telegram_id = $1",
telegram_id_str,
)
if settings and settings["digest_mode"]:
# Buffer for digest
await pool.execute(
"""INSERT INTO digest_buffer (telegram_id, ad_id, keyword, title, price, url)
VALUES ($1, $2, $3, $4, $5, $6)""",
telegram_id_str,
ad_id,
kw_row["keyword"],
ad_dict.get("title", "Unknown"),
_extract_price(ad_dict),
ad_dict.get("url"),
)
# Still log the notification (for stats)
await log_notify(pool, ad_id, telegram_id, "new")
else:
# Send immediately (current behavior)
await notify_new(bot, pool, kw_row["keyword"],
telegram_id, ad_dict, ad_id)
```
### 3. Add digest flushing task to scheduler
In `main.py`, add a new async function and call it at the start of each cycle:
```python
async def flush_digest_buffers() -> int:
"""Process pending digest buffers for users whose interval has elapsed."""
from db import get_pool
pool = await get_pool()
# Get all users with digest mode ON
users = await pool.fetch("""
SELECT telegram_id, digest_interval
FROM user_settings
WHERE digest_mode = true
""")
sent_count = 0
for user in users:
interval_min = user["digest_interval"] or 60
cutoff = datetime.now(tz=timezone.utc) - timedelta(minutes=interval_min)
# Get buffered items older than the interval
buffered = await pool.fetch("""
SELECT db.id, db.keyword, db.title, db.price, db.url
FROM digest_buffer db
WHERE db.telegram_id = $1
AND db.created_at <= $2
ORDER BY db.keyword, db.created_at DESC
""", user["telegram_id"], cutoff)
if not buffered:
continue
# Group by keyword
from collections import defaultdict
groups: dict[str, list] = defaultdict(list)
for item in buffered:
price_str = f"€{item['price']/100:.2f}" if item['price'] else "Free"
entry = f"• {item['title']} - {price_str}"
groups[item["keyword"]].append(entry)
# Build summary message
lines = [f"📋 Digest — {len(buffered)} new ads ({cutoff:%H:%M}–{datetime.now(tz=timezone.utc):%H:%M} UTC)\n"]
for kw_name, entries in groups.items():
lines.append(f"\n🔑 \"{kw_name}\" ({len(entries)} ads):")
# Limit to 10 entries per keyword to avoid spam
for entry in entries[:10]:
lines.append(entry)
if len(entries) > 10:
lines.append(f" ... and {len(entries)-10} more")
message_text = "\n".join(lines)
# Send the digest
try:
from bot import get_application_bot
bot = get_application_bot()
telegram_id_int = int(user["telegram_id"])
await bot.send_message(chat_id=telegram_id_int, text=message_text)
sent_count += 1
# Log all buffered notifications as delivered
buffer_ids = [item["id"] for item in buffered]
for ad_item in buffered:
await log_notify(pool, ad_item["ad_id"], telegram_id_int, "new")
except TelegramError as e:
logger.error("Digest send failed for %s: %s", user["telegram_id"], e)
finally:
# Clear the buffer (whether sent or not — if it failed, log entries remain in DB)
await pool.execute(
"""DELETE FROM digest_buffer
WHERE telegram_id = $1 AND created_at <= $2""",
user["telegram_id"], cutoff,
)
return sent_count
# Call at the start of run_scheduler():
async def run_scheduler() -> None:
while True:
try:
record_scheduler_run()
# Flush digest buffers first
digests_sent = await flush_digest_buffers()
if digests_sent:
logger.info("Sent %d digest summaries", digests_sent)
# Process notification queue...
processed = await process_notification_queue()
if processed:
logger.info("Retried %d queued notifications", processed)
# ... existing keyword iteration ...
```
### 4. Add bot commands in `bot.py`
```python
async def cmd_digest_on(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Enable digest mode with optional interval."""
interval = 60 # default minutes
if len(context.args) > 0:
try:
interval = int(context.args[0])
if interval < 5 or interval > 1440:
raise ValueError
except ValueError:
await update.message.reply_text(
"Usage: /digest_on [minutes]\n"
"Interval must be between 5 and 1440 minutes (24h)."
)
return
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
await pool.execute(
"""INSERT INTO user_settings (telegram_id, digest_mode, digest_interval)
VALUES ($1, true, $2)
ON CONFLICT (telegram_id)
DO UPDATE SET digest_mode = EXCLUDED.digest_mode,
digest_interval = EXCLUDED.digest_interval""",
telegram_id, interval,
)
await update.message.reply_text(
f"✅ Digest mode ENABLED\n"
f"Digests will be sent every {interval} minutes.\n"
"Use /digest_off to return to instant notifications."
)
async def cmd_digest_off(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Disable digest mode."""
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
await pool.execute(
"""INSERT INTO user_settings (telegram_id, digest_mode)
VALUES ($1, false)
ON CONFLICT (telegram_id)
DO UPDATE SET digest_mode = EXCLUDED.digest_mode""",
telegram_id,
)
# Flush any remaining buffered items immediately
await pool.execute(
"""DELETE FROM digest_buffer WHERE telegram_id = $1""",
telegram_id,
)
await update.message.reply_text(
"✅ Digest mode DISABLED — notifications are now instant."
)
async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Show current user settings (extended from mute hours task)."""
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
settings = await pool.fetchrow(
"SELECT mute_start, mute_end, digest_mode, digest_interval "
"FROM user_settings WHERE telegram_id = $1",
telegram_id,
)
reply_parts = []
if not settings or not settings["mute_start"]:
reply_parts.append("🔕 Mute hours: OFF")
else:
reply_parts.append(
f"🔕 Mute hours: {settings['mute_start']} — {settings['mute_end']} UTC"
)
if settings and settings["digest_mode"]:
reply_parts.append(
f"📋 Digest: ON (every {settings['digest_interval']} min)"
)
else:
reply_parts.append("📋 Digest: OFF (instant notifications)")
await update.message.reply_text("\n".join(reply_parts))
```
Register handlers:
```python
dp.add_handler(CommandHandler("digest_on", cmd_digest_on))
dp.add_handler(CommandHandler("digest_off", cmd_digest_off))
```
## Acceptance Criteria
- [ ] `/digest_on` enables digest mode with 60-minute default interval
- [ ] `/digest_on 30` sets digest interval to 30 minutes
- [ ] New ads are inserted into `digest_buffer` instead of being sent immediately when digest is ON
- [ ] At the configured interval, all buffered items are flushed as a single summary message
- [ ] The summary groups ads by keyword and includes price information
- [ ] After flushing, buffered items are deleted from the table
- [ ] `/digest_off` disables digest mode and sends any remaining buffered items immediately
- [ ] Mute hours take precedence over digest — muted notifications are discarded, not buffered
docs/phase-2/task-mute-hours.md
+211
View File
@@ -0,0 +1,211 @@
# Task: Mute hours per user
## Description
Users may add keywords that are popular enough to trigger notifications at any hour. Currently, there's no way to suppress alerts during sleeping hours — the bot sends notifications 24/7.
This task creates a `user_settings` table with configurable mute windows (start/end time). During the mute window, notifications for that user are suppressed entirely. The notification is not lost — it's still logged in `log_notifications`, but the Telegram message is not sent.
## Architecture
```
┌───────────────────────────────────────┐
│ user_settings table (new) │
│ │
│ telegram_id text PRIMARY KEY │
│ mute_start time │
│ mute_end time │
│ digest_mode bool DEFAULT false │
│ digest_interval int DEFAULT 60 │
│ │
│ Example: │
│ telegram_id = '298181113' │
│ mute_start = '22:00:00' │
│ mute_end = '07:00:00' │
│ → no alerts between 10PM-7AM UTC │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ Notification pipeline (in main.py) │
│ │
│ For each new ad that passes filters: │
│ user_settings = get from DB │
│ if in_mute_hours(user_settings): │
│ log_notify() │
│ → skip Telegram send │
│ else: │
│ notify_new() / notify_drop() │
└───────────────────────────────────────┘
```
### Key design decisions
- **Time stored as `time` type in PostgreSQL** — native, efficient for range checks. Default is NULL (no mute window).
- *Alternative*: Could store as integer hours (e.g., 22, 7), but `time` type gives flexibility for minute-level precision and clearer UI.
- **UTC timezone**: The bot operates in UTC internally. Users should be informed that mute times are in UTC. Adding timezone support per-user is a Phase 3 consideration.
- **Mute window can cross midnight** — start > end means the window wraps around midnight (e.g., 22:00–07:00). The check handles this correctly.
## Implementation Details
### 1. Add migration
In `worker/src/migrations/04-user-settings.sql`:
```sql
CREATE TABLE IF NOT EXISTS user_settings (
telegram_id text PRIMARY KEY,
mute_start time,
mute_end time,
digest_mode bool NOT NULL DEFAULT false,
digest_interval int NOT NULL DEFAULT 60, -- minutes
CONSTRAINT chk_mute_hours CHECK (
mute_start IS NULL AND mute_end IS NULL
OR mute_start IS NOT NULL AND mute_end IS NOT NULL
)
);
COMMENT ON TABLE user_settings IS
'User-specific settings for notification behavior';
```
### 2. Add mute hours check in `notifier.py` or `main.py`
```python
async def _is_in_mute_hours(
telegram_id: str,
pool: asyncpg.Pool
) -> bool:
"""Check if the current time is within the user's mute window."""
settings = await pool.fetchrow(
"SELECT mute_start, mute_end FROM user_settings WHERE telegram_id = $1",
telegram_id,
)
if not settings or not settings["mute_start"] or not settings["mute_end"]:
return False # no mute configured
now_utc = datetime.now(tz=timezone.utc).time()
start = settings["mute_start"]
end = settings["mute_end"]
if start < end:
# Normal window (e.g., 22:00–07:00 → actually wraps, so this is rare)
return start <= now_utc <= end
else:
# Window crosses midnight (e.g., 22:00 to 07:00 next day)
return now_utc >= start or now_utc <= end
# In main.py scheduler loop, before calling notify_new():
telegram_id_str = str(telegram_id)
in_mute = await _is_in_mute_hours(telegram_id_str, pool)
if in_mute:
logger.debug("Muted notification for user %s (mute window active)", telegram_id)
# Still log it but don't send Telegram message
await log_notify(pool, ad_id, telegram_id, "new")
return
# Proceed with normal notification...
```
### 3. Add bot commands in `bot.py`
```python
async def cmd_set_mute(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Set mute hours window (HH:MM-HH:MM format)."""
if len(context.args) < 1:
await update.message.reply_text(
"Usage: /mute_hours HH:MM-HH:MM\n"
"Example: /mute_hours 22:00-07:00 (mutes from 10PM to 7AM UTC)\n"
"Use /mute_off to disable."
)
return
try:
start_str, end_str = context.args[0].split("-")
mute_start = datetime.strptime(start_str.strip(), "%H:%M").time()
mute_end = datetime.strptime(end_str.strip(), "%H:%M").time()
except (ValueError, TypeError) as e:
await update.message.reply_text(
f"Invalid format. Use HH:MM-HH:MM.\nExample: /mute_hours 22:00-07:00"
)
return
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
await pool.execute(
"""INSERT INTO user_settings (telegram_id, mute_start, mute_end)
VALUES ($1, $2, $3)
ON CONFLICT (telegram_id)
DO UPDATE SET mute_start = EXCLUDED.mute_start, mute_end = EXCLUDED.mute_end""",
telegram_id, mute_start, mute_end,
)
await update.message.reply_text(
f"✅ Mute hours set: {mute_start} — {mute_end} UTC\n"
"No notifications will be sent during this window.\n"
"Use /mute_off to disable or change."
)
async def cmd_mute_off(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Disable mute hours."""
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
await pool.execute(
"""INSERT INTO user_settings (telegram_id, mute_start, mute_end)
VALUES ($1, NULL, NULL)
ON CONFLICT (telegram_id)
DO UPDATE SET mute_start = EXCLUDED.mute_start, mute_end = EXCLUDED.mute_end""",
telegram_id,
)
await update.message.reply_text("✅ Mute hours disabled.")
async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Show current user settings."""
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
settings = await pool.fetchrow(
"SELECT mute_start, mute_end FROM user_settings WHERE telegram_id = $1",
telegram_id,
)
if not settings or not settings["mute_start"]:
reply = "🔕 Mute hours: OFF (notifications sent 24/7)"
else:
reply = f"🔕 Mute hours: {settings['mute_start']} — {settings['mute_end']} UTC"
await update.message.reply_text(reply)
```
Register handlers:
```python
dp.add_handler(MessageHandler(REGEX(r"^/mute_hours"), cmd_set_mute))
dp.add_handler(CommandHandler("mute_off", cmd_mute_off))
dp.add_handler(CommandHandler("status", cmd_status))
```
## Acceptance Criteria
- [ ] `/mute_hours 22:00-07:00` sets mute window from 10 PM to 7 AM UTC
- [ ] Notifications during the mute window are logged but NOT sent via Telegram
- [ ] Notifications outside the mute window work normally (no regression)
- [ ] Mute windows that cross midnight (start > end) are handled correctly
- [ ] `/mute_off` clears both start and end times, restoring 24/7 notifications
- [ ] `/status` shows current mute settings clearly
- [ ] Users without any settings in `user_settings` table receive all notifications (default behavior unchanged)
docs/phase-2/task-postcode-filters.md
+199
View File
@@ -0,0 +1,199 @@
# Task: Location / postcode filters per keyword
## Description
Ads can match a user's keyword interest but be located in a completely different region (e.g., "rtx 3090" in Graz when the user only cares about Vienna). This task adds optional postcode filtering so users receive alerts only for ads in their desired locations.
The willhaben API returns location data in `LOCATION_CityName`, `LOCATION_ZIP` (postcode), and similar fields. We'll match against these fields.
## Architecture
```
┌───────────────────────────────────────┐
│ keywords table (extended) │
│ │
│ ... │
│ allowed_postcodes text[] │
│ ... │
│ │
│ Example: │
│ allowed_postcodes = {'1010','1020'} │
│ → only ads in Vienna 1st/2nd dist. │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ Processing filter (in main.py) │
│ │
│ for ad in ads_raw: │
│ ad_zip = _extract_postcode(ad) │
│ if kw.allowed_postcodes and │
│ ad_zip not in postcodes: │
│ skip │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ Bot commands (in bot.py): │
│ │
│ /postcode <keyword> p1,p2,p3 │
│ → sets allowed_postcodes = {'p1', │
│ 'p2','p3'} │
│ /clear_postcode <keyword> │
│ → removes filter (NULL) │
└───────────────────────────────────────┘
```
### Key design decisions
- **Text array** (`text[]`) instead of a separate lookup table. Simple, efficient for the typical case (<10 postcodes per keyword), and leverages PostgreSQL's native array support.
- *Alternative*: A `keyword_postcodes` junction table allows individual postcode management but adds unnecessary complexity for this use case.
- **Match against willhaben's ZIP code field** (`LOCATION_ZIP` in the ad attributes). This is the most reliable location identifier and works across all Austrian postcodes (4 digits, e.g., "1010", "8010").
- **Empty or missing postcode = skip if filter active**. If `allowed_postcodes` is set but an ad has no ZIP code, it's excluded. This prevents noise from unlocated ads.
## Implementation Details
### 1. Add migration
In `worker/src/migrations/03-keyword-filters.sql`:
```sql
ALTER TABLE keywords ADD COLUMN IF NOT EXISTS allowed_postcodes text[];
COMMENT ON COLUMN keywords.allowed_postcodes IS
'Austrian postcodes (4-digit strings). Only ads matching these are notified.';
```
### 2. Extract postcode from ad data in `notifier.py` or `scraper.py`
Add helper function:
```python
def _extract_postcode(ad_dict: dict) -> str | None:
"""Extract the postal code (ZIP) from a willhaben ad."""
attrs = _parse_attributes(ad_dict)
# Try multiple field names that willhaben might use
for key in ("LOCATION_ZIP", "LocationZip", "postalcode"):
val = attrs.get(key) or attrs.get(f"{key}_String")
if val:
return str(val).strip()
return None
```
### 3. Add postcode filter check in `main.py` scheduler loop
```python
async def _check_postcode_filter(
ad_dict: dict,
kw_row: dict
) -> bool:
"""Return True if the ad passes the postcode filter."""
postcodes = kw_row.get("allowed_postcodes") # list or None
if not postcodes:
return True # no filter active
from notifier import _extract_postcode # or wherever it lives
ad_zip = _extract_postcode(ad_dict)
if not ad_zip:
logger.debug("No postcode found in ad, skipping")
return False
# Normalize: willhaben returns "1010" as string, we store same way
return ad_zip in postcodes
# In the scheduler loop (after price check):
for ad_dict in ads_raw:
if not await _check_price_filters(ad_dict, kw_row):
continue
if not await _check_postcode_filter(ad_dict, kw_row):
continue
# ... rest of processing
```
### 4. Add bot commands in `bot.py`
```python
async def cmd_set_postcode(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Set allowed postcodes for a keyword."""
if len(context.args) < 2:
await update.message.reply_text("Usage: /postcode <keyword> <p1,p2,p3>")
return
kw_name = context.args[0]
postcode_strs = [pc.strip() for pc in context.args[1].split(",")]
# Validate format (4-digit Austrian postcodes)
invalid = [pc for pc in postcode_strs if not re.match(r"^\d{3,5}$", pc)]
if invalid:
await update.message.reply_text(
f"Invalid postcode(s): {', '.join(invalid)}. "
"Use 4-digit format like 1010, 8010."
)
return
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
kw_id = await pool.fetchval(
"""SELECT id FROM keywords
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
kw_name.lower(), telegram_id,
)
if not kw_id:
await update.message.reply_text(f"Keyword '{kw_name}' not found.")
return
await pool.execute(
"UPDATE keywords SET allowed_postcodes = $1 WHERE id = $2",
postcode_strs, kw_id, # asyncpg handles text[] natively
)
await update.message.reply_text(
f"✅ Keyword '{kw_name}': postcodes set to {', '.join(postcode_strs)}"
)
async def cmd_clear_postcode(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Remove postcode filter for a keyword."""
if len(context.args) < 1:
await update.message.reply_text("Usage: /clear_postcode <keyword>")
return
# ... same pattern, set allowed_postcodes = NULL
```
Register handlers:
```python
dp.add_handler(MessageHandler(REGEX(r"^/postcode"), cmd_set_postcode))
dp.add_handler(MessageHandler(REGEX(r"^/clear_postcode"), cmd_clear_postcode))
```
### 5. Update `/keywords` command output
Add postcode info to the listing:
```python
if row["allowed_postcodes"]:
line += f"\n 📍 {', '.join(row['allowed_postcodes'])}"
```
## Acceptance Criteria
- [ ] `/postcode keyword 1010,1020` sets allowed postcodes to `{'1010', '1020'}` for that keyword
- [ ] Ads with ZIP codes NOT in the allowed list are skipped during processing
- [ ] Ads with no ZIP code at all are skipped when a filter is active
- [ ] `/clear_postcode keyword` removes the filter (NULL)
- [ ] Invalid postcodes (non-numeric or wrong length) are rejected by the bot
- [ ] The `/keywords` command shows active postcodes next to each keyword
- [ ] Both price AND postcode filters work correctly together (ad must pass both to be notified)
docs/phase-2/task-price-filters.md
+254
View File
@@ -0,0 +1,254 @@
# Task: Price range filters per keyword
## Description
Currently, every ad that matches a keyword triggers a notification regardless of price. For keywords like "rtx" or "3090", this means users receive alerts for €5 listings (accessories) alongside relevant hardware deals.
This task adds `price_min` and `price_max` columns to the `keywords` table and implements server-side filtering during ad processing. Users set these via bot commands, and ads outside the range are silently skipped without being stored in the `ads` table or triggering notifications.
## Architecture
```
┌───────────────────────────────────────┐
│ keywords table (extended) │
│ │
│ ... │
│ price_min int │
│ price_max int │
│ ... │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ Processing pipeline (in main.py) │
│ │
│ for ad in ads_raw: │
│ price = _extract_price(ad) │
│ if price is None: │
│ continue │
│ if kw.price_min and price < min: │
│ skip (below minimum) │
│ if kw.price_max and price > max: │
│ skip (above maximum) │
│ → process ad normally │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ Bot commands (in bot.py): │
│ │
│ /price_min <keyword> <€amount> │
│ /price_max <keyword> <€amount> │
│ /clear_price <keyword> │
│ /keywords → shows price ranges │
└───────────────────────────────────────┘
```
### Key design decisions
- **Filter during processing, not at DB time** — the filter is applied in Python before inserting into `ads`. This keeps the `ads` table clean (only relevant ads are stored) and avoids extra WHERE clauses on every scrape cycle.
- *Alternative*: Could use a computed column or trigger, but adds complexity for simple numeric comparison.
- **Both filters optional** — NULL = no limit. Users can set only min, only max, both, or neither.
- **Price extraction from ad data**: Use the existing `_extract_price()` function in `notifier.py`. If price is not available (free items), treat as €0 for filtering purposes.
## Implementation Details
### 1. Add migration
In `worker/src/migrations/03-keyword-filters.sql`:
```sql
ALTER TABLE keywords
ADD COLUMN IF NOT EXISTS price_min int,
ADD COLUMN IF NOT EXISTS price_max int;
COMMENT ON COLUMN keywords.price_min IS 'Minimum price in cents (e.g. 5000 = €50). NULL = no limit.';
COMMENT ON COLUMN keywords.price_max IS 'Maximum price in cents (e.g. 500000 = €5000). NULL = no limit.';
```
### 2. Update `main.py` — filter ads by price during processing
In the scheduler's keyword loop, before calling `_process_ad()`:
```python
# After extracting ad info in _process_ad():
async def _check_price_filters(
ad_dict: dict,
kw_row: dict # from keywords table
) -> bool:
"""Return True if the ad passes price filters for this keyword."""
from notifier import _extract_price
price = _extract_price(ad_dict)
if price is None:
# No price found — include it (could be "free" or missing data)
return True
price_min = kw_row.get("price_min")
price_max = kw_row.get("price_max")
if price_min is not None and price < price_min:
logger.debug(
"Price filter skip: ad %s price=%d min=%d",
ad_id, price, price_min
)
return False
if price_max is not None and price > price_max:
logger.debug(
"Price filter skip: ad %s price=%d max=%d",
ad_id, price, price_max
)
return False
return True
# In the scheduler loop:
for kw_row in keywords:
if not kw_row["is_active"]:
continue
ads_raw = await fetch_ads(kw_row["keyword"])
for ad_dict in ads_raw:
# Check price filters BEFORE processing
if not await _check_price_filters(ad_dict, kw_row):
continue
# Existing _process_ad logic continues here...
```
### 3. Add bot commands in `bot.py`
```python
from telegram import Message
from telegram.ext import ContextTypes
async def cmd_set_price_min(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Set minimum price for a keyword."""
if len(context.args) < 2:
await update.message.reply_text("Usage: /price_min <keyword> <amount_in_euro>")
return
kw_name = context.args[0]
try:
amount_eur = float(context.args[1])
price_cents = int(amount_eur * 100)
except ValueError:
await update.message.reply_text("Invalid amount. Use a number like 50 or 99.99")
return
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
# Find keyword by name for this user
kw_id = await pool.fetchval(
"""SELECT id FROM keywords
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
kw_name.lower(), telegram_id,
)
if not kw_id:
await update.message.reply_text(f"Keyword '{kw_name}' not found for your account.")
return
await pool.execute(
"UPDATE keywords SET price_min = $1 WHERE id = $2",
price_cents, kw_id,
)
await update.message.reply_text(
f"✅ Keyword '{kw_name}': minimum price set to €{amount_eur:.2f}"
)
async def cmd_set_price_max(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Set maximum price for a keyword."""
# Same pattern as cmd_set_price_min but sets price_max
if len(context.args) < 2:
await update.message.reply_text("Usage: /price_max <keyword> <amount_in_euro>")
return
kw_name = context.args[0]
try:
amount_eur = float(context.args[1])
price_cents = int(amount_eur * 100)
except ValueError:
await update.message.reply_text("Invalid amount. Use a number like 50 or 99.99")
return
telegram_id = str(update.effective_user.id)
from db import get_pool
pool = await get_pool()
kw_id = await pool.fetchval(
"""SELECT id FROM keywords
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
kw_name.lower(), telegram_id,
)
if not kw_id:
await update.message.reply_text(f"Keyword '{kw_name}' not found.")
return
await pool.execute(
"UPDATE keywords SET price_max = $1 WHERE id = $2",
price_cents, kw_id,
)
await update.message.reply_text(
f"✅ Keyword '{kw_name}': maximum price set to €{amount_eur:.2f}"
)
async def cmd_clear_price(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Clear price filters for a keyword."""
if len(context.args) < 1:
await update.message.reply_text("Usage: /clear_price <keyword>")
return
# ... similar pattern, set both to NULL
```
Register handlers in `register_handlers()`:
```python
dp.add_handler(MessageHandler(
REGEX(r"^/price_min"), cmd_set_price_min))
dp.add_handler(MessageHandler(
REGEX(r"^/price_max"), cmd_set_price_max))
dp.add_handler(MessageHandler(
REGEX(r"^/clear_price"), cmd_clear_price))
```
### 4. Update `/keywords` command output to show price ranges
In the existing `/keywords` handler in `bot.py`, add:
```python
price_info = ""
if row["price_min"] is not None:
price_info += f"min €{row['price_min']/100:.2f} "
if row["price_max"] is not None:
price_info += f"max €{row['price_max']/100:.2f}"
if price_info:
line += f"\n 💰 {price_info.strip()}"
```
## Acceptance Criteria
- [ ] `/price_min keyword 50` sets the minimum price to €50.00 (stored as 5000 cents) for that keyword
- [ ] `/price_max keyword 1000` sets the maximum price to €1000.00
- [ ] Ads with price below `price_min` are skipped during processing and not inserted into `ads` table
- [ ] Ads with price above `price_max` are skipped during processing
- [ ] `/clear_price keyword` removes both limits (sets to NULL)
- [ ] Keywords without price filters continue to work as before (no regression)
- [ ] The `/keywords` command displays the active price range for each keyword
- [ ] Non-admin users can only modify their own keywords' price filters
docs/phase-3/Phase.md
+93
View File
@@ -0,0 +1,93 @@
# Phase 3 — Scalability & Advanced Features
## Scope
This phase introduces **structural improvements** that make the project maintainable, extensible, and testable. Currently, the entire system is a single async Python process with no tests and no CI/CD pipeline. After this phase:
- Automated tests provide confidence for every change (≥80% coverage)
- CI/CD pipeline runs on every push to validate code quality
- Multi-marketplace architecture enables adding new sources without modifying core logic
## Architecture
```
┌──────────────────────────────────────────────────────┐
│ Project Structure (post-Phase-3) │
│ │
│ willhaben-tracker/ │
│ ├── worker/ │
│ │ ├── src/ │
│ │ │ ├── main.py (entry point, scheduler) │
│ │ │ ├── db.py (asyncpg pool mgmt) │
│ │ │ ├── bot.py (Telegram handlers) │
│ │ │ ├── notifier.py (message sending) │
│ │ │ ├── scraper.py (base scraper class) │
│ │ │ ├── scrapers/ │
│ │ │ │ ├── __init__.py │
│ │ │ │ ├── willhaben.py (willhaben-specific) │
│ │ │ │ └── base.py (abstract base class) │
│ │ │ ├── health.py (healthcheck endpoint) │
│ │ │ └── migrate.py (migration runner) │
│ │ ├── tests/ │
│ │ │ ├── conftest.py │
│ │ │ ├── test_scraper.py │
│ │ │ ├── test_notifier.py │
│ │ │ └── ... │
│ │ ├── Dockerfile │
│ │ └── requirements.txt │
│ ├── .github/ │
│ │ └── workflows/ │
│ │ └── ci.yml (pytest + flake8 + coverage) │
│ ├── pyproject.toml (coverage config, tools) │
│ └── docker-compose.yml │
└──────────────────────────────────────────────────────┘
Multi-marketplace abstraction:
ScraperBase (abstract):
- async fetch_ads(keyword) → list[dict]
- async parse_response(html/json) → list[dict]
- normalize_ad(raw) → dict with standard keys
WillhabenScraper(ScraperBase):
- implements willhaben-specific URL, headers, parsing
Future: KleinAnzeigenScraper, MobileScraper, ...
CI/CD Pipeline (.github/workflows/ci.yml):
on: push to main, feat/*; pull_request
jobs:
lint-and-test:
└─ python 3.12
├─ flake8 (linting)
├─ pytest --cov=src tests/ (unit + integration tests)
└─ coverage >= 80% (fail if not met)
Tests Structure:
Unit tests:
- test_scraper_pagination() — verify pagination logic with mock responses
- test_price_filters() — verify filter functions
- test_notification_retry() — verify retry queue behavior
Integration tests:
- Test against real willhaben API (rate-limited, cached)
- PostgreSQL test container via docker-compose
```
## Tasks
| Task | File | Description |
|------|------|-------------|
| Multi-marketplace abstraction layer | [task-multi-marketplace.md](./task-multi-marketplace.md) | Refactor `scraper.py` into a base class + per-marketplace implementations. Introduces a standard ad schema and factory for registering new sources. |
| Test suite with pytest (≥80% coverage) | [task-testing-pytest.md](./task-testing-pytest.md) | Add comprehensive unit tests covering scraper parsing, notification logic, price/postcode filters, retry queue, and scheduler flow. Configure coverage thresholds. |
## General Acceptance Criteria
- [ ] CI pipeline runs on every push to `main` and feature branches — fails if lint or coverage checks are not met
- [ ] Code coverage is ≥80% across all source files in `worker/src/`
- [ ] Multi-marketplace abstraction works — adding a new marketplace requires only creating one file under `scrapers/` with no changes to core logic
- [ ] All existing functionality (willhaben scraping, notifications) continues to work after refactoring
- [ ] The `/health` endpoint exposes test results or coverage stats (optional enhancement)
docs/phase-3/task-multi-marketplace.md
+363
View File
@@ -0,0 +1,363 @@
# Task: Multi-marketplace abstraction layer
## Description
Currently, `scraper.py` is tightly coupled to willhaben's API format and URL. Adding a second marketplace (e.g., Kleinanzeigen, Facebook Marketplace) would require extensive refactoring of the core logic — duplicating pagination, error handling, and notification code with subtle differences per source.
This task introduces an **abstract base class** for scrapers and a **standardized ad schema**, making it trivial to add new marketplaces by implementing only marketplace-specific parsing logic.
## Architecture
```
┌───────────────────────────────────────┐
│ scraper.py (module) │
│ │
│ ┌───────────────────────────────────┐│
│ │ ScraperBase (ABC) ││
│ │ ││
│ │ Properties: ││
│ │ name str ││
│ │ base_url str ││
│ │ max_pages int ││
│ │ ││
│ │ Abstract methods: ││
│ │ build_query(url, params) → URL ││
│ │ parse_page(html/json) → list ││
│ │ normalize_ad(raw) → dict ││
│ │ ││
│ │ Concrete methods (shared): ││
│ │ fetch_ads(keyword, cursor) ││
│ │ _fetch_with_retry(url) ││
│ └───────────────────────────────────┘│
└──────┬────────────────────────────────┘
│ inherits
▼
┌───────────────────────────────────────┐
│ scrapers/willhaben.py │
│ │
│ class WillhabenScraper(ScraperBase): │
│ name = "willhaben" ││
│ base_url = ".../api/v1/ad-search" ││
│ build_query() → willhaben URL ││
│ parse_page(json) → list ││
│ normalize_ad(raw) → standard dict ││
└───────────────────────────────────────┘
Standard ad schema (dict):
{
"id": str, # marketplace-specific ID
"marketplace": str, # e.g. "willhaben"
"title": str, # ad title
"price": int | None, # price in cents
"currency": str, # e.g. "EUR"
"url": str, # full URL to the ad page
"published_at": datetime | None,
"location": { │
"city": str, │
"postcode": str | None, │
"region": str | None │
}, │
"attributes": dict # marketplace-specific extras
}
Scheduler (in main.py):
scrapers: list[ScraperBase] = [
WillhabenScraper(),
KleinanzeigenScraper(), # future
]
for scraper in scrapers:
ads_raw, total_hits = await scraper.fetch_ads(keyword)
# ... process with same pipeline (filters, notifications)
```
### Key design decisions
- **Abstract base class** defines the contract. Concrete scrapers only implement what's different per marketplace — URL building, response parsing, and field normalization.
- *Alternative*: Could use a plugin architecture with entry_points, but that adds significant complexity for what is currently expected to be ≤3 marketplaces.
- **Standardized output schema** ensures the downstream pipeline (filters, notifications) works identically regardless of source. Marketplace-specific fields are stored in `attributes`.
- **Pagination logic lives in the base class**. Most marketplaces use offset/limit pagination; the abstract method handles this generically. Special cases override `_fetch_page()`.
## Implementation Details
### 1. Create `worker/src/scrapers/__init__.py`
```python
from .willhaben import WillhabenScraper
__all__ = ["WillhabenScraper"]
def get_scriper_by_name(name: str) -> "ScraperBase":
"""Factory function to instantiate scrapers by name."""
registry = {
"willhaben": WillhabenScraper,
}
cls = registry.get(name.lower())
if not cls:
raise ValueError(f"Unknown marketplace: {name}")
return cls()
```
### 2. Create `worker/src/scrapers/base.py` (abstract base class)
```python
import abc
import asyncio
import logging
from datetime import datetime, timezone
from typing import Any
logger = logging.getLogger(__name__)
class ScraperBase(abc.ABC):
"""Abstract base class for marketplace scrapers."""
name: str = "unknown"
base_url: str = ""
max_pages: int = 2
@abc.abstractmethod
def build_query(self, keyword: str, offset: int) -> str:
"""Build the full API URL/endpoint for a keyword + offset."""
...
@abc.abstractmethod
def parse_page(self, response_content: Any) -> list[dict]:
"""Parse raw response into list of ad dicts (marketplace-specific)."""
...
@abc.abstractmethod
def normalize_ad(self, raw_ad: dict) -> dict:
"""Convert marketplace-specific format to standard schema."""
...
async def fetch_ads(
self,
keyword: str,
cursor_at: datetime | None = None,
max_pages: int | None = None,
) -> tuple[list[dict], int]:
"""Fetch ads with pagination. Shared implementation."""
pages = max_pages or self.max_pages
all_ads: list[dict] = []
total_hits = 0
from ..scraper import get_client # httpx singleton
client = await get_client()
for page in range(pages):
url = self.build_query(keyword, offset=page * 30)
try:
content = await self._fetch_with_retry(client, url)
except Exception as exc:
logger.warning(
"%s: fetch failed at page %d for '%s': %s",
self.name, page, keyword, exc
)
break
raw_ads = self.parse_page(content)
if not raw_ads:
logger.info("%s: no more ads on page %d for '%s'",
self.name, page, keyword)
break
# Normalize and filter by cursor
normalized = []
for raw in raw_ads:
ad = self.normalize_ad(raw)
if cursor_at and ad["published_at"] and ad["published_at"] <= cursor_at:
continue
normalized.append(ad)
all_ads.extend(normalized)
# Politeness delay
if page < pages - 1 and normalized:
await asyncio.sleep(1.0)
return all_ads, total_hits
async def _fetch_with_retry(
self,
client: Any,
url: str,
max_retries: int = 3,
) -> Any:
"""Generic retry wrapper for HTTP fetches."""
import httpx
for attempt in range(max_retries):
try:
resp = await client.get(url)
resp.raise_for_status()
return resp.json() if "application/json" in (resp.headers.get("content-type") or "") else resp.text
except httpx.ConnectError as exc:
logger.warning("%s: transport error attempt %d: %s",
self.name, attempt + 1, exc)
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
else:
raise
@property
def headers(self) -> dict[str, str]:
"""HTTP headers for requests. Override per marketplace."""
return {}
```
### 3. Create `worker/src/scrapers/willhaben.py` (refactor existing logic)
Move the current willhaben-specific code from `scraper.py` into this implementation:
```python
import logging
from datetime import datetime, timezone
from typing import Any
from .base import ScraperBase
logger = logging.getLogger(__name__)
class WillhabenScraper(ScraperBase):
name = "willhaben"
base_url = "https://api.willhaben.at/external/api/v1/ad-search"
@property
def headers(self) -> dict[str, str]:
return {
"Accept": "application/json",
"User-Agent": "Mozilla/5.0 (compatible; WillhabenTracker/1.0)",
}
def build_query(self, keyword: str, offset: int = 0) -> str:
"""Build willhaben API URL with keyword + pagination."""
import urllib.parse
params = {
"keyword": keyword,
"rows": 30,
"sort": 1, # newest first
"offset": offset,
}
return f"{self.base_url}?{urllib.parse.urlencode(params)}"
def parse_page(self, response_content: dict) -> list[dict]:
"""Parse willhaben JSON response into raw ad dicts."""
ads_list = (response_content.get("advertSummaryList") or {}).get(
"advertSummary", []
)
total_hits = int(response_content.get("rowsFound", 0))
return ads_list
def normalize_ad(self, raw_ad: dict) -> dict:
"""Convert willhaben ad format to standard schema."""
# Extract attributes from the nested format
attrs = self._parse_attributes(raw_ad)
# Extract ID
ad_id_raw = raw_ad.get("id", "")
# Extract title (handle various formats)
title_raw = raw_ad.get("title") or raw_ad.get("Title", {})
title = title_raw.get("Value", title_raw) if isinstance(title_raw, dict) else str(title_raw or "")
# Extract price
price_str = attrs.get("PRICE_String") or attrs.get("priceString", "")
try:
price_cents = int(float(price_str.replace(".", ""))) if price_str else None
except (ValueError, TypeError):
price_cents = None
# Extract published date
pub_str = attrs.get("PUBLISHED_String") or attrs.get("publishedString", "")
published_at = None
try:
published_at = datetime.fromisoformat(pub_str.replace("Z", "+00:00"))
except (ValueError, TypeError):
pass
# Extract location
city = attrs.get("LOCATION_CityName") or ""
postcode = attrs.get("LOCATION_ZIP") or ""
region = attrs.get("LOCATION_Region") or ""
return {
"id": ad_id_raw,
"marketplace": self.name,
"title": title.strip(),
"price": price_cents,
"currency": "EUR",
"url": raw_ad.get("linkUrl", ""),
"published_at": published_at,
"location": {
"city": city,
"postcode": postcode if postcode else None,
"region": region if region else None,
},
"attributes": attrs, # preserve marketplace-specific fields
}
@staticmethod
def _parse_attributes(ad_dict: dict) -> dict:
"""Parse willhaben's nested attribute format into flat dict."""
result = {}
for attr_group in ad_dict.get("attributes", []):
if not isinstance(attr_group, dict):
continue
group_name = attr_group.get("name") or ""
for item in attr_group.get("items", []):
key = f"{group_name}_{item['name']}" if group_name else item["name"]
result[key] = item.get("valueString", "")
return result
```
### 4. Update `main.py` scheduler to use the new scraper factory
Replace direct calls to `fetch_ads(keyword)` with:
```python
from scrapers import get_scriper_by_name
# At startup:
scrapers_config = os.getenv("SCRAPERS", "willhaben").split(",")
active_scrapers = [get_scriper_by_name(s) for s in scrapers_config]
# In scheduler loop:
for scraper in active_scrapers:
ads_raw, total_hits = await scraper.fetch_ads(keyword, cursor_at=cursor)
# ... process with existing pipeline (price filter, postcode filter, etc.)
```
### 5. Add `.env.example` configuration for scrapers
```bash
# Marketplace sources to scrape (comma-separated)
SCRAPERS=willhaben
```
## Acceptance Criteria
- [ ] `WillhabenScraper` produces identical output to the current `scraper.py` implementation (no regression in ad extraction)
- [ ] Adding a new marketplace requires only: creating one file under `scrapers/`, registering it in `__init__.py`, and listing it in SCRAPERS env var
- [ ] The standardized ad schema includes all fields needed by the downstream pipeline (price, postcode, published_at, URL)
- [ ] Pagination logic works correctly through the base class for willhaben
- [ ] Error handling (retries, timeouts) continues to work with the new abstraction
- [ ] All existing bot commands and notifications function identically after refactoring
docs/phase-3/task-testing-pytest.md
+777
View File
@@ -0,0 +1,777 @@
# Task: Test suite with pytest (≥80% coverage)
## Description
The project currently has **zero automated tests**. Every change is verified manually by watching logs or sending test messages to the bot. This makes refactoring risky and prevents CI/CD automation.
This task introduces a comprehensive pytest test suite covering all critical paths: scraper parsing, notification logic, price/postcode filters, retry queue behavior, and scheduler flow. Coverage threshold is set to 80% minimum.
## Architecture
```
┌───────────────────────────────────────┐
│ Tests structure │
│ │
│ worker/tests/ │
│ ├── conftest.py │
│ │ (fixtures: mock_pool, mock_bot)│
│ ├── test_scraper.py │
│ │ (parsing, pagination) │
│ ├── test_notifier.py │
│ │ (send, retry queue, digest) │
│ ├── test_filters.py │
│ │ (price, postcode, mute hours) │
│ ├── test_scheduler.py │
│ │ (cycle flow, shutdown) │
│ └── test_health.py │
│ (healthcheck endpoint) │
│ │
└──────────┬────────────────────────────┘
│
▼
┌───────────────────────────────────────┐
│ CI/CD Pipeline (.github/workflows/ci.yml)
│ │
│ on: push to main, feat/*; pull_request│
│ │
│ jobs: │
│ lint-and-test: │
│ └─ python 3.12 │
│ ├─ flake8 (error-only) │
│ ├─ pytest --cov=src tests/ │
│ └─ coverage >= 80% │
└───────────────────────────────────────┘
Test strategy:
Unit tests (majority):
- Isolate each function/method with mocks
- Test edge cases: missing fields, NULL prices, empty results
- Fast (<1s per test)
Integration tests (minority):
- Real HTTP to willhaben API (cached responses only)
- PostgreSQL test container via docker-compose
Mocking strategy:
- Telegram Bot: mock `bot.send_message()` → verify call count + content
- Asyncpg pool: mock fetchval/fetch/execute → return canned data
- httpx client: use pytest-httpx to intercept and return fixtures
```
### Key design decisions
- **pytest over unittest**. Cleaner syntax, better fixture system, easier async support.
- **pytest-asyncio** for testing async functions directly without wrapping in `loop.run_until_complete()`.
- Adding as test dependency: `pip install pytest pytest-asyncio pytest-cov httpx[socks]`
- **Coverage threshold at 80%** enforced via `pyproject.toml`. Failures are actionable (which files/functions need coverage).
- **Snapshot testing for HTTP responses**. Save real willhaben API responses as JSON fixtures to avoid live network calls in CI.
## Implementation Details
### 1. Add test dependencies and configuration
In `worker/requirements-test.txt`:
```txt
pytest>=8.0
pytest-asyncio>=0.24
pytest-cov>=6.0
aioresponses>=0.7 # mock aiohttp responses for health endpoint tests
```
Create `pyproject.toml` in the project root:
```toml
[tool.pytest.ini_options]
testpaths = ["worker/tests"]
asyncio_mode = "auto"
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
[tool.coverage.run]
source = ["worker/src"]
omit = [
"*/tests/*",
"*/migrate.py", # migration runner — tested manually against real DB
]
[tool.coverage.report]
fail_under = 80.0
show_missing = true
exclude_lines = [
"pragma: no cover",
"if TYPE_CHECKING:",
]
```
### 2. Create `worker/tests/conftest.py` (fixtures)
```python
import pytest
from unittest.mock import AsyncMock, MagicMock
@pytest.fixture
def mock_pool():
"""Mock asyncpg.Pool with fetchval/fetch/execute."""
pool = AsyncMock()
pool.fetchval = AsyncMock(return_value=None)
pool.fetch = AsyncMock(return_value=[])
pool.execute = AsyncMock(return_value="DONE 1")
return pool
@pytest.fixture
def mock_bot():
"""Mock Telegram Bot instance."""
bot = MagicMock()
bot.send_message = AsyncMock(return_value=True)
bot.get_me = AsyncMock(return_value={"id": "bot_user", "is_bot": True})
return bot
@pytest.fixture
def sample_willhaben_response():
"""Realistic willhaben API response for testing."""
return {
"rowsFound": 45,
"advertSummaryList": {
"advertSummary": [
{
"id": "123456789",
"title": {"Value": "RTX 3090 Gaming X - Top Zustand"},
"linkUrl": "https://www.willhaben.at/iad/markt/123456789-rtx-3090-gaming-x",
"attributes": [
{
"name": "PRICE",
"items": [{"name": "priceString", "valueString": "750.00"}]
},
{
"name": "PUBLISHED",
"items": [{"name": "publishedString", "valueString": "2026-07-04T12:30:00+02:00"}]
},
{
"name": "LOCATION",
"items": [
{"name": "CityName", "valueString": "Wien"},
{"name": "ZIP", "valueString": "1010"},
]
}
]
},
]
}
}
@pytest.fixture
def sample_ad_normalized():
"""Expected normalized ad dict from willhaben response."""
return {
"id": "123456789",
"marketplace": "willhaben",
"title": "RTX 3090 Gaming X - Top Zustand",
"price": 75000, # in cents
"currency": "EUR",
"url": "https://www.willhaben.at/iad/markt/123456789-rtx-3090-gaming-x",
"published_at": ..., # will be datetime object
"location": {
"city": "Wien",
"postcode": "1010",
"region": None,
},
"attributes": {...},
}
@pytest.fixture
def sample_keyword_row():
"""Sample keyword DB row."""
return {
"id": "kw-uuid-here",
"keyword_name": "rtx 3090",
"telegram_id": "298181113",
"is_active": True,
"price_min": 50000, # €500 minimum
"price_max": 1000000, # €10000 maximum
"allowed_postcodes": ["1010", "1020"],
"last_scraped_at": None,
"ads_cursor": None,
}
```
### 3. Create `worker/tests/test_scraper.py`
```python
import pytest
from unittest.mock import AsyncMock, patch
from scrapers.willhaben import WillhabenScraper
from datetime import datetime, timezone
class TestWillhabenScraper:
def test_build_query(self):
scraper = WillhabenScraper()
url = scraper.build_query("rtx 3090", offset=30)
assert "keyword=rtx+3090" in url
assert "offset=30" in url
assert "rows=30" in url
def test_build_query_default_offset(self):
scraper = WillhabenScraper()
url = scraper.build_query("gtx 1660")
assert "offset=0" in url
def test_parse_page_empty_response(self, sample_willhaben_response):
scraper = WillhabenScraper()
# Empty response
empty = {"rowsFound": 0, "advertSummaryList": {"advertSummary": []}}
result = scraper.parse_page(empty)
assert result == []
def test_normalize_ad(self, sample_willhaben_response):
scraper = WillhabenScraper()
raw_ad = sample_willhaben_response["advertSummaryList"]["advertSummary"][0]
ad = scraper.normalize_ad(raw_ad)
assert ad["id"] == "123456789"
assert ad["marketplace"] == "willhaben"
assert ad["price"] == 75000 # cents
assert ad["location"]["postcode"] == "1010"
def test_normalize_ad_missing_price(self):
scraper = WillhabenScraper()
raw_ad = {
"id": "no-price",
"title": {"Value": "Free RTX"},
"linkUrl": "https://example.com",
"attributes": [], # no price
}
ad = scraper.normalize_ad(raw_ad)
assert ad["price"] is None
@pytest.mark.asyncio
async def test_fetch_ads_pagination(self):
scraper = WillhabenScraper()
with patch.object(scraper, '_fetch_with_retry') as mock_fetch:
# Simulate 2 pages of results
page1 = {
"rowsFound": 45,
"advertSummaryList": {"advertSummary": [
{"id": f"ad{i}", "title": {"Value": f"Ad {i}"},
"linkUrl": f"https://example.com/{i}",
"attributes": [{"name": "PUBLISHED", "items": [
{"name": "publishedString",
"valueString": "2026-07-04T15:30:00+02:00"}]}]},
] for i in range(3)}]
}
page2 = {
"rowsFound": 45,
"advertSummaryList": {"advertSummary": [
{"id": f"ad{i}", "title": {"Value": f"Ad {i}"},
"linkUrl": f"https://example.com/{i}",
"attributes": [{"name": "PUBLISHED", "items": [
{"name": "publishedString",
"valueString": "2026-07-04T15:30:00+02:00"}]}]},
] for i in range(3, 6)}]
}
mock_fetch.side_effect = [page1, page2]
ads, total = await scraper.fetch_ads("test keyword", max_pages=2)
assert len(ads) == 6 # 3 from each page
class TestScraperBase:
def test_headers_default(self):
scraper = WillhabenScraper()
headers = scraper.headers
assert "Accept" in headers
assert "User-Agent" in headers
@pytest.mark.asyncio
async def test_fetch_with_retry_exhausts_retries(self):
import httpx
scraper = WillhabenScraper()
client = AsyncMock()
client.get = AsyncMock(side_effect=httpx.ConnectError("connection refused"))
with pytest.raises(httpx.ConnectError):
await scraper._fetch_with_retry(client, "http://example.com", max_retries=2)
```
### 4. Create `worker/tests/test_filters.py`
```python
import pytest
from unittest.mock import AsyncMock
class TestPriceFilters:
async def test_pass_when_no_min(self):
"""Ad passes when no price minimum is set."""
# Import the actual function being tested
from notifier import _extract_price # or wherever it lives
kw_row = {"price_min": None, "price_max": None}
ad_dict = {
"attributes": [{"name": "PRICE", "items": [
{"name": "priceString", "valueString": "50.00"}]}]
}
# The check function (to be implemented in main.py)
from ..main import _check_price_filters # actual implementation
result = await _check_price_filters(ad_dict, kw_row)
assert result is True
async def test_pass_when_no_max(self):
kw_row = {"price_min": 1000, "price_max": None}
ad_dict = {
"attributes": [{"name": "PRICE", "items": [
{"name": "priceString", "valueString": "100.00"}]}]
}
from ..main import _check_price_filters
result = await _check_price_filters(ad_dict, kw_row)
assert result is True
async def test_fail_below_min(self):
kw_row = {"price_min": 50000, "price_max": None} # €500 min
ad_dict = {
"attributes": [{"name": "PRICE", "items": [
{"name": "priceString", "valueString": "10.00"}]}] # €10 — too cheap
}
from ..main import _check_price_filters
result = await _check_price_filters(ad_dict, kw_row)
assert result is False
async def test_fail_above_max(self):
kw_row = {"price_min": None, "price_max": 1000} # €10 max
ad_dict = {
"attributes": [{"name": "PRICE", "items": [
{"name": "priceString", "valueString": "500.00"}]}] # €500 — too expensive
}
from ..main import _check_price_filters
result = await _check_price_filters(ad_dict, kw_row)
assert result is False
class TestPostcodeFilters:
async def test_pass_when_no_filter(self):
kw_row = {"allowed_postcodes": None}
ad_dict = {
"attributes": [{"name": "LOCATION", "items": [
{"name": "ZIP", "valueString": "1010"}]}]
}
from ..main import _check_postcode_filter
result = await _check_postcode_filter(ad_dict, kw_row)
assert result is True
async def test_pass_when_matching(self):
kw_row = {"allowed_postcodes": ["1010", "1020"]}
ad_dict = {
"attributes": [{"name": "LOCATION", "items": [
{"name": "ZIP", "valueString": "1010"}]}]
}
from ..main import _check_postcode_filter
result = await _check_postcode_filter(ad_dict, kw_row)
assert result is True
async def test_fail_when_not_matching(self):
kw_row = {"allowed_postcodes": ["1010", "1020"]}
ad_dict = {
"attributes": [{"name": "LOCATION", "items": [
{"name": "ZIP", "valueString": "8010"}]}] # Graz — not in allowed list
}
from ..main import _check_postcode_filter
result = await _check_postcode_filter(ad_dict, kw_row)
assert result is False
async def test_fail_when_no_postcode_in_ad(self):
kw_row = {"allowed_postcodes": ["1010", "1020"]}
ad_dict = {
"attributes": [] # no location info
}
from ..main import _check_postcode_filter
result = await _check_postcode_filter(ad_dict, kw_row)
assert result is False
class TestMuteHours:
async def test_no_mute_when_not_configured(self, mock_pool):
mock_pool.fetchrow.return_value = None
from ..notifier import _is_in_mute_hours
result = await _is_in_mute_hours("298181113", mock_pool)
assert result is False
async def test_no_mute_outside_window(self, mock_pool):
from datetime import time
mock_pool.fetchrow.return_value = {
"mute_start": time(22, 0), # 10 PM
"mute_end": time(7, 0), # 7 AM
}
# Mock current time to noon UTC (outside mute window)
with patch("notifier.datetime") as mock_dt:
from datetime import datetime, timezone
mock_dt.now.return_value = datetime(2026, 1, 1, 12, 0, tzinfo=timezone.utc)
from ..notifier import _is_in_mute_hours
result = await _is_in_mute_hours("298181113", mock_pool)
assert result is False
async def test_muted_during_window(self, mock_pool):
from datetime import time
mock_pool.fetchrow.return_value = {
"mute_start": time(22, 0),
"mute_end": time(7, 0),
}
with patch("notifier.datetime") as mock_dt:
from datetime import datetime, timezone
mock_dt.now.return_value = datetime(2026, 1, 1, 3, 0, tzinfo=timezone.utc)
from ..notifier import _is_in_mute_hours
result = await _is_in_mute_hours("298181113", mock_pool)
assert result is True
class TestNotificationQueue:
async def test_enqueue_on_failure(self, mock_pool):
from ..notifier import _enqueue_retry
await _enqueue_retry(
ad_id="ad-uuid-here",
telegram_id="298181113",
message_text="Test notification",
notif_type="new",
error_msg="Telegram API timeout",
)
mock_pool.execute.assert_called_once()
async def test_no_duplicate_enqueue(self, mock_pool):
"""Same ad+user should not create duplicate queue entries."""
mock_pool.fetchval.return_value = "already-exists-uuid"
from ..notifier import _enqueue_retry
await _enqueue_retry(
ad_id="ad-uuid-here",
telegram_id="298181113",
message_text="Test",
notif_type="new",
error_msg="timeout",
)
# fetchval called (to check), but execute NOT called (no insert)
mock_pool.fetchval.assert_called_once()
mock_pool.execute.assert_not_called()
```
### 5. Create `worker/tests/test_scheduler.py`
```python
import pytest
from unittest.mock import AsyncMock, patch
class TestSchedulerFlow:
@pytest.mark.asyncio
async def test_process_notification_queue_retries_pending(self, mock_pool):
"""Pending items should be retried if backoff period has elapsed."""
from datetime import datetime, timezone
mock_pool.fetch.return_value = [
{
"id": "queue-item-1",
"ad_id": "ad-uuid",
"telegram_id": "298181113",
"message_text": "Retry this ad",
"type": "new",
"attempts": 0,
"max_attempts": 5,
"last_error": "timeout",
"updated_at": datetime(2026, 7, 4, 10, 0, tzinfo=timezone.utc),
},
]
with patch("main.get_application_bot") as mock_get_bot:
mock_bot = AsyncMock()
mock_bot.send_message = AsyncMock()
mock_get_bot.return_value = mock_bot
from ..main import process_notification_queue
result = await process_notification_queue()
assert result == 1
@pytest.mark.asyncio
async def test_process_notification_queue_dead_after_max_attempts(self, mock_pool):
"""Items exceeding max_attempts should be marked as dead."""
from datetime import datetime, timezone
mock_pool.fetch.return_value = [
{
"id": "queue-item-2",
"ad_id": "ad-uuid",
"telegram_id": "298181113",
"message_text": "Will fail again",
"type": "new",
"attempts": 5, # already at max
"max_attempts": 5,
"last_error": "user blocked bot",
"updated_at": datetime(2026, 7, 4, 10, 0, tzinfo=timezone.utc),
},
]
with patch("main.get_application_bot") as mock_get_bot:
import telegram.error
mock_bot = AsyncMock()
mock_bot.send_message = AsyncMock(
side_effect=telegram.error.TelegramError("blocked")
)
mock_get_bot.return_value = mock_bot
from ..main import process_notification_queue
await process_notification_queue()
# Should have updated to 'dead' status
calls = [c[0] for c in mock_pool.execute.call_args_list]
assert any("status = 'dead'" in str(c) or "status=$2" in str(c)
for c in calls), "Item should be marked as dead"
class TestDigestFlushing:
@pytest.mark.asyncio
async def test_flush_digest_buffers(self, mock_pool):
"""Buffered items older than interval should be flushed."""
from datetime import datetime, timezone
mock_pool.fetch.side_effect = [
# First fetch: get digest-enabled users
[{"telegram_id": "298181113", "digest_interval": 60}],
# Second fetch: get buffered items
[
{"id": "buf-1", "keyword": "rtx 3090",
"title": "Ad 1", "price": 75000,
"url": "https://...", "ad_id": "ad-uuid"},
{"id": "buf-2", "keyword": "rtx 3090",
"title": "Ad 2", "price": 68000,
"url": "https://...", "ad_id": "ad-uuid-2"},
],
]
with patch("main.get_application_bot") as mock_get_bot:
mock_bot = AsyncMock()
mock_bot.send_message = AsyncMock()
mock_get_bot.return_value = mock_bot
from ..main import flush_digest_buffers
result = await flush_digest_buffers()
assert result == 1 # one digest sent
@pytest.mark.asyncio
async def test_flush_empty_buffer(self, mock_pool):
"""No buffered items should result in no action."""
mock_pool.fetch.side_effect = [
[{"telegram_id": "298181113", "digest_interval": 60}],
[], # empty buffer
]
from ..main import flush_digest_buffers
result = await flush_digest_buffers()
assert result == 0
class TestGracefulShutdown:
@pytest.mark.asyncio
async def test_cleanup_stops_scheduler_and_closes_pool(self):
"""Cleanup should cancel scheduler, stop bot, close DB pool."""
from telegram.ext import Application
app = AsyncMock(spec=Application)
app.updater.running = True
with patch("main._scheduler_task") as mock_task:
mock_task.done.return_value = False
from ..main import cleanup
await cleanup(app)
mock_task.cancel.assert_called_once()
app.updater.stop_polling.assert_called_once()
```
### 6. Create `worker/tests/test_health.py`
```python
import pytest
@pytest.mark.asyncio
async def test_health_endpoint_ok():
"""Health endpoint should return 200 when system is healthy."""
from health import create_health_app
app = create_health_app()
with patch("health.get_pool") as mock_pool_get:
mock_pool = AsyncMock()
mock_pool.fetchval = AsyncMock(return_value=1)
mock_pool_get.return_value = mock_pool
runner = web.AppRunner(app)
await runner.setup()
try:
from aiohttp.test_utils import TestClient, TestServer
client = TestClient(TestServer(runner))
async with client:
resp = await client.get("/health")
assert resp.status == 200
data = await resp.json()
assert data["status"] == "ok"
assert "db_connected" in data
finally:
await runner.cleanup()
@pytest.mark.asyncio
async def test_health_endpoint_unhealthy_db():
"""Health endpoint should return 503 when DB is unreachable."""
from health import create_health_app
app = create_health_app()
with patch("health.get_pool") as mock_pool_get:
mock_pool_get.side_effect = Exception("connection refused")
runner = web.AppRunner(app)
await runner.setup()
try:
from aiohttp.test_utils import TestClient, TestServer
client = TestClient(TestServer(runner))
async with client:
resp = await client.get("/health")
assert resp.status == 503
data = await resp.json()
assert data["status"] == "unhealthy"
finally:
await runner.cleanup()
```
### 7. Create GitHub Actions workflow `.github/workflows/ci.yml`
```yaml
name: CI — Lint & Test
on:
push:
branches: [main, 'feat/*']
pull_request:
branches: [main]
jobs:
lint-and-test:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: |
pip install --upgrade pip
pip install -r worker/requirements.txt
pip install -r worker/requirements-test.txt
- name: Lint with flake8
run: |
flake8 worker/src/ --count --select=E9,F63,F7,F82 --show-source --statistics
flake8 worker/tests/ --count --select=E9,F63,F7,F82 --show-source --statistics
- name: Test with pytest + coverage
run: |
cd worker
python -m pytest tests/ --cov=src --cov-report=term-missing --cov-fail-under=80
build-docker:
runs-on: ubuntu-latest
needs: lint-and-test
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Build Docker image
run: docker compose -f docker-compose.yml build worker
- name: Test healthcheck
run: |
docker compose up -d worker
sleep 5
docker inspect --format='{{.State.Health.Status}}' willhaben-tracker-worker-1 || true
docker compose down
```
## Acceptance Criteria
- [ ] `pytest` runs with 0 failures and ≥80% coverage on all source files
- [ ] GitHub Actions pipeline passes on every push to `main` and feature branches
- [ ] Tests cover: scraper parsing, pagination, price filters, postcode filters, mute hours, notification retry queue, digest flushing, graceful shutdown, healthcheck endpoint
- [ ] Flake8 linting (error-level checks) passes in CI
- [ ] Docker image builds successfully after all tests pass
- [ ] Adding a new test file automatically includes it in the coverage report