docs(plan): add Phase 1, 2, 3 implementation specs
This commit is contained in:
hermes
@@ -0,0 +1,60 @@
|
||||
# Phase 1 — Reliability & Completeness Improvements
|
||||
|
||||
## Scope
|
||||
|
||||
This phase addresses the **three highest-impact reliability gaps** identified in the code review: incomplete ad coverage due to page-limited scraping, inefficient HTTP client usage, and silent notification loss. After this phase, the worker will:
|
||||
|
||||
- Capture a larger window of ads per scrape cycle (no longer limited to 30 newest)
|
||||
- Reuse TCP/TLS connections for willhaben API calls instead of creating one per request
|
||||
- Retry failed Telegram notifications instead of dropping them permanently
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ worker container │
|
||||
│ │
|
||||
│ ┌───────────┐ │
|
||||
│ │ scraper.py│ ← SINGLETON AsyncClient │
|
||||
│ │ │ (connection pool, keepalive) │
|
||||
│ │ │ │
|
||||
│ │ fetch_ads() │
|
||||
│ │ ├─ page 1: rows=30 & published_after=<cursor> │
|
||||
│ │ ├─ page 2: rows=30 & offset=30 │
|
||||
│ │ └─ ... until no new ads or max_pages reached │
|
||||
│ └───────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────┐ ┌──────────────────┐ │
|
||||
│ │ notifier.py │──►│ notification_queue│ │
|
||||
│ │ │ │ table (new) │ │
|
||||
│ │ notify_new() │ │ │ │
|
||||
│ │ notify_drop()│ │ - ad_id │ │
|
||||
│ │ │ │ - telegram_id │ │
|
||||
│ │ if success: │ │ - attempts (0→5) │ │
|
||||
│ │ log_notify │ │ - last_error │ │
|
||||
│ │ if fail: │ │ - status │ │
|
||||
│ │ enqueue! │ └────────┬─────────┘ │
|
||||
│ └──────────────┘ │ │
|
||||
│ ▼ │
|
||||
│ scheduler retries │
|
||||
│ pending items each cycle │
|
||||
└──────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Tasks
|
||||
|
||||
| Task | File | Description |
|
||||
|------|------|-------------|
|
||||
| Pagination in willhaben scraper | [task-scraper-pagination.md](./task-scraper-pagination.md) | Implement cursor-based or offset pagination to fetch more than 30 ads per cycle, tracking the last seen timestamp to avoid duplicates across cycles. |
|
||||
| httpx singleton with connection pool | [task-httpx-singleton.md](./task-httpx-singleton.md) | Replace per-call AsyncClient creation with a module-level singleton using keepalive connections and configurable limits. |
|
||||
| Retry queue for failed notifications | [task-notification-retry-queue.md](./task-notification-retry-queue.md) | Add a `notification_queue` table to persist failed Telegram sends with exponential backoff retries (up to 5 attempts). |
|
||||
|
||||
## General Acceptance Criteria
|
||||
|
||||
- [ ] A single scrape cycle captures at least 90 ads for high-volume keywords (3 pages × 30 rows) instead of the current hard cap of 30
|
||||
- [ ] Duplicate ads between cycles are not re-notified (cursor/offset tracking prevents this)
|
||||
- [ ] HTTP connection reuse reduces willhaben API call latency by ≥40% (measured via logs)
|
||||
- [ ] Failed notifications are retried up to 5 times with exponential backoff (1m, 2m, 4m, 8m, 16m between attempts)
|
||||
- [ ] After 5 failed retries the notification is marked as `dead` and logged — not silently dropped
|
||||
- [ ] The scheduler processes queued notifications at the start of each cycle before scraping new keywords
|
||||
@@ -0,0 +1,157 @@
|
||||
# Task: httpx singleton with connection pool
|
||||
|
||||
## Description
|
||||
|
||||
The current `scraper.fetch_ads()` creates a **new** `httpx.AsyncClient` on every call:
|
||||
|
||||
```python
|
||||
async def fetch_ads(keyword: str):
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
resp = await client.get(_API_URL, ...)
|
||||
```
|
||||
|
||||
This means each scrape cycle incurs the full cost of TCP handshake + TLS negotiation (≈100-300ms per call on a cold connection). For keywords scraped every 5 minutes with multiple pages, this overhead adds up to **seconds of unnecessary latency per cycle**.
|
||||
|
||||
This task replaces the per-call client with a module-level singleton that reuses connections via keepalive.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Current:
|
||||
Cycle 1: create AsyncClient → fetch → close → ~300ms overhead
|
||||
Cycle 2: create AsyncClient → fetch → close → ~300ms overhead
|
||||
Cycle N: ... (repeated forever)
|
||||
|
||||
Target:
|
||||
Module load: create AsyncClient (singleton, keepalive pool)
|
||||
Cycle 1: use client → fetch → ~50ms (warm connection)
|
||||
Cycle 2: use client → fetch → ~50ms (warm connection)
|
||||
Cycle N: ...
|
||||
Shutdown: close client gracefully
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Module-level singleton** (`_client = None`, lazy init). Simpler than dependency injection and works with the existing async context.
|
||||
- **Keepalive connections**: Default `max_keepalive_connections=5` handles concurrent keyword scrapes efficiently.
|
||||
- **Client recreation on error**: If the client is closed or encounters a fatal transport error, it's recreated on the next call. This prevents stale connection issues.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add singleton getter to `scraper.py`
|
||||
|
||||
```python
|
||||
import os
|
||||
|
||||
_client: httpx.AsyncClient | None = None
|
||||
|
||||
|
||||
async def get_client() -> httpx.AsyncClient:
|
||||
"""Return a shared AsyncClient with keepalive connection pool."""
|
||||
global _client
|
||||
|
||||
if _client is None or _client.is_closed:
|
||||
max_conns = int(os.getenv("HTTP_MAX_CONNECTIONS", "10"))
|
||||
max_keepalive = int(os.getenv("HTTP_KEEPALIVE_CONNECTIONS", "5"))
|
||||
|
||||
_client = httpx.AsyncClient(
|
||||
timeout=float(os.getenv("HTTP_TIMEOUT_S", "30.0")),
|
||||
limits=httpx.Limits(
|
||||
max_connections=max_conns,
|
||||
max_keepalive_connections=max_keepalive,
|
||||
keepalive_expiry=60, # seconds
|
||||
),
|
||||
)
|
||||
logger.info(
|
||||
"Created httpx client: max_conns=%d, keepalive=%d",
|
||||
max_conns, max_keepalive,
|
||||
)
|
||||
|
||||
return _client
|
||||
|
||||
|
||||
async def close_client() -> None:
|
||||
"""Close the shared AsyncClient. Call during shutdown."""
|
||||
global _client
|
||||
if _client and not _client.is_closed:
|
||||
await _client.aclose()
|
||||
logger.info("Closed httpx client")
|
||||
_client = None
|
||||
```
|
||||
|
||||
### 2. Update `fetch_ads()` to use the singleton
|
||||
|
||||
**Replace:**
|
||||
```python
|
||||
async def fetch_ads(keyword: str):
|
||||
params = {...}
|
||||
async with httpx.AsyncClient(timeout=30.0) as client:
|
||||
for attempt in range(1, 4):
|
||||
try:
|
||||
resp = await client.get(_API_URL, headers=_HEADERS, params=params)
|
||||
```
|
||||
|
||||
**With:**
|
||||
```python
|
||||
async def fetch_ads(keyword: str):
|
||||
params = {...}
|
||||
client = await get_client()
|
||||
|
||||
for attempt in range(1, 4):
|
||||
try:
|
||||
resp = await client.get(_API_URL, headers=_HEADERS, params=params)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
break
|
||||
except httpx.ConnectError as exc:
|
||||
# Transport error — recreate client on next attempt
|
||||
logger.warning("Transport error on attempt %d: %s", attempt, exc)
|
||||
await close_client() # force recreation
|
||||
if attempt < 3:
|
||||
await asyncio.sleep(2 ** attempt)
|
||||
continue
|
||||
raise
|
||||
except Exception as exc:
|
||||
logger.warning("fetch_ads attempt %d failed: %s", attempt, exc)
|
||||
if attempt < 3:
|
||||
await asyncio.sleep(2 ** attempt)
|
||||
continue
|
||||
raise
|
||||
|
||||
# ... rest unchanged (extract ads_raw, total_hits)
|
||||
```
|
||||
|
||||
### 3. Call `close_client()` during shutdown in `main.py`
|
||||
|
||||
Add to the cleanup function (from Phase 0 task-graceful-shutdown):
|
||||
|
||||
```python
|
||||
async def cleanup(app: Application) -> None:
|
||||
logger.info("Shutting down...")
|
||||
|
||||
# ... existing cleanup steps ...
|
||||
|
||||
# Close HTTP client
|
||||
from scraper import close_client
|
||||
await close_client()
|
||||
|
||||
# ... rest of cleanup (close DB pool, etc.)
|
||||
```
|
||||
|
||||
### 4. Update `.env.example` with new config options
|
||||
|
||||
```bash
|
||||
# HTTP Client Configuration
|
||||
HTTP_MAX_CONNECTIONS=10 # Max concurrent connections to willhaben API
|
||||
HTTP_KEEPALIVE_CONNECTIONS=5 # Connections kept alive in the pool
|
||||
HTTP_TIMEOUT_S=30.0 # Request timeout in seconds
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Only one `httpx.AsyncClient` is created per process lifetime (logged once at startup)
|
||||
- [ ] Subsequent calls to `fetch_ads()` reuse the existing client (no "Created httpx client" log)
|
||||
- [ ] After calling `close_client()`, a new call to `get_client()` creates a fresh client
|
||||
- [ ] Connection keepalive reduces latency for sequential API calls (verifiable via timing in logs)
|
||||
- [ ] Fatal transport errors trigger client recreation without crashing the scheduler
|
||||
- [ ] The client is properly closed during graceful shutdown (no resource warnings)
|
||||
@@ -0,0 +1,301 @@
|
||||
# Task: Retry queue for failed notifications
|
||||
|
||||
## Description
|
||||
|
||||
The current `notifier.notify_new()` and `notify_drop()` call `_send_message()` directly. If the Telegram API returns an error (rate limiting, network hiccup, user deleted the bot), the notification is **silently logged** and never retried:
|
||||
|
||||
```python
|
||||
async def _send_message(bot: Bot, chat_id: int, message_text):
|
||||
try:
|
||||
await bot.send_message(chat_id=chat_id, text=message_text)
|
||||
except TelegramError as e:
|
||||
logger.warning("Failed to send notification ...") # ← notification LOST forever
|
||||
```
|
||||
|
||||
This task introduces a **persistent retry queue** backed by the `notification_queue` table. Failed notifications are stored with an attempt counter and retried on subsequent scheduler cycles with exponential backoff. After 5 failed attempts, they're marked as `dead` and logged — not silently dropped.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ notification_queue table (new) │
|
||||
│ │
|
||||
│ id uuid PK │
|
||||
│ ad_id uuid │
|
||||
│ telegram_id text │
|
||||
│ message_text text │
|
||||
│ type enum('new','drop') │
|
||||
│ attempts int DEFAULT 0 │
|
||||
│ max_attempts int DEFAULT 5 │
|
||||
│ last_error text │
|
||||
│ status enum │
|
||||
│ ('pending','sent', │
|
||||
│ 'failed','dead') │
|
||||
│ created_at timestamptz │
|
||||
│ updated_at timestamptz │
|
||||
│ │
|
||||
│ INDEX: status, attempts (composite) │
|
||||
└───────────┬───────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ notifier.py flow: │
|
||||
│ │
|
||||
│ send_notification(): │
|
||||
│ try: │
|
||||
│ await bot.send_message(...) │
|
||||
│ → log_notify() (as before) │
|
||||
│ except TelegramError as e: │
|
||||
│ INSERT INTO notification_queue │
|
||||
│ (ad_id, telegram_id, ...) │
|
||||
│ VALUES (...) │
|
||||
└───────────┬───────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ scheduler.py / main.py: │
|
||||
│ At the START of each cycle: │
|
||||
│ │
|
||||
│ for item in pending_queue: │
|
||||
│ if attempts < max_attempts: │
|
||||
│ backoff = 2^attempts minutes │
|
||||
│ if now >= updated_at + backoff:│
|
||||
│ try send again │
|
||||
│ success → mark 'sent' │
|
||||
│ fail → increment count │
|
||||
│ elif attempts >= max_attempts: │
|
||||
│ mark as 'dead' │
|
||||
│ log warning │
|
||||
└───────────────────────────────────────┘
|
||||
|
||||
Exponential backoff schedule:
|
||||
Attempt 1 → wait 1 min (2^0)
|
||||
Attempt 2 → wait 2 min (2^1)
|
||||
Attempt 3 → wait 4 min (2^2)
|
||||
Attempt 4 → wait 8 min (2^3)
|
||||
Attempt 5 → wait 16 min (2^4)
|
||||
Total worst case: ~31 min before giving up
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Store full message text** in the queue table so we can retry without re-rendering. This is important because ad data might change or be removed from willhaben by the time we retry.
|
||||
- **Process at start of scheduler cycle** — ensures queued items are attempted before new scraping starts, prioritizing user notifications over fresh data collection.
|
||||
- **Backoff based on `updated_at`**, not wall-clock from first attempt. Each retry resets the backoff timer. This handles edge cases where a failure was transient but then another transient follows.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration for `notification_queue` table
|
||||
|
||||
In `worker/src/migrations/02-notification-queue.sql`:
|
||||
|
||||
```sql
|
||||
-- Notification retry queue table
|
||||
|
||||
CREATE TYPE notification_type AS ENUM ('new', 'drop');
|
||||
CREATE TYPE notification_status AS ENUM ('pending', 'sent', 'failed', 'dead');
|
||||
|
||||
CREATE TABLE IF NOT EXISTS notification_queue (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
ad_id uuid NOT NULL REFERENCES ads(id) ON DELETE SET NULL,
|
||||
telegram_id text NOT NULL,
|
||||
message_text text NOT NULL,
|
||||
type notification_type NOT NULL,
|
||||
attempts int NOT NULL DEFAULT 0,
|
||||
max_attempts int NOT NULL DEFAULT 5,
|
||||
last_error text,
|
||||
status notification_status NOT NULL DEFAULT 'pending',
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Index for efficient queue polling
|
||||
CREATE INDEX IF NOT EXISTS idx_notif_queue_poll
|
||||
ON notification_queue(status, attempts, updated_at)
|
||||
WHERE status IN ('pending', 'failed');
|
||||
|
||||
COMMENT ON TABLE notification_queue IS
|
||||
'Persistent retry queue for failed Telegram notifications';
|
||||
```
|
||||
|
||||
### 2. Update `notifier.py` — enqueue on failure
|
||||
|
||||
**Current:**
|
||||
```python
|
||||
async def _send_message(bot: Bot, chat_id: int, message_text):
|
||||
try:
|
||||
await bot.send_message(chat_id=chat_id, text=message_text)
|
||||
except TelegramError as e:
|
||||
logger.warning(
|
||||
"Failed to send notification to %d: %s", chat_id, str(e)[:30]
|
||||
)
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
async def _send_message(
|
||||
bot: Bot,
|
||||
chat_id: int,
|
||||
message_text: str,
|
||||
ad_id: uuid.UUID | None = None,
|
||||
notif_type: str = "new",
|
||||
):
|
||||
try:
|
||||
await bot.send_message(chat_id=chat_id, text=message_text)
|
||||
except TelegramError as e:
|
||||
error_msg = str(e)[:300] # cap length
|
||||
logger.warning("Telegram send failed for %s: %s", chat_id, error_msg)
|
||||
|
||||
if ad_id:
|
||||
await _enqueue_retry(
|
||||
ad_id=ad_id,
|
||||
telegram_id=str(chat_id),
|
||||
message_text=message_text,
|
||||
notif_type=notif_type,
|
||||
error_msg=error_msg,
|
||||
)
|
||||
|
||||
|
||||
async def _enqueue_retry(
|
||||
ad_id: uuid.UUID,
|
||||
telegram_id: str,
|
||||
message_text: str,
|
||||
notif_type: str,
|
||||
error_msg: str,
|
||||
) -> None:
|
||||
"""Store a failed notification for later retry."""
|
||||
from db import get_pool
|
||||
|
||||
pool = await get_pool()
|
||||
|
||||
# Check if already queued (avoid duplicates for same ad+user)
|
||||
existing = await pool.fetchval(
|
||||
"""SELECT id FROM notification_queue
|
||||
WHERE ad_id = $1 AND telegram_id = $2 AND status IN ('pending', 'failed')""",
|
||||
ad_id, telegram_id,
|
||||
)
|
||||
|
||||
if existing:
|
||||
logger.info("Already queued: ad=%s user=%s", ad_id[:8], telegram_id)
|
||||
return
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO notification_queue
|
||||
(ad_id, telegram_id, message_text, type, last_error, status)
|
||||
VALUES ($1, $2, $3, $4, $5, 'pending')""",
|
||||
ad_id, telegram_id, message_text, notif_type, error_msg,
|
||||
)
|
||||
|
||||
logger.info("Queued for retry: ad=%s user=%s", ad_id[:8], telegram_id)
|
||||
```
|
||||
|
||||
### 3. Add queue processing to scheduler in `main.py`
|
||||
|
||||
At the start of each scheduler cycle (before keyword scraping):
|
||||
|
||||
```python
|
||||
async def process_notification_queue() -> int:
|
||||
"""Process pending notifications from the retry queue."""
|
||||
from db import get_pool
|
||||
|
||||
pool = await get_pool()
|
||||
|
||||
# Get items eligible for retry (backoff respected)
|
||||
rows = await pool.fetch("""
|
||||
SELECT id, ad_id, telegram_id, message_text, type, attempts,
|
||||
max_attempts, last_error, updated_at
|
||||
FROM notification_queue
|
||||
WHERE status IN ('pending', 'failed')
|
||||
AND updated_at + ($1 || ' minutes')::interval <= now()
|
||||
ORDER BY attempts ASC, updated_at ASC
|
||||
""", "2^attempts" if pool.is_pg else 0)
|
||||
|
||||
# Actually use a computed backoff in Python since PG expressions are tricky:
|
||||
rows = await pool.fetch("""
|
||||
SELECT id, ad_id, telegram_id, message_text, type, attempts,
|
||||
max_attempts, last_error, updated_at
|
||||
FROM notification_queue
|
||||
WHERE status IN ('pending', 'failed')
|
||||
ORDER BY attempts ASC, updated_at ASC
|
||||
LIMIT 50
|
||||
""")
|
||||
|
||||
processed = 0
|
||||
|
||||
for row in rows:
|
||||
backoff_min = min(2 ** row["attempts"], 60) # cap at 60 min
|
||||
retry_after = row["updated_at"] + timedelta(minutes=backoff_min)
|
||||
|
||||
if datetime.now(tz=timezone.utc) < retry_after:
|
||||
continue # not yet eligible
|
||||
|
||||
try:
|
||||
from bot import get_application_bot
|
||||
bot = get_application_bot()
|
||||
|
||||
await bot.send_message(
|
||||
chat_id=int(row["telegram_id"]),
|
||||
text=row["message_text"]
|
||||
)
|
||||
|
||||
await pool.execute(
|
||||
"UPDATE notification_queue SET status = 'sent', updated_at = now() WHERE id = $1",
|
||||
row["id"],
|
||||
)
|
||||
|
||||
await log_notify(pool, row["ad_id"], int(row["telegram_id"]), row["type"])
|
||||
|
||||
processed += 1
|
||||
|
||||
except TelegramError as e:
|
||||
new_attempts = row["attempts"] + 1
|
||||
|
||||
if new_attempts >= row["max_attempts"]:
|
||||
await pool.execute(
|
||||
"""UPDATE notification_queue
|
||||
SET status = 'dead', attempts = $2, last_error = $3, updated_at = now()
|
||||
WHERE id = $1""",
|
||||
row["id"], new_attempts, str(e)[:300],
|
||||
)
|
||||
logger.error(
|
||||
"Notification DEAD after %d attempts: ad=%s user=%s err=%s",
|
||||
new_attempts, row["ad_id"][:8], row["telegram_id"], e,
|
||||
)
|
||||
else:
|
||||
await pool.execute(
|
||||
"""UPDATE notification_queue
|
||||
SET status = 'failed', attempts = $2, last_error = $3, updated_at = now()
|
||||
WHERE id = $1""",
|
||||
row["id"], new_attempts, str(e)[:300],
|
||||
)
|
||||
|
||||
return processed
|
||||
```
|
||||
|
||||
### 4. Call `process_notification_queue()` in the scheduler loop
|
||||
|
||||
In `main.py`, before iterating keywords:
|
||||
|
||||
```python
|
||||
async def run_scheduler() -> None:
|
||||
while True:
|
||||
try:
|
||||
record_scheduler_run() # healthcheck
|
||||
|
||||
# Process pending notifications FIRST
|
||||
processed = await process_notification_queue()
|
||||
if processed:
|
||||
logger.info("Retried %d queued notifications", processed)
|
||||
|
||||
# ... existing keyword iteration ...
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] When `_send_message()` raises `TelegramError`, the notification is INSERTed into `notification_queue` with status='pending'
|
||||
- [ ] On the next scheduler cycle, pending items are attempted (respecting backoff)
|
||||
- [ ] After 5 failed attempts, the notification status becomes 'dead' and a warning is logged
|
||||
- [ ] The queue processes at most 50 items per cycle to avoid blocking the scheduler
|
||||
- [ ] Duplicate enqueue prevention works: calling `_enqueue_retry` twice for the same ad+user creates only one queue entry
|
||||
- [ ] Successful retries update `log_notifications` table (same as direct notifications)
|
||||
- [ ] The `/health` endpoint or logs can show the current count of pending/dead items
|
||||
@@ -0,0 +1,189 @@
|
||||
# Task: Pagination in willhaben scraper
|
||||
|
||||
## Description
|
||||
|
||||
The current `scraper.fetch_ads()` only fetches page 1 (30 ads, sorted newest first). For popular keywords, this means many new listings are missed between scrape cycles — especially when the cycle interval is ≥5 minutes.
|
||||
|
||||
This task implements **cursor-based pagination** that:
|
||||
- Fetches multiple pages per cycle (configurable, default: 2 pages = 60 ads)
|
||||
- Tracks a cursor timestamp to avoid re-processing already-seen ads from the previous cycle
|
||||
- Respects API rate limits by adding delays between page fetches
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Current flow (page 1 only):
|
||||
scheduler → fetch_ads("keyword")
|
||||
→ GET .../ad-search?rows=30&sort=1
|
||||
→ process 30 ads → done
|
||||
|
||||
Target flow (paginated with cursor):
|
||||
scheduler → fetch_ads("keyword", last_seen_cursor)
|
||||
├─ GET ?rows=30&offset=0 → process batch, track latest timestamp
|
||||
├─ sleep 1s (politeness)
|
||||
├─ GET ?rows=30&offset=30 → process batch, stop if duplicates detected
|
||||
└─ ... until max_pages or no new ads
|
||||
|
||||
After processing: update last_seen_cursor for this keyword
|
||||
|
||||
Database tracking:
|
||||
keywords table adds: last_seen_cursor timestamptz
|
||||
(or use existing last_scraped_at as cursor — simpler)
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Use `last_scraped_at` as cursor** instead of adding a new column. After each cycle, the earliest ad processed becomes the cursor for the next cycle. Only ads newer than this are candidates for notification.
|
||||
- *Tradeoff*: If an ad was posted exactly between cycles, it could be missed if it appears on page 2+. Mitigated by processing at least 2 pages and keeping intervals short.
|
||||
- **`max_pages` config via env var** (`SCRAPE_MAX_PAGES=2`). Default is conservative (2) to balance coverage vs API load. Users with expensive keywords can increase per-keyword later.
|
||||
- **Stop early on duplicate detection**: If page N has the same `PUBLISHED_String` as page N-1's last ad, stop — we've exhausted newer results.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Update `keywords` table schema
|
||||
|
||||
Add a cursor column (or reuse `last_scraped_at`). Recommendation: **reuse** since it already exists and is indexed:
|
||||
|
||||
```sql
|
||||
-- No new column needed. Use last_scraped_at as the cursor.
|
||||
-- Ads published after last_scraped_at are "new" for this cycle.
|
||||
```
|
||||
|
||||
If we want a dedicated, more precise cursor (in case last_scraped_at is set before processing completes):
|
||||
|
||||
```sql
|
||||
ALTER TABLE keywords ADD COLUMN IF NOT EXISTS ads_cursor timestamptz;
|
||||
COMMENT ON COLUMN keywords.ads_cursor IS
|
||||
'Timestamp of the oldest ad processed in the last cycle. Used for pagination.';
|
||||
```
|
||||
|
||||
### 2. Update `scraper.py` — add pagination support
|
||||
|
||||
```python
|
||||
_MAX_PAGES = int(os.getenv("SCRAPE_MAX_PAGES", "2"))
|
||||
_PAGE_DELAY_S = float(os.getenv("SCRAPE_PAGE_DELAY_S", "1.0"))
|
||||
|
||||
|
||||
async def fetch_ads(
|
||||
keyword: str,
|
||||
cursor_at: datetime | None = None,
|
||||
max_pages: int | None = None,
|
||||
) -> tuple[list[dict[str, Any]], int]:
|
||||
"""Fetch ads with pagination, deduping by cursor timestamp."""
|
||||
pages = max_pages or _MAX_PAGES
|
||||
all_ads_raw: list[dict[str, Any]] = []
|
||||
total_hits: int = 0
|
||||
|
||||
client = await get_client() # from task-httpx-singleton
|
||||
|
||||
for page in range(pages):
|
||||
params = {
|
||||
"keyword": keyword,
|
||||
"rows": 30,
|
||||
"sort": 1, # newest first
|
||||
"offset": page * 30,
|
||||
}
|
||||
|
||||
try:
|
||||
resp = await fetch_with_retry(client, _API_URL, params)
|
||||
data = resp.json()
|
||||
total_hits = int(data.get("rowsFound", 0))
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"fetch_ads page %d failed for '%s': %s", page, keyword, exc
|
||||
)
|
||||
break
|
||||
|
||||
page_ads = (data.get("advertSummaryList") or {}).get("advertSummary", [])
|
||||
|
||||
if not page_ads:
|
||||
logger.info("No more ads on page %d for '%s'", page, keyword)
|
||||
break
|
||||
|
||||
# Check early stop: if oldest ad on this page is at or before cursor
|
||||
oldest_published = _get_oldest_published(page_ads)
|
||||
if cursor_at and oldest_published and oldest_published <= cursor_at:
|
||||
logger.info(
|
||||
"Early stop at page %d for '%s' — reached cursor",
|
||||
page, keyword
|
||||
)
|
||||
break
|
||||
|
||||
# Filter out already-seen ads within this batch
|
||||
new_batch = [
|
||||
ad for ad in page_ads
|
||||
if not cursor_at or _get_published(ad) is None or _get_published(ad) > cursor_at
|
||||
]
|
||||
|
||||
all_ads_raw.extend(new_batch)
|
||||
|
||||
# Politeness delay between pages (not after last page)
|
||||
if page < pages - 1 and new_batch:
|
||||
await asyncio.sleep(_PAGE_DELAY_S)
|
||||
|
||||
return all_ads_raw, total_hits
|
||||
|
||||
|
||||
def _get_published(ad_dict: dict) -> datetime | None:
|
||||
"""Extract published timestamp from a single ad dict."""
|
||||
attrs = _parse_attributes(ad_dict)
|
||||
raw = attrs.get("PUBLISHED_String") or attrs.get("CHANGED_String")
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(raw.replace("Z", "+00:00"))
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def _get_oldest_published(ads: list[dict]) -> datetime | None:
|
||||
"""Get the oldest published timestamp from a batch of ads."""
|
||||
timestamps = [_get_published(ad) for ad in ads]
|
||||
timestamps = [t for t in timestamps if t is not None]
|
||||
return min(timestamps) if timestamps else None
|
||||
```
|
||||
|
||||
### 3. Update `main.py` scheduler to pass cursor and update it
|
||||
|
||||
In the scheduler loop, before calling `fetch_ads`:
|
||||
|
||||
```python
|
||||
# Get current cursor (last_scraped_at or ads_cursor)
|
||||
cursor = row["ads_cursor"] or row["last_scraped_at"]
|
||||
|
||||
ads_raw, total_hits = await fetch_ads(keyword, cursor_at=cursor)
|
||||
# ... process ads ...
|
||||
|
||||
# Update cursor to the oldest new ad processed
|
||||
if new_timestamps:
|
||||
oldest_new = min(new_timestamps)
|
||||
await pool.execute(
|
||||
"UPDATE keywords SET last_scraped_at = now(), ads_cursor = $1 WHERE id = $2",
|
||||
oldest_new, kw_id,
|
||||
)
|
||||
else:
|
||||
await pool.execute(
|
||||
"UPDATE keywords SET last_scraped_at = now() WHERE id = $2",
|
||||
kw_id,
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Add `ads_cursor` to migration file
|
||||
|
||||
In `worker/src/migrations/01-schema.sql`, add after the keywords table:
|
||||
|
||||
```sql
|
||||
ALTER TABLE keywords ADD COLUMN IF NOT EXISTS ads_cursor timestamptz;
|
||||
COMMENT ON COLUMN keywords.ads_cursor IS
|
||||
'Timestamp of oldest ad processed in last cycle, for pagination cursor';
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] A scrape cycle fetches at least 2 pages (60 ads) by default for active keywords
|
||||
- [ ] The `max_pages` limit is configurable via `SCRAPE_MAX_PAGES` environment variable
|
||||
- [ ] Early stop detection works: if page N has no newer ads than the cursor, pagination stops without fetching remaining pages
|
||||
- [ ] Ads are not re-notified across cycles (cursor prevents duplicates)
|
||||
- [ ] A 1-second delay between page fetches is logged and respected
|
||||
- [ ] `total_hits` from willhaben API is still returned for logging/stats purposes
|
||||
- [ ] No regression in single-page behavior when max_pages=1
|
||||
@@ -0,0 +1,82 @@
|
||||
# Phase 2 — User Experience & Advanced Filtering
|
||||
|
||||
## Scope
|
||||
|
||||
This phase introduces **user-facing features** that significantly improve the experience of keyword tracking. Currently, every matching ad triggers an instant notification regardless of price, location, or time of day — leading to noise for popular keywords.
|
||||
|
||||
After this phase:
|
||||
- Users can configure price ranges and postcodes per keyword
|
||||
- Notifications respect mute hours (no alerts at 3 AM)
|
||||
- Users opt into digest mode (bundled summaries instead of individual pings)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ User Interaction Layer │
|
||||
│ │
|
||||
│ Telegram Bot Commands: │
|
||||
│ /set_price_min <kw> <€> │
|
||||
│ /set_price_max <kw> <€> │
|
||||
│ /set_postcode <kw> <list> │
|
||||
│ /mute_hours <start>-<end> │
|
||||
│ /digest on|off │
|
||||
│ │
|
||||
└──────────┬───────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ Database Schema (extended) │
|
||||
│ │
|
||||
│ keywords table: │
|
||||
│ + price_min int │
|
||||
│ + price_max int │
|
||||
│ + allowed_postcodes text[] │
|
||||
│ │
|
||||
│ user_settings table (new): │
|
||||
│ telegram_id text PK │
|
||||
│ mute_start time │
|
||||
│ mute_end time │
|
||||
│ digest_mode bool DEFAULT false │
|
||||
│ digest_interval int DEFAULT 60 │
|
||||
└──────────┬───────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ Notification Pipeline (modified) │
|
||||
│ │
|
||||
│ For each new ad: │
|
||||
│ ├─ filter by price_min/max? → skip │
|
||||
│ ├─ filter by allowed_postcodes? → skip │
|
||||
│ ├─ user in mute hours? │
|
||||
│ │ digest_on → buffer to digest_table │
|
||||
│ │ digest_off→ skip notification │
|
||||
│ └─ normal → send now │
|
||||
│ │
|
||||
│ Digest scheduler (separate task): │
|
||||
│ every digest_interval: │
|
||||
│ collect buffered notifications per user │
|
||||
│ format as summary message │
|
||||
│ send single message │
|
||||
│ clear buffer │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Tasks
|
||||
|
||||
| Task | File | Description |
|
||||
|------|------|-------------|
|
||||
| Price range filters per keyword | [task-price-filters.md](./task-price-filters.md) | Add `price_min` and `price_max` columns to the keywords table; filter ads during processing based on these thresholds. Bot commands to set/unset. |
|
||||
| Location / postcode filters per keyword | [task-postcode-filters.md](./task-postcode-filters.md) | Add `allowed_postcodes` text[] column to keywords; only notify if an ad's location matches any allowed postcode. |
|
||||
| Mute hours per user | [task-mute-hours.md](./task-mute-hours.md) | Create `user_settings` table with configurable mute window (start/end time in UTC); suppress notifications during this window. |
|
||||
| Digest / summary notifications | [task-digest-notifications.md](./task-digest-notifications.md) | Buffer notifications for users with digest mode enabled; send a bundled summary at configured intervals instead of individual alerts. |
|
||||
|
||||
## General Acceptance Criteria
|
||||
|
||||
- [ ] Users can set price min/max on any keyword and only receive notifications within that range
|
||||
- [ ] Postcode filtering works — ads outside allowed postcodes are silently skipped (not counted as new)
|
||||
- [ ] Mute hours suppress all notifications to a user during the configured window, regardless of keyword
|
||||
- [ ] Digest mode buffers individual alerts and sends one summary message at the configured interval
|
||||
- [ ] All filters combine correctly: an ad is only notified if it passes price + postcode checks AND the user is not muted (or digest mode active)
|
||||
- [ ] The bot provides clear feedback when a filter setting is changed ("Keyword X: price range set to €100–€500")
|
||||
- [ ] Admin can view all keyword filters and user settings via `/keywords` command output
|
||||
@@ -0,0 +1,355 @@
|
||||
# Task: Digest / summary notifications
|
||||
|
||||
## Description
|
||||
|
||||
For popular keywords that generate many matches per cycle, users may receive 10–20 individual notifications in quick succession. This task introduces **digest mode** — instead of immediate alerts, notifications are buffered and sent as a single summary message at configurable intervals.
|
||||
|
||||
Digest mode is complementary to mute hours: during mute hours, all messages are suppressed; with digest mode ON, messages are collected and sent as a batch at the configured interval (default: every 60 minutes).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ user_settings table │
|
||||
│ digest_mode bool DEFAULT false │
|
||||
│ digest_interval int DEFAULT 60 │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Notification Pipeline (modified) │
|
||||
│ │
|
||||
│ For each new ad: │
|
||||
│ if user.digest_mode == false: │
|
||||
│ → send immediately (current) │
|
||||
│ elif in mute hours: │
|
||||
│ → discard (already handled) │
|
||||
│ else: │
|
||||
│ → insert into digest_buffer │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ digest_buffer table (new) │
|
||||
│ │
|
||||
│ id uuid PK │
|
||||
│ telegram_id text │
|
||||
│ ad_id uuid REFERENCES ads │
|
||||
│ keyword text │
|
||||
│ title text │
|
||||
│ price int │
|
||||
│ url text │
|
||||
│ created_at timestamptz │
|
||||
│ │
|
||||
│ INDEX: telegram_id, created_at │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Digest Scheduler (separate task) │
|
||||
│ │
|
||||
│ Runs every digest_interval per user │
|
||||
│ ├─ SELECT all buffered items │
|
||||
│ ├─ GROUP BY telegram_id │
|
||||
│ ├─ Format summary message │
|
||||
│ └─ DELETE buffered items │
|
||||
│ │
|
||||
│ Summary format: │
|
||||
│ 📋 Digest — 5 new ads (14:30 UTC) │
|
||||
│ │
|
||||
│ 🔑 "rtx 3090" (3 ads): │
|
||||
│ • RTX 3090 Ti - €750 [link] │
|
||||
│ • ASUS RTX 3090 - €680 [link] │
|
||||
│ • MSI RTX 3090 Gaming X - €720 │
|
||||
│ │
|
||||
│ 🔑 "gtx 1660" (2 ads): │
|
||||
│ • GTX 1660 Super - €120 [link] │
|
||||
│ • EVGA GTX 1660 - €95 [link] │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Separate buffer table** instead of in-memory only. Survives restarts, visible via pgAdmin for debugging.
|
||||
- **Per-user interval**: Each user configures their own digest frequency (default 60 min). Implemented with a single scheduler task that checks all users' intervals on each cycle.
|
||||
- **Group by keyword** in the summary message. Makes it easy to scan relevant categories without digging through unrelated listings.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration
|
||||
|
||||
In `worker/src/migrations/04-user-settings.sql`:
|
||||
|
||||
```sql
|
||||
-- digest_buffer table for accumulating notifications
|
||||
|
||||
CREATE TABLE IF NOT EXISTS digest_buffer (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
telegram_id text NOT NULL,
|
||||
ad_id uuid REFERENCES ads(id) ON DELETE CASCADE,
|
||||
keyword text NOT NULL,
|
||||
title text NOT NULL,
|
||||
price int, -- in cents
|
||||
url text,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_digest_buffer
|
||||
ON digest_buffer(telegram_id, created_at DESC);
|
||||
|
||||
COMMENT ON TABLE digest_buffer IS
|
||||
'Buffer for digest-mode notifications. Flushed to Telegram at intervals.';
|
||||
```
|
||||
|
||||
### 2. Buffer notifications instead of sending immediately
|
||||
|
||||
In `main.py` scheduler loop, after all filters pass and mute check passes:
|
||||
|
||||
```python
|
||||
# Check if user has digest mode enabled
|
||||
settings = await pool.fetchrow(
|
||||
"SELECT digest_mode FROM user_settings WHERE telegram_id = $1",
|
||||
telegram_id_str,
|
||||
)
|
||||
|
||||
if settings and settings["digest_mode"]:
|
||||
# Buffer for digest
|
||||
await pool.execute(
|
||||
"""INSERT INTO digest_buffer (telegram_id, ad_id, keyword, title, price, url)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)""",
|
||||
telegram_id_str,
|
||||
ad_id,
|
||||
kw_row["keyword"],
|
||||
ad_dict.get("title", "Unknown"),
|
||||
_extract_price(ad_dict),
|
||||
ad_dict.get("url"),
|
||||
)
|
||||
|
||||
# Still log the notification (for stats)
|
||||
await log_notify(pool, ad_id, telegram_id, "new")
|
||||
else:
|
||||
# Send immediately (current behavior)
|
||||
await notify_new(bot, pool, kw_row["keyword"],
|
||||
telegram_id, ad_dict, ad_id)
|
||||
```
|
||||
|
||||
### 3. Add digest flushing task to scheduler
|
||||
|
||||
In `main.py`, add a new async function and call it at the start of each cycle:
|
||||
|
||||
```python
|
||||
async def flush_digest_buffers() -> int:
|
||||
"""Process pending digest buffers for users whose interval has elapsed."""
|
||||
from db import get_pool
|
||||
|
||||
pool = await get_pool()
|
||||
|
||||
# Get all users with digest mode ON
|
||||
users = await pool.fetch("""
|
||||
SELECT telegram_id, digest_interval
|
||||
FROM user_settings
|
||||
WHERE digest_mode = true
|
||||
""")
|
||||
|
||||
sent_count = 0
|
||||
|
||||
for user in users:
|
||||
interval_min = user["digest_interval"] or 60
|
||||
cutoff = datetime.now(tz=timezone.utc) - timedelta(minutes=interval_min)
|
||||
|
||||
# Get buffered items older than the interval
|
||||
buffered = await pool.fetch("""
|
||||
SELECT db.id, db.keyword, db.title, db.price, db.url
|
||||
FROM digest_buffer db
|
||||
WHERE db.telegram_id = $1
|
||||
AND db.created_at <= $2
|
||||
ORDER BY db.keyword, db.created_at DESC
|
||||
""", user["telegram_id"], cutoff)
|
||||
|
||||
if not buffered:
|
||||
continue
|
||||
|
||||
# Group by keyword
|
||||
from collections import defaultdict
|
||||
groups: dict[str, list] = defaultdict(list)
|
||||
|
||||
for item in buffered:
|
||||
price_str = f"€{item['price']/100:.2f}" if item['price'] else "Free"
|
||||
entry = f"• {item['title']} - {price_str}"
|
||||
groups[item["keyword"]].append(entry)
|
||||
|
||||
# Build summary message
|
||||
lines = [f"📋 Digest — {len(buffered)} new ads ({cutoff:%H:%M}–{datetime.now(tz=timezone.utc):%H:%M} UTC)\n"]
|
||||
|
||||
for kw_name, entries in groups.items():
|
||||
lines.append(f"\n🔑 \"{kw_name}\" ({len(entries)} ads):")
|
||||
# Limit to 10 entries per keyword to avoid spam
|
||||
for entry in entries[:10]:
|
||||
lines.append(entry)
|
||||
if len(entries) > 10:
|
||||
lines.append(f" ... and {len(entries)-10} more")
|
||||
|
||||
message_text = "\n".join(lines)
|
||||
|
||||
# Send the digest
|
||||
try:
|
||||
from bot import get_application_bot
|
||||
bot = get_application_bot()
|
||||
|
||||
telegram_id_int = int(user["telegram_id"])
|
||||
await bot.send_message(chat_id=telegram_id_int, text=message_text)
|
||||
|
||||
sent_count += 1
|
||||
|
||||
# Log all buffered notifications as delivered
|
||||
buffer_ids = [item["id"] for item in buffered]
|
||||
for ad_item in buffered:
|
||||
await log_notify(pool, ad_item["ad_id"], telegram_id_int, "new")
|
||||
|
||||
except TelegramError as e:
|
||||
logger.error("Digest send failed for %s: %s", user["telegram_id"], e)
|
||||
|
||||
finally:
|
||||
# Clear the buffer (whether sent or not — if it failed, log entries remain in DB)
|
||||
await pool.execute(
|
||||
"""DELETE FROM digest_buffer
|
||||
WHERE telegram_id = $1 AND created_at <= $2""",
|
||||
user["telegram_id"], cutoff,
|
||||
)
|
||||
|
||||
return sent_count
|
||||
|
||||
|
||||
# Call at the start of run_scheduler():
|
||||
async def run_scheduler() -> None:
|
||||
while True:
|
||||
try:
|
||||
record_scheduler_run()
|
||||
|
||||
# Flush digest buffers first
|
||||
digests_sent = await flush_digest_buffers()
|
||||
if digests_sent:
|
||||
logger.info("Sent %d digest summaries", digests_sent)
|
||||
|
||||
# Process notification queue...
|
||||
processed = await process_notification_queue()
|
||||
if processed:
|
||||
logger.info("Retried %d queued notifications", processed)
|
||||
|
||||
# ... existing keyword iteration ...
|
||||
```
|
||||
|
||||
### 4. Add bot commands in `bot.py`
|
||||
|
||||
```python
|
||||
async def cmd_digest_on(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Enable digest mode with optional interval."""
|
||||
|
||||
interval = 60 # default minutes
|
||||
if len(context.args) > 0:
|
||||
try:
|
||||
interval = int(context.args[0])
|
||||
if interval < 5 or interval > 1440:
|
||||
raise ValueError
|
||||
except ValueError:
|
||||
await update.message.reply_text(
|
||||
"Usage: /digest_on [minutes]\n"
|
||||
"Interval must be between 5 and 1440 minutes (24h)."
|
||||
)
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO user_settings (telegram_id, digest_mode, digest_interval)
|
||||
VALUES ($1, true, $2)
|
||||
ON CONFLICT (telegram_id)
|
||||
DO UPDATE SET digest_mode = EXCLUDED.digest_mode,
|
||||
digest_interval = EXCLUDED.digest_interval""",
|
||||
telegram_id, interval,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Digest mode ENABLED\n"
|
||||
f"Digests will be sent every {interval} minutes.\n"
|
||||
"Use /digest_off to return to instant notifications."
|
||||
)
|
||||
|
||||
|
||||
async def cmd_digest_off(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Disable digest mode."""
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO user_settings (telegram_id, digest_mode)
|
||||
VALUES ($1, false)
|
||||
ON CONFLICT (telegram_id)
|
||||
DO UPDATE SET digest_mode = EXCLUDED.digest_mode""",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
# Flush any remaining buffered items immediately
|
||||
await pool.execute(
|
||||
"""DELETE FROM digest_buffer WHERE telegram_id = $1""",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
"✅ Digest mode DISABLED — notifications are now instant."
|
||||
)
|
||||
|
||||
|
||||
async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Show current user settings (extended from mute hours task)."""
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
settings = await pool.fetchrow(
|
||||
"SELECT mute_start, mute_end, digest_mode, digest_interval "
|
||||
"FROM user_settings WHERE telegram_id = $1",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
reply_parts = []
|
||||
|
||||
if not settings or not settings["mute_start"]:
|
||||
reply_parts.append("🔕 Mute hours: OFF")
|
||||
else:
|
||||
reply_parts.append(
|
||||
f"🔕 Mute hours: {settings['mute_start']} — {settings['mute_end']} UTC"
|
||||
)
|
||||
|
||||
if settings and settings["digest_mode"]:
|
||||
reply_parts.append(
|
||||
f"📋 Digest: ON (every {settings['digest_interval']} min)"
|
||||
)
|
||||
else:
|
||||
reply_parts.append("📋 Digest: OFF (instant notifications)")
|
||||
|
||||
await update.message.reply_text("\n".join(reply_parts))
|
||||
```
|
||||
|
||||
Register handlers:
|
||||
```python
|
||||
dp.add_handler(CommandHandler("digest_on", cmd_digest_on))
|
||||
dp.add_handler(CommandHandler("digest_off", cmd_digest_off))
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/digest_on` enables digest mode with 60-minute default interval
|
||||
- [ ] `/digest_on 30` sets digest interval to 30 minutes
|
||||
- [ ] New ads are inserted into `digest_buffer` instead of being sent immediately when digest is ON
|
||||
- [ ] At the configured interval, all buffered items are flushed as a single summary message
|
||||
- [ ] The summary groups ads by keyword and includes price information
|
||||
- [ ] After flushing, buffered items are deleted from the table
|
||||
- [ ] `/digest_off` disables digest mode and sends any remaining buffered items immediately
|
||||
- [ ] Mute hours take precedence over digest — muted notifications are discarded, not buffered
|
||||
@@ -0,0 +1,211 @@
|
||||
# Task: Mute hours per user
|
||||
|
||||
## Description
|
||||
|
||||
Users may add keywords that are popular enough to trigger notifications at any hour. Currently, there's no way to suppress alerts during sleeping hours — the bot sends notifications 24/7.
|
||||
|
||||
This task creates a `user_settings` table with configurable mute windows (start/end time). During the mute window, notifications for that user are suppressed entirely. The notification is not lost — it's still logged in `log_notifications`, but the Telegram message is not sent.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ user_settings table (new) │
|
||||
│ │
|
||||
│ telegram_id text PRIMARY KEY │
|
||||
│ mute_start time │
|
||||
│ mute_end time │
|
||||
│ digest_mode bool DEFAULT false │
|
||||
│ digest_interval int DEFAULT 60 │
|
||||
│ │
|
||||
│ Example: │
|
||||
│ telegram_id = '298181113' │
|
||||
│ mute_start = '22:00:00' │
|
||||
│ mute_end = '07:00:00' │
|
||||
│ → no alerts between 10PM-7AM UTC │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Notification pipeline (in main.py) │
|
||||
│ │
|
||||
│ For each new ad that passes filters: │
|
||||
│ user_settings = get from DB │
|
||||
│ if in_mute_hours(user_settings): │
|
||||
│ log_notify() │
|
||||
│ → skip Telegram send │
|
||||
│ else: │
|
||||
│ notify_new() / notify_drop() │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Time stored as `time` type in PostgreSQL** — native, efficient for range checks. Default is NULL (no mute window).
|
||||
- *Alternative*: Could store as integer hours (e.g., 22, 7), but `time` type gives flexibility for minute-level precision and clearer UI.
|
||||
- **UTC timezone**: The bot operates in UTC internally. Users should be informed that mute times are in UTC. Adding timezone support per-user is a Phase 3 consideration.
|
||||
- **Mute window can cross midnight** — start > end means the window wraps around midnight (e.g., 22:00–07:00). The check handles this correctly.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration
|
||||
|
||||
In `worker/src/migrations/04-user-settings.sql`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS user_settings (
|
||||
telegram_id text PRIMARY KEY,
|
||||
mute_start time,
|
||||
mute_end time,
|
||||
digest_mode bool NOT NULL DEFAULT false,
|
||||
digest_interval int NOT NULL DEFAULT 60, -- minutes
|
||||
|
||||
CONSTRAINT chk_mute_hours CHECK (
|
||||
mute_start IS NULL AND mute_end IS NULL
|
||||
OR mute_start IS NOT NULL AND mute_end IS NOT NULL
|
||||
)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE user_settings IS
|
||||
'User-specific settings for notification behavior';
|
||||
```
|
||||
|
||||
### 2. Add mute hours check in `notifier.py` or `main.py`
|
||||
|
||||
```python
|
||||
async def _is_in_mute_hours(
|
||||
telegram_id: str,
|
||||
pool: asyncpg.Pool
|
||||
) -> bool:
|
||||
"""Check if the current time is within the user's mute window."""
|
||||
|
||||
settings = await pool.fetchrow(
|
||||
"SELECT mute_start, mute_end FROM user_settings WHERE telegram_id = $1",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
if not settings or not settings["mute_start"] or not settings["mute_end"]:
|
||||
return False # no mute configured
|
||||
|
||||
now_utc = datetime.now(tz=timezone.utc).time()
|
||||
start = settings["mute_start"]
|
||||
end = settings["mute_end"]
|
||||
|
||||
if start < end:
|
||||
# Normal window (e.g., 22:00–07:00 → actually wraps, so this is rare)
|
||||
return start <= now_utc <= end
|
||||
else:
|
||||
# Window crosses midnight (e.g., 22:00 to 07:00 next day)
|
||||
return now_utc >= start or now_utc <= end
|
||||
|
||||
|
||||
# In main.py scheduler loop, before calling notify_new():
|
||||
telegram_id_str = str(telegram_id)
|
||||
|
||||
in_mute = await _is_in_mute_hours(telegram_id_str, pool)
|
||||
if in_mute:
|
||||
logger.debug("Muted notification for user %s (mute window active)", telegram_id)
|
||||
# Still log it but don't send Telegram message
|
||||
await log_notify(pool, ad_id, telegram_id, "new")
|
||||
return
|
||||
|
||||
# Proceed with normal notification...
|
||||
```
|
||||
|
||||
### 3. Add bot commands in `bot.py`
|
||||
|
||||
```python
|
||||
async def cmd_set_mute(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Set mute hours window (HH:MM-HH:MM format)."""
|
||||
if len(context.args) < 1:
|
||||
await update.message.reply_text(
|
||||
"Usage: /mute_hours HH:MM-HH:MM\n"
|
||||
"Example: /mute_hours 22:00-07:00 (mutes from 10PM to 7AM UTC)\n"
|
||||
"Use /mute_off to disable."
|
||||
)
|
||||
return
|
||||
|
||||
try:
|
||||
start_str, end_str = context.args[0].split("-")
|
||||
mute_start = datetime.strptime(start_str.strip(), "%H:%M").time()
|
||||
mute_end = datetime.strptime(end_str.strip(), "%H:%M").time()
|
||||
except (ValueError, TypeError) as e:
|
||||
await update.message.reply_text(
|
||||
f"Invalid format. Use HH:MM-HH:MM.\nExample: /mute_hours 22:00-07:00"
|
||||
)
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO user_settings (telegram_id, mute_start, mute_end)
|
||||
VALUES ($1, $2, $3)
|
||||
ON CONFLICT (telegram_id)
|
||||
DO UPDATE SET mute_start = EXCLUDED.mute_start, mute_end = EXCLUDED.mute_end""",
|
||||
telegram_id, mute_start, mute_end,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Mute hours set: {mute_start} — {mute_end} UTC\n"
|
||||
"No notifications will be sent during this window.\n"
|
||||
"Use /mute_off to disable or change."
|
||||
)
|
||||
|
||||
|
||||
async def cmd_mute_off(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Disable mute hours."""
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO user_settings (telegram_id, mute_start, mute_end)
|
||||
VALUES ($1, NULL, NULL)
|
||||
ON CONFLICT (telegram_id)
|
||||
DO UPDATE SET mute_start = EXCLUDED.mute_start, mute_end = EXCLUDED.mute_end""",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
await update.message.reply_text("✅ Mute hours disabled.")
|
||||
|
||||
|
||||
async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Show current user settings."""
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
settings = await pool.fetchrow(
|
||||
"SELECT mute_start, mute_end FROM user_settings WHERE telegram_id = $1",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
if not settings or not settings["mute_start"]:
|
||||
reply = "🔕 Mute hours: OFF (notifications sent 24/7)"
|
||||
else:
|
||||
reply = f"🔕 Mute hours: {settings['mute_start']} — {settings['mute_end']} UTC"
|
||||
|
||||
await update.message.reply_text(reply)
|
||||
```
|
||||
|
||||
Register handlers:
|
||||
```python
|
||||
dp.add_handler(MessageHandler(REGEX(r"^/mute_hours"), cmd_set_mute))
|
||||
dp.add_handler(CommandHandler("mute_off", cmd_mute_off))
|
||||
dp.add_handler(CommandHandler("status", cmd_status))
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/mute_hours 22:00-07:00` sets mute window from 10 PM to 7 AM UTC
|
||||
- [ ] Notifications during the mute window are logged but NOT sent via Telegram
|
||||
- [ ] Notifications outside the mute window work normally (no regression)
|
||||
- [ ] Mute windows that cross midnight (start > end) are handled correctly
|
||||
- [ ] `/mute_off` clears both start and end times, restoring 24/7 notifications
|
||||
- [ ] `/status` shows current mute settings clearly
|
||||
- [ ] Users without any settings in `user_settings` table receive all notifications (default behavior unchanged)
|
||||
@@ -0,0 +1,199 @@
|
||||
# Task: Location / postcode filters per keyword
|
||||
|
||||
## Description
|
||||
|
||||
Ads can match a user's keyword interest but be located in a completely different region (e.g., "rtx 3090" in Graz when the user only cares about Vienna). This task adds optional postcode filtering so users receive alerts only for ads in their desired locations.
|
||||
|
||||
The willhaben API returns location data in `LOCATION_CityName`, `LOCATION_ZIP` (postcode), and similar fields. We'll match against these fields.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ keywords table (extended) │
|
||||
│ │
|
||||
│ ... │
|
||||
│ allowed_postcodes text[] │
|
||||
│ ... │
|
||||
│ │
|
||||
│ Example: │
|
||||
│ allowed_postcodes = {'1010','1020'} │
|
||||
│ → only ads in Vienna 1st/2nd dist. │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Processing filter (in main.py) │
|
||||
│ │
|
||||
│ for ad in ads_raw: │
|
||||
│ ad_zip = _extract_postcode(ad) │
|
||||
│ if kw.allowed_postcodes and │
|
||||
│ ad_zip not in postcodes: │
|
||||
│ skip │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Bot commands (in bot.py): │
|
||||
│ │
|
||||
│ /postcode <keyword> p1,p2,p3 │
|
||||
│ → sets allowed_postcodes = {'p1', │
|
||||
│ 'p2','p3'} │
|
||||
│ /clear_postcode <keyword> │
|
||||
│ → removes filter (NULL) │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Text array** (`text[]`) instead of a separate lookup table. Simple, efficient for the typical case (<10 postcodes per keyword), and leverages PostgreSQL's native array support.
|
||||
- *Alternative*: A `keyword_postcodes` junction table allows individual postcode management but adds unnecessary complexity for this use case.
|
||||
- **Match against willhaben's ZIP code field** (`LOCATION_ZIP` in the ad attributes). This is the most reliable location identifier and works across all Austrian postcodes (4 digits, e.g., "1010", "8010").
|
||||
- **Empty or missing postcode = skip if filter active**. If `allowed_postcodes` is set but an ad has no ZIP code, it's excluded. This prevents noise from unlocated ads.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration
|
||||
|
||||
In `worker/src/migrations/03-keyword-filters.sql`:
|
||||
|
||||
```sql
|
||||
ALTER TABLE keywords ADD COLUMN IF NOT EXISTS allowed_postcodes text[];
|
||||
|
||||
COMMENT ON COLUMN keywords.allowed_postcodes IS
|
||||
'Austrian postcodes (4-digit strings). Only ads matching these are notified.';
|
||||
```
|
||||
|
||||
### 2. Extract postcode from ad data in `notifier.py` or `scraper.py`
|
||||
|
||||
Add helper function:
|
||||
|
||||
```python
|
||||
def _extract_postcode(ad_dict: dict) -> str | None:
|
||||
"""Extract the postal code (ZIP) from a willhaben ad."""
|
||||
attrs = _parse_attributes(ad_dict)
|
||||
|
||||
# Try multiple field names that willhaben might use
|
||||
for key in ("LOCATION_ZIP", "LocationZip", "postalcode"):
|
||||
val = attrs.get(key) or attrs.get(f"{key}_String")
|
||||
if val:
|
||||
return str(val).strip()
|
||||
|
||||
return None
|
||||
```
|
||||
|
||||
### 3. Add postcode filter check in `main.py` scheduler loop
|
||||
|
||||
```python
|
||||
async def _check_postcode_filter(
|
||||
ad_dict: dict,
|
||||
kw_row: dict
|
||||
) -> bool:
|
||||
"""Return True if the ad passes the postcode filter."""
|
||||
|
||||
postcodes = kw_row.get("allowed_postcodes") # list or None
|
||||
|
||||
if not postcodes:
|
||||
return True # no filter active
|
||||
|
||||
from notifier import _extract_postcode # or wherever it lives
|
||||
|
||||
ad_zip = _extract_postcode(ad_dict)
|
||||
|
||||
if not ad_zip:
|
||||
logger.debug("No postcode found in ad, skipping")
|
||||
return False
|
||||
|
||||
# Normalize: willhaben returns "1010" as string, we store same way
|
||||
return ad_zip in postcodes
|
||||
|
||||
|
||||
# In the scheduler loop (after price check):
|
||||
for ad_dict in ads_raw:
|
||||
if not await _check_price_filters(ad_dict, kw_row):
|
||||
continue
|
||||
|
||||
if not await _check_postcode_filter(ad_dict, kw_row):
|
||||
continue
|
||||
|
||||
# ... rest of processing
|
||||
```
|
||||
|
||||
### 4. Add bot commands in `bot.py`
|
||||
|
||||
```python
|
||||
async def cmd_set_postcode(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Set allowed postcodes for a keyword."""
|
||||
if len(context.args) < 2:
|
||||
await update.message.reply_text("Usage: /postcode <keyword> <p1,p2,p3>")
|
||||
return
|
||||
|
||||
kw_name = context.args[0]
|
||||
postcode_strs = [pc.strip() for pc in context.args[1].split(",")]
|
||||
|
||||
# Validate format (4-digit Austrian postcodes)
|
||||
invalid = [pc for pc in postcode_strs if not re.match(r"^\d{3,5}$", pc)]
|
||||
if invalid:
|
||||
await update.message.reply_text(
|
||||
f"Invalid postcode(s): {', '.join(invalid)}. "
|
||||
"Use 4-digit format like 1010, 8010."
|
||||
)
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
kw_id = await pool.fetchval(
|
||||
"""SELECT id FROM keywords
|
||||
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
|
||||
kw_name.lower(), telegram_id,
|
||||
)
|
||||
|
||||
if not kw_id:
|
||||
await update.message.reply_text(f"Keyword '{kw_name}' not found.")
|
||||
return
|
||||
|
||||
await pool.execute(
|
||||
"UPDATE keywords SET allowed_postcodes = $1 WHERE id = $2",
|
||||
postcode_strs, kw_id, # asyncpg handles text[] natively
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Keyword '{kw_name}': postcodes set to {', '.join(postcode_strs)}"
|
||||
)
|
||||
|
||||
|
||||
async def cmd_clear_postcode(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Remove postcode filter for a keyword."""
|
||||
if len(context.args) < 1:
|
||||
await update.message.reply_text("Usage: /clear_postcode <keyword>")
|
||||
return
|
||||
|
||||
# ... same pattern, set allowed_postcodes = NULL
|
||||
```
|
||||
|
||||
Register handlers:
|
||||
```python
|
||||
dp.add_handler(MessageHandler(REGEX(r"^/postcode"), cmd_set_postcode))
|
||||
dp.add_handler(MessageHandler(REGEX(r"^/clear_postcode"), cmd_clear_postcode))
|
||||
```
|
||||
|
||||
### 5. Update `/keywords` command output
|
||||
|
||||
Add postcode info to the listing:
|
||||
```python
|
||||
if row["allowed_postcodes"]:
|
||||
line += f"\n 📍 {', '.join(row['allowed_postcodes'])}"
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/postcode keyword 1010,1020` sets allowed postcodes to `{'1010', '1020'}` for that keyword
|
||||
- [ ] Ads with ZIP codes NOT in the allowed list are skipped during processing
|
||||
- [ ] Ads with no ZIP code at all are skipped when a filter is active
|
||||
- [ ] `/clear_postcode keyword` removes the filter (NULL)
|
||||
- [ ] Invalid postcodes (non-numeric or wrong length) are rejected by the bot
|
||||
- [ ] The `/keywords` command shows active postcodes next to each keyword
|
||||
- [ ] Both price AND postcode filters work correctly together (ad must pass both to be notified)
|
||||
@@ -0,0 +1,254 @@
|
||||
# Task: Price range filters per keyword
|
||||
|
||||
## Description
|
||||
|
||||
Currently, every ad that matches a keyword triggers a notification regardless of price. For keywords like "rtx" or "3090", this means users receive alerts for €5 listings (accessories) alongside relevant hardware deals.
|
||||
|
||||
This task adds `price_min` and `price_max` columns to the `keywords` table and implements server-side filtering during ad processing. Users set these via bot commands, and ads outside the range are silently skipped without being stored in the `ads` table or triggering notifications.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ keywords table (extended) │
|
||||
│ │
|
||||
│ ... │
|
||||
│ price_min int │
|
||||
│ price_max int │
|
||||
│ ... │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Processing pipeline (in main.py) │
|
||||
│ │
|
||||
│ for ad in ads_raw: │
|
||||
│ price = _extract_price(ad) │
|
||||
│ if price is None: │
|
||||
│ continue │
|
||||
│ if kw.price_min and price < min: │
|
||||
│ skip (below minimum) │
|
||||
│ if kw.price_max and price > max: │
|
||||
│ skip (above maximum) │
|
||||
│ → process ad normally │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Bot commands (in bot.py): │
|
||||
│ │
|
||||
│ /price_min <keyword> <€amount> │
|
||||
│ /price_max <keyword> <€amount> │
|
||||
│ /clear_price <keyword> │
|
||||
│ /keywords → shows price ranges │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Filter during processing, not at DB time** — the filter is applied in Python before inserting into `ads`. This keeps the `ads` table clean (only relevant ads are stored) and avoids extra WHERE clauses on every scrape cycle.
|
||||
- *Alternative*: Could use a computed column or trigger, but adds complexity for simple numeric comparison.
|
||||
- **Both filters optional** — NULL = no limit. Users can set only min, only max, both, or neither.
|
||||
- **Price extraction from ad data**: Use the existing `_extract_price()` function in `notifier.py`. If price is not available (free items), treat as €0 for filtering purposes.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration
|
||||
|
||||
In `worker/src/migrations/03-keyword-filters.sql`:
|
||||
|
||||
```sql
|
||||
ALTER TABLE keywords
|
||||
ADD COLUMN IF NOT EXISTS price_min int,
|
||||
ADD COLUMN IF NOT EXISTS price_max int;
|
||||
|
||||
COMMENT ON COLUMN keywords.price_min IS 'Minimum price in cents (e.g. 5000 = €50). NULL = no limit.';
|
||||
COMMENT ON COLUMN keywords.price_max IS 'Maximum price in cents (e.g. 500000 = €5000). NULL = no limit.';
|
||||
```
|
||||
|
||||
### 2. Update `main.py` — filter ads by price during processing
|
||||
|
||||
In the scheduler's keyword loop, before calling `_process_ad()`:
|
||||
|
||||
```python
|
||||
# After extracting ad info in _process_ad():
|
||||
async def _check_price_filters(
|
||||
ad_dict: dict,
|
||||
kw_row: dict # from keywords table
|
||||
) -> bool:
|
||||
"""Return True if the ad passes price filters for this keyword."""
|
||||
|
||||
from notifier import _extract_price
|
||||
|
||||
price = _extract_price(ad_dict)
|
||||
|
||||
if price is None:
|
||||
# No price found — include it (could be "free" or missing data)
|
||||
return True
|
||||
|
||||
price_min = kw_row.get("price_min")
|
||||
price_max = kw_row.get("price_max")
|
||||
|
||||
if price_min is not None and price < price_min:
|
||||
logger.debug(
|
||||
"Price filter skip: ad %s price=%d min=%d",
|
||||
ad_id, price, price_min
|
||||
)
|
||||
return False
|
||||
|
||||
if price_max is not None and price > price_max:
|
||||
logger.debug(
|
||||
"Price filter skip: ad %s price=%d max=%d",
|
||||
ad_id, price, price_max
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
# In the scheduler loop:
|
||||
for kw_row in keywords:
|
||||
if not kw_row["is_active"]:
|
||||
continue
|
||||
|
||||
ads_raw = await fetch_ads(kw_row["keyword"])
|
||||
|
||||
for ad_dict in ads_raw:
|
||||
# Check price filters BEFORE processing
|
||||
if not await _check_price_filters(ad_dict, kw_row):
|
||||
continue
|
||||
|
||||
# Existing _process_ad logic continues here...
|
||||
```
|
||||
|
||||
### 3. Add bot commands in `bot.py`
|
||||
|
||||
```python
|
||||
from telegram import Message
|
||||
from telegram.ext import ContextTypes
|
||||
|
||||
|
||||
async def cmd_set_price_min(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Set minimum price for a keyword."""
|
||||
if len(context.args) < 2:
|
||||
await update.message.reply_text("Usage: /price_min <keyword> <amount_in_euro>")
|
||||
return
|
||||
|
||||
kw_name = context.args[0]
|
||||
try:
|
||||
amount_eur = float(context.args[1])
|
||||
price_cents = int(amount_eur * 100)
|
||||
except ValueError:
|
||||
await update.message.reply_text("Invalid amount. Use a number like 50 or 99.99")
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
# Find keyword by name for this user
|
||||
kw_id = await pool.fetchval(
|
||||
"""SELECT id FROM keywords
|
||||
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
|
||||
kw_name.lower(), telegram_id,
|
||||
)
|
||||
|
||||
if not kw_id:
|
||||
await update.message.reply_text(f"Keyword '{kw_name}' not found for your account.")
|
||||
return
|
||||
|
||||
await pool.execute(
|
||||
"UPDATE keywords SET price_min = $1 WHERE id = $2",
|
||||
price_cents, kw_id,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Keyword '{kw_name}': minimum price set to €{amount_eur:.2f}"
|
||||
)
|
||||
|
||||
|
||||
async def cmd_set_price_max(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Set maximum price for a keyword."""
|
||||
# Same pattern as cmd_set_price_min but sets price_max
|
||||
|
||||
if len(context.args) < 2:
|
||||
await update.message.reply_text("Usage: /price_max <keyword> <amount_in_euro>")
|
||||
return
|
||||
|
||||
kw_name = context.args[0]
|
||||
try:
|
||||
amount_eur = float(context.args[1])
|
||||
price_cents = int(amount_eur * 100)
|
||||
except ValueError:
|
||||
await update.message.reply_text("Invalid amount. Use a number like 50 or 99.99")
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
kw_id = await pool.fetchval(
|
||||
"""SELECT id FROM keywords
|
||||
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
|
||||
kw_name.lower(), telegram_id,
|
||||
)
|
||||
|
||||
if not kw_id:
|
||||
await update.message.reply_text(f"Keyword '{kw_name}' not found.")
|
||||
return
|
||||
|
||||
await pool.execute(
|
||||
"UPDATE keywords SET price_max = $1 WHERE id = $2",
|
||||
price_cents, kw_id,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Keyword '{kw_name}': maximum price set to €{amount_eur:.2f}"
|
||||
)
|
||||
|
||||
|
||||
async def cmd_clear_price(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Clear price filters for a keyword."""
|
||||
if len(context.args) < 1:
|
||||
await update.message.reply_text("Usage: /clear_price <keyword>")
|
||||
return
|
||||
|
||||
# ... similar pattern, set both to NULL
|
||||
```
|
||||
|
||||
Register handlers in `register_handlers()`:
|
||||
```python
|
||||
dp.add_handler(MessageHandler(
|
||||
REGEX(r"^/price_min"), cmd_set_price_min))
|
||||
dp.add_handler(MessageHandler(
|
||||
REGEX(r"^/price_max"), cmd_set_price_max))
|
||||
dp.add_handler(MessageHandler(
|
||||
REGEX(r"^/clear_price"), cmd_clear_price))
|
||||
```
|
||||
|
||||
### 4. Update `/keywords` command output to show price ranges
|
||||
|
||||
In the existing `/keywords` handler in `bot.py`, add:
|
||||
|
||||
```python
|
||||
price_info = ""
|
||||
if row["price_min"] is not None:
|
||||
price_info += f"min €{row['price_min']/100:.2f} "
|
||||
if row["price_max"] is not None:
|
||||
price_info += f"max €{row['price_max']/100:.2f}"
|
||||
if price_info:
|
||||
line += f"\n 💰 {price_info.strip()}"
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/price_min keyword 50` sets the minimum price to €50.00 (stored as 5000 cents) for that keyword
|
||||
- [ ] `/price_max keyword 1000` sets the maximum price to €1000.00
|
||||
- [ ] Ads with price below `price_min` are skipped during processing and not inserted into `ads` table
|
||||
- [ ] Ads with price above `price_max` are skipped during processing
|
||||
- [ ] `/clear_price keyword` removes both limits (sets to NULL)
|
||||
- [ ] Keywords without price filters continue to work as before (no regression)
|
||||
- [ ] The `/keywords` command displays the active price range for each keyword
|
||||
- [ ] Non-admin users can only modify their own keywords' price filters
|
||||
@@ -0,0 +1,93 @@
|
||||
# Phase 3 — Scalability & Advanced Features
|
||||
|
||||
## Scope
|
||||
|
||||
This phase introduces **structural improvements** that make the project maintainable, extensible, and testable. Currently, the entire system is a single async Python process with no tests and no CI/CD pipeline. After this phase:
|
||||
|
||||
- Automated tests provide confidence for every change (≥80% coverage)
|
||||
- CI/CD pipeline runs on every push to validate code quality
|
||||
- Multi-marketplace architecture enables adding new sources without modifying core logic
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Project Structure (post-Phase-3) │
|
||||
│ │
|
||||
│ willhaben-tracker/ │
|
||||
│ ├── worker/ │
|
||||
│ │ ├── src/ │
|
||||
│ │ │ ├── main.py (entry point, scheduler) │
|
||||
│ │ │ ├── db.py (asyncpg pool mgmt) │
|
||||
│ │ │ ├── bot.py (Telegram handlers) │
|
||||
│ │ │ ├── notifier.py (message sending) │
|
||||
│ │ │ ├── scraper.py (base scraper class) │
|
||||
│ │ │ ├── scrapers/ │
|
||||
│ │ │ │ ├── __init__.py │
|
||||
│ │ │ │ ├── willhaben.py (willhaben-specific) │
|
||||
│ │ │ │ └── base.py (abstract base class) │
|
||||
│ │ │ ├── health.py (healthcheck endpoint) │
|
||||
│ │ │ └── migrate.py (migration runner) │
|
||||
│ │ ├── tests/ │
|
||||
│ │ │ ├── conftest.py │
|
||||
│ │ │ ├── test_scraper.py │
|
||||
│ │ │ ├── test_notifier.py │
|
||||
│ │ │ └── ... │
|
||||
│ │ ├── Dockerfile │
|
||||
│ │ └── requirements.txt │
|
||||
│ ├── .github/ │
|
||||
│ │ └── workflows/ │
|
||||
│ │ └── ci.yml (pytest + flake8 + coverage) │
|
||||
│ ├── pyproject.toml (coverage config, tools) │
|
||||
│ └── docker-compose.yml │
|
||||
└──────────────────────────────────────────────────────┘
|
||||
|
||||
Multi-marketplace abstraction:
|
||||
|
||||
ScraperBase (abstract):
|
||||
- async fetch_ads(keyword) → list[dict]
|
||||
- async parse_response(html/json) → list[dict]
|
||||
- normalize_ad(raw) → dict with standard keys
|
||||
|
||||
WillhabenScraper(ScraperBase):
|
||||
- implements willhaben-specific URL, headers, parsing
|
||||
|
||||
Future: KleinAnzeigenScraper, MobileScraper, ...
|
||||
|
||||
CI/CD Pipeline (.github/workflows/ci.yml):
|
||||
|
||||
on: push to main, feat/*; pull_request
|
||||
|
||||
jobs:
|
||||
lint-and-test:
|
||||
└─ python 3.12
|
||||
├─ flake8 (linting)
|
||||
├─ pytest --cov=src tests/ (unit + integration tests)
|
||||
└─ coverage >= 80% (fail if not met)
|
||||
|
||||
Tests Structure:
|
||||
|
||||
Unit tests:
|
||||
- test_scraper_pagination() — verify pagination logic with mock responses
|
||||
- test_price_filters() — verify filter functions
|
||||
- test_notification_retry() — verify retry queue behavior
|
||||
|
||||
Integration tests:
|
||||
- Test against real willhaben API (rate-limited, cached)
|
||||
- PostgreSQL test container via docker-compose
|
||||
```
|
||||
|
||||
## Tasks
|
||||
|
||||
| Task | File | Description |
|
||||
|------|------|-------------|
|
||||
| Multi-marketplace abstraction layer | [task-multi-marketplace.md](./task-multi-marketplace.md) | Refactor `scraper.py` into a base class + per-marketplace implementations. Introduces a standard ad schema and factory for registering new sources. |
|
||||
| Test suite with pytest (≥80% coverage) | [task-testing-pytest.md](./task-testing-pytest.md) | Add comprehensive unit tests covering scraper parsing, notification logic, price/postcode filters, retry queue, and scheduler flow. Configure coverage thresholds. |
|
||||
|
||||
## General Acceptance Criteria
|
||||
|
||||
- [ ] CI pipeline runs on every push to `main` and feature branches — fails if lint or coverage checks are not met
|
||||
- [ ] Code coverage is ≥80% across all source files in `worker/src/`
|
||||
- [ ] Multi-marketplace abstraction works — adding a new marketplace requires only creating one file under `scrapers/` with no changes to core logic
|
||||
- [ ] All existing functionality (willhaben scraping, notifications) continues to work after refactoring
|
||||
- [ ] The `/health` endpoint exposes test results or coverage stats (optional enhancement)
|
||||
@@ -0,0 +1,363 @@
|
||||
# Task: Multi-marketplace abstraction layer
|
||||
|
||||
## Description
|
||||
|
||||
Currently, `scraper.py` is tightly coupled to willhaben's API format and URL. Adding a second marketplace (e.g., Kleinanzeigen, Facebook Marketplace) would require extensive refactoring of the core logic — duplicating pagination, error handling, and notification code with subtle differences per source.
|
||||
|
||||
This task introduces an **abstract base class** for scrapers and a **standardized ad schema**, making it trivial to add new marketplaces by implementing only marketplace-specific parsing logic.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ scraper.py (module) │
|
||||
│ │
|
||||
│ ┌───────────────────────────────────┐│
|
||||
│ │ ScraperBase (ABC) ││
|
||||
│ │ ││
|
||||
│ │ Properties: ││
|
||||
│ │ name str ││
|
||||
│ │ base_url str ││
|
||||
│ │ max_pages int ││
|
||||
│ │ ││
|
||||
│ │ Abstract methods: ││
|
||||
│ │ build_query(url, params) → URL ││
|
||||
│ │ parse_page(html/json) → list ││
|
||||
│ │ normalize_ad(raw) → dict ││
|
||||
│ │ ││
|
||||
│ │ Concrete methods (shared): ││
|
||||
│ │ fetch_ads(keyword, cursor) ││
|
||||
│ │ _fetch_with_retry(url) ││
|
||||
│ └───────────────────────────────────┘│
|
||||
└──────┬────────────────────────────────┘
|
||||
│ inherits
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ scrapers/willhaben.py │
|
||||
│ │
|
||||
│ class WillhabenScraper(ScraperBase): │
|
||||
│ name = "willhaben" ││
|
||||
│ base_url = ".../api/v1/ad-search" ││
|
||||
│ build_query() → willhaben URL ││
|
||||
│ parse_page(json) → list ││
|
||||
│ normalize_ad(raw) → standard dict ││
|
||||
└───────────────────────────────────────┘
|
||||
|
||||
Standard ad schema (dict):
|
||||
{
|
||||
"id": str, # marketplace-specific ID
|
||||
"marketplace": str, # e.g. "willhaben"
|
||||
"title": str, # ad title
|
||||
"price": int | None, # price in cents
|
||||
"currency": str, # e.g. "EUR"
|
||||
"url": str, # full URL to the ad page
|
||||
"published_at": datetime | None,
|
||||
"location": { │
|
||||
"city": str, │
|
||||
"postcode": str | None, │
|
||||
"region": str | None │
|
||||
}, │
|
||||
"attributes": dict # marketplace-specific extras
|
||||
}
|
||||
|
||||
Scheduler (in main.py):
|
||||
|
||||
scrapers: list[ScraperBase] = [
|
||||
WillhabenScraper(),
|
||||
KleinanzeigenScraper(), # future
|
||||
]
|
||||
|
||||
for scraper in scrapers:
|
||||
ads_raw, total_hits = await scraper.fetch_ads(keyword)
|
||||
# ... process with same pipeline (filters, notifications)
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Abstract base class** defines the contract. Concrete scrapers only implement what's different per marketplace — URL building, response parsing, and field normalization.
|
||||
- *Alternative*: Could use a plugin architecture with entry_points, but that adds significant complexity for what is currently expected to be ≤3 marketplaces.
|
||||
- **Standardized output schema** ensures the downstream pipeline (filters, notifications) works identically regardless of source. Marketplace-specific fields are stored in `attributes`.
|
||||
- **Pagination logic lives in the base class**. Most marketplaces use offset/limit pagination; the abstract method handles this generically. Special cases override `_fetch_page()`.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Create `worker/src/scrapers/__init__.py`
|
||||
|
||||
```python
|
||||
from .willhaben import WillhabenScraper
|
||||
|
||||
__all__ = ["WillhabenScraper"]
|
||||
|
||||
|
||||
def get_scriper_by_name(name: str) -> "ScraperBase":
|
||||
"""Factory function to instantiate scrapers by name."""
|
||||
registry = {
|
||||
"willhaben": WillhabenScraper,
|
||||
}
|
||||
|
||||
cls = registry.get(name.lower())
|
||||
if not cls:
|
||||
raise ValueError(f"Unknown marketplace: {name}")
|
||||
|
||||
return cls()
|
||||
```
|
||||
|
||||
### 2. Create `worker/src/scrapers/base.py` (abstract base class)
|
||||
|
||||
```python
|
||||
import abc
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ScraperBase(abc.ABC):
|
||||
"""Abstract base class for marketplace scrapers."""
|
||||
|
||||
name: str = "unknown"
|
||||
base_url: str = ""
|
||||
max_pages: int = 2
|
||||
|
||||
@abc.abstractmethod
|
||||
def build_query(self, keyword: str, offset: int) -> str:
|
||||
"""Build the full API URL/endpoint for a keyword + offset."""
|
||||
...
|
||||
|
||||
@abc.abstractmethod
|
||||
def parse_page(self, response_content: Any) -> list[dict]:
|
||||
"""Parse raw response into list of ad dicts (marketplace-specific)."""
|
||||
...
|
||||
|
||||
@abc.abstractmethod
|
||||
def normalize_ad(self, raw_ad: dict) -> dict:
|
||||
"""Convert marketplace-specific format to standard schema."""
|
||||
...
|
||||
|
||||
async def fetch_ads(
|
||||
self,
|
||||
keyword: str,
|
||||
cursor_at: datetime | None = None,
|
||||
max_pages: int | None = None,
|
||||
) -> tuple[list[dict], int]:
|
||||
"""Fetch ads with pagination. Shared implementation."""
|
||||
pages = max_pages or self.max_pages
|
||||
all_ads: list[dict] = []
|
||||
total_hits = 0
|
||||
|
||||
from ..scraper import get_client # httpx singleton
|
||||
|
||||
client = await get_client()
|
||||
|
||||
for page in range(pages):
|
||||
url = self.build_query(keyword, offset=page * 30)
|
||||
|
||||
try:
|
||||
content = await self._fetch_with_retry(client, url)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"%s: fetch failed at page %d for '%s': %s",
|
||||
self.name, page, keyword, exc
|
||||
)
|
||||
break
|
||||
|
||||
raw_ads = self.parse_page(content)
|
||||
|
||||
if not raw_ads:
|
||||
logger.info("%s: no more ads on page %d for '%s'",
|
||||
self.name, page, keyword)
|
||||
break
|
||||
|
||||
# Normalize and filter by cursor
|
||||
normalized = []
|
||||
for raw in raw_ads:
|
||||
ad = self.normalize_ad(raw)
|
||||
if cursor_at and ad["published_at"] and ad["published_at"] <= cursor_at:
|
||||
continue
|
||||
normalized.append(ad)
|
||||
|
||||
all_ads.extend(normalized)
|
||||
|
||||
# Politeness delay
|
||||
if page < pages - 1 and normalized:
|
||||
await asyncio.sleep(1.0)
|
||||
|
||||
return all_ads, total_hits
|
||||
|
||||
async def _fetch_with_retry(
|
||||
self,
|
||||
client: Any,
|
||||
url: str,
|
||||
max_retries: int = 3,
|
||||
) -> Any:
|
||||
"""Generic retry wrapper for HTTP fetches."""
|
||||
import httpx
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
resp = await client.get(url)
|
||||
resp.raise_for_status()
|
||||
return resp.json() if "application/json" in (resp.headers.get("content-type") or "") else resp.text
|
||||
except httpx.ConnectError as exc:
|
||||
logger.warning("%s: transport error attempt %d: %s",
|
||||
self.name, attempt + 1, exc)
|
||||
if attempt < max_retries - 1:
|
||||
await asyncio.sleep(2 ** attempt)
|
||||
else:
|
||||
raise
|
||||
|
||||
@property
|
||||
def headers(self) -> dict[str, str]:
|
||||
"""HTTP headers for requests. Override per marketplace."""
|
||||
return {}
|
||||
```
|
||||
|
||||
### 3. Create `worker/src/scrapers/willhaben.py` (refactor existing logic)
|
||||
|
||||
Move the current willhaben-specific code from `scraper.py` into this implementation:
|
||||
|
||||
```python
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from .base import ScraperBase
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WillhabenScraper(ScraperBase):
|
||||
name = "willhaben"
|
||||
base_url = "https://api.willhaben.at/external/api/v1/ad-search"
|
||||
|
||||
@property
|
||||
def headers(self) -> dict[str, str]:
|
||||
return {
|
||||
"Accept": "application/json",
|
||||
"User-Agent": "Mozilla/5.0 (compatible; WillhabenTracker/1.0)",
|
||||
}
|
||||
|
||||
def build_query(self, keyword: str, offset: int = 0) -> str:
|
||||
"""Build willhaben API URL with keyword + pagination."""
|
||||
import urllib.parse
|
||||
|
||||
params = {
|
||||
"keyword": keyword,
|
||||
"rows": 30,
|
||||
"sort": 1, # newest first
|
||||
"offset": offset,
|
||||
}
|
||||
|
||||
return f"{self.base_url}?{urllib.parse.urlencode(params)}"
|
||||
|
||||
def parse_page(self, response_content: dict) -> list[dict]:
|
||||
"""Parse willhaben JSON response into raw ad dicts."""
|
||||
ads_list = (response_content.get("advertSummaryList") or {}).get(
|
||||
"advertSummary", []
|
||||
)
|
||||
|
||||
total_hits = int(response_content.get("rowsFound", 0))
|
||||
|
||||
return ads_list
|
||||
|
||||
def normalize_ad(self, raw_ad: dict) -> dict:
|
||||
"""Convert willhaben ad format to standard schema."""
|
||||
# Extract attributes from the nested format
|
||||
attrs = self._parse_attributes(raw_ad)
|
||||
|
||||
# Extract ID
|
||||
ad_id_raw = raw_ad.get("id", "")
|
||||
|
||||
# Extract title (handle various formats)
|
||||
title_raw = raw_ad.get("title") or raw_ad.get("Title", {})
|
||||
title = title_raw.get("Value", title_raw) if isinstance(title_raw, dict) else str(title_raw or "")
|
||||
|
||||
# Extract price
|
||||
price_str = attrs.get("PRICE_String") or attrs.get("priceString", "")
|
||||
try:
|
||||
price_cents = int(float(price_str.replace(".", ""))) if price_str else None
|
||||
except (ValueError, TypeError):
|
||||
price_cents = None
|
||||
|
||||
# Extract published date
|
||||
pub_str = attrs.get("PUBLISHED_String") or attrs.get("publishedString", "")
|
||||
published_at = None
|
||||
try:
|
||||
published_at = datetime.fromisoformat(pub_str.replace("Z", "+00:00"))
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
# Extract location
|
||||
city = attrs.get("LOCATION_CityName") or ""
|
||||
postcode = attrs.get("LOCATION_ZIP") or ""
|
||||
region = attrs.get("LOCATION_Region") or ""
|
||||
|
||||
return {
|
||||
"id": ad_id_raw,
|
||||
"marketplace": self.name,
|
||||
"title": title.strip(),
|
||||
"price": price_cents,
|
||||
"currency": "EUR",
|
||||
"url": raw_ad.get("linkUrl", ""),
|
||||
"published_at": published_at,
|
||||
"location": {
|
||||
"city": city,
|
||||
"postcode": postcode if postcode else None,
|
||||
"region": region if region else None,
|
||||
},
|
||||
"attributes": attrs, # preserve marketplace-specific fields
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _parse_attributes(ad_dict: dict) -> dict:
|
||||
"""Parse willhaben's nested attribute format into flat dict."""
|
||||
result = {}
|
||||
|
||||
for attr_group in ad_dict.get("attributes", []):
|
||||
if not isinstance(attr_group, dict):
|
||||
continue
|
||||
|
||||
group_name = attr_group.get("name") or ""
|
||||
|
||||
for item in attr_group.get("items", []):
|
||||
key = f"{group_name}_{item['name']}" if group_name else item["name"]
|
||||
result[key] = item.get("valueString", "")
|
||||
|
||||
return result
|
||||
```
|
||||
|
||||
### 4. Update `main.py` scheduler to use the new scraper factory
|
||||
|
||||
Replace direct calls to `fetch_ads(keyword)` with:
|
||||
|
||||
```python
|
||||
from scrapers import get_scriper_by_name
|
||||
|
||||
# At startup:
|
||||
scrapers_config = os.getenv("SCRAPERS", "willhaben").split(",")
|
||||
active_scrapers = [get_scriper_by_name(s) for s in scrapers_config]
|
||||
|
||||
# In scheduler loop:
|
||||
for scraper in active_scrapers:
|
||||
ads_raw, total_hits = await scraper.fetch_ads(keyword, cursor_at=cursor)
|
||||
|
||||
# ... process with existing pipeline (price filter, postcode filter, etc.)
|
||||
```
|
||||
|
||||
### 5. Add `.env.example` configuration for scrapers
|
||||
|
||||
```bash
|
||||
# Marketplace sources to scrape (comma-separated)
|
||||
SCRAPERS=willhaben
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `WillhabenScraper` produces identical output to the current `scraper.py` implementation (no regression in ad extraction)
|
||||
- [ ] Adding a new marketplace requires only: creating one file under `scrapers/`, registering it in `__init__.py`, and listing it in SCRAPERS env var
|
||||
- [ ] The standardized ad schema includes all fields needed by the downstream pipeline (price, postcode, published_at, URL)
|
||||
- [ ] Pagination logic works correctly through the base class for willhaben
|
||||
- [ ] Error handling (retries, timeouts) continues to work with the new abstraction
|
||||
- [ ] All existing bot commands and notifications function identically after refactoring
|
||||
@@ -0,0 +1,777 @@
|
||||
# Task: Test suite with pytest (≥80% coverage)
|
||||
|
||||
## Description
|
||||
|
||||
The project currently has **zero automated tests**. Every change is verified manually by watching logs or sending test messages to the bot. This makes refactoring risky and prevents CI/CD automation.
|
||||
|
||||
This task introduces a comprehensive pytest test suite covering all critical paths: scraper parsing, notification logic, price/postcode filters, retry queue behavior, and scheduler flow. Coverage threshold is set to 80% minimum.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ Tests structure │
|
||||
│ │
|
||||
│ worker/tests/ │
|
||||
│ ├── conftest.py │
|
||||
│ │ (fixtures: mock_pool, mock_bot)│
|
||||
│ ├── test_scraper.py │
|
||||
│ │ (parsing, pagination) │
|
||||
│ ├── test_notifier.py │
|
||||
│ │ (send, retry queue, digest) │
|
||||
│ ├── test_filters.py │
|
||||
│ │ (price, postcode, mute hours) │
|
||||
│ ├── test_scheduler.py │
|
||||
│ │ (cycle flow, shutdown) │
|
||||
│ └── test_health.py │
|
||||
│ (healthcheck endpoint) │
|
||||
│ │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ CI/CD Pipeline (.github/workflows/ci.yml)
|
||||
│ │
|
||||
│ on: push to main, feat/*; pull_request│
|
||||
│ │
|
||||
│ jobs: │
|
||||
│ lint-and-test: │
|
||||
│ └─ python 3.12 │
|
||||
│ ├─ flake8 (error-only) │
|
||||
│ ├─ pytest --cov=src tests/ │
|
||||
│ └─ coverage >= 80% │
|
||||
└───────────────────────────────────────┘
|
||||
|
||||
Test strategy:
|
||||
|
||||
Unit tests (majority):
|
||||
- Isolate each function/method with mocks
|
||||
- Test edge cases: missing fields, NULL prices, empty results
|
||||
- Fast (<1s per test)
|
||||
|
||||
Integration tests (minority):
|
||||
- Real HTTP to willhaben API (cached responses only)
|
||||
- PostgreSQL test container via docker-compose
|
||||
|
||||
Mocking strategy:
|
||||
- Telegram Bot: mock `bot.send_message()` → verify call count + content
|
||||
- Asyncpg pool: mock fetchval/fetch/execute → return canned data
|
||||
- httpx client: use pytest-httpx to intercept and return fixtures
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **pytest over unittest**. Cleaner syntax, better fixture system, easier async support.
|
||||
- **pytest-asyncio** for testing async functions directly without wrapping in `loop.run_until_complete()`.
|
||||
- Adding as test dependency: `pip install pytest pytest-asyncio pytest-cov httpx[socks]`
|
||||
- **Coverage threshold at 80%** enforced via `pyproject.toml`. Failures are actionable (which files/functions need coverage).
|
||||
- **Snapshot testing for HTTP responses**. Save real willhaben API responses as JSON fixtures to avoid live network calls in CI.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add test dependencies and configuration
|
||||
|
||||
In `worker/requirements-test.txt`:
|
||||
```txt
|
||||
pytest>=8.0
|
||||
pytest-asyncio>=0.24
|
||||
pytest-cov>=6.0
|
||||
aioresponses>=0.7 # mock aiohttp responses for health endpoint tests
|
||||
```
|
||||
|
||||
Create `pyproject.toml` in the project root:
|
||||
```toml
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["worker/tests"]
|
||||
asyncio_mode = "auto"
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["worker/src"]
|
||||
omit = [
|
||||
"*/tests/*",
|
||||
"*/migrate.py", # migration runner — tested manually against real DB
|
||||
]
|
||||
|
||||
[tool.coverage.report]
|
||||
fail_under = 80.0
|
||||
show_missing = true
|
||||
exclude_lines = [
|
||||
"pragma: no cover",
|
||||
"if TYPE_CHECKING:",
|
||||
]
|
||||
```
|
||||
|
||||
### 2. Create `worker/tests/conftest.py` (fixtures)
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_pool():
|
||||
"""Mock asyncpg.Pool with fetchval/fetch/execute."""
|
||||
pool = AsyncMock()
|
||||
pool.fetchval = AsyncMock(return_value=None)
|
||||
pool.fetch = AsyncMock(return_value=[])
|
||||
pool.execute = AsyncMock(return_value="DONE 1")
|
||||
return pool
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_bot():
|
||||
"""Mock Telegram Bot instance."""
|
||||
bot = MagicMock()
|
||||
bot.send_message = AsyncMock(return_value=True)
|
||||
bot.get_me = AsyncMock(return_value={"id": "bot_user", "is_bot": True})
|
||||
return bot
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_willhaben_response():
|
||||
"""Realistic willhaben API response for testing."""
|
||||
return {
|
||||
"rowsFound": 45,
|
||||
"advertSummaryList": {
|
||||
"advertSummary": [
|
||||
{
|
||||
"id": "123456789",
|
||||
"title": {"Value": "RTX 3090 Gaming X - Top Zustand"},
|
||||
"linkUrl": "https://www.willhaben.at/iad/markt/123456789-rtx-3090-gaming-x",
|
||||
"attributes": [
|
||||
{
|
||||
"name": "PRICE",
|
||||
"items": [{"name": "priceString", "valueString": "750.00"}]
|
||||
},
|
||||
{
|
||||
"name": "PUBLISHED",
|
||||
"items": [{"name": "publishedString", "valueString": "2026-07-04T12:30:00+02:00"}]
|
||||
},
|
||||
{
|
||||
"name": "LOCATION",
|
||||
"items": [
|
||||
{"name": "CityName", "valueString": "Wien"},
|
||||
{"name": "ZIP", "valueString": "1010"},
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_ad_normalized():
|
||||
"""Expected normalized ad dict from willhaben response."""
|
||||
return {
|
||||
"id": "123456789",
|
||||
"marketplace": "willhaben",
|
||||
"title": "RTX 3090 Gaming X - Top Zustand",
|
||||
"price": 75000, # in cents
|
||||
"currency": "EUR",
|
||||
"url": "https://www.willhaben.at/iad/markt/123456789-rtx-3090-gaming-x",
|
||||
"published_at": ..., # will be datetime object
|
||||
"location": {
|
||||
"city": "Wien",
|
||||
"postcode": "1010",
|
||||
"region": None,
|
||||
},
|
||||
"attributes": {...},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_keyword_row():
|
||||
"""Sample keyword DB row."""
|
||||
return {
|
||||
"id": "kw-uuid-here",
|
||||
"keyword_name": "rtx 3090",
|
||||
"telegram_id": "298181113",
|
||||
"is_active": True,
|
||||
"price_min": 50000, # €500 minimum
|
||||
"price_max": 1000000, # €10000 maximum
|
||||
"allowed_postcodes": ["1010", "1020"],
|
||||
"last_scraped_at": None,
|
||||
"ads_cursor": None,
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Create `worker/tests/test_scraper.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
from scrapers.willhaben import WillhabenScraper
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
class TestWillhabenScraper:
|
||||
|
||||
def test_build_query(self):
|
||||
scraper = WillhabenScraper()
|
||||
url = scraper.build_query("rtx 3090", offset=30)
|
||||
|
||||
assert "keyword=rtx+3090" in url
|
||||
assert "offset=30" in url
|
||||
assert "rows=30" in url
|
||||
|
||||
def test_build_query_default_offset(self):
|
||||
scraper = WillhabenScraper()
|
||||
url = scraper.build_query("gtx 1660")
|
||||
|
||||
assert "offset=0" in url
|
||||
|
||||
def test_parse_page_empty_response(self, sample_willhaben_response):
|
||||
scraper = WillhabenScraper()
|
||||
|
||||
# Empty response
|
||||
empty = {"rowsFound": 0, "advertSummaryList": {"advertSummary": []}}
|
||||
result = scraper.parse_page(empty)
|
||||
|
||||
assert result == []
|
||||
|
||||
def test_normalize_ad(self, sample_willhaben_response):
|
||||
scraper = WillhabenScraper()
|
||||
raw_ad = sample_willhaben_response["advertSummaryList"]["advertSummary"][0]
|
||||
ad = scraper.normalize_ad(raw_ad)
|
||||
|
||||
assert ad["id"] == "123456789"
|
||||
assert ad["marketplace"] == "willhaben"
|
||||
assert ad["price"] == 75000 # cents
|
||||
assert ad["location"]["postcode"] == "1010"
|
||||
|
||||
def test_normalize_ad_missing_price(self):
|
||||
scraper = WillhabenScraper()
|
||||
raw_ad = {
|
||||
"id": "no-price",
|
||||
"title": {"Value": "Free RTX"},
|
||||
"linkUrl": "https://example.com",
|
||||
"attributes": [], # no price
|
||||
}
|
||||
ad = scraper.normalize_ad(raw_ad)
|
||||
|
||||
assert ad["price"] is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_ads_pagination(self):
|
||||
scraper = WillhabenScraper()
|
||||
|
||||
with patch.object(scraper, '_fetch_with_retry') as mock_fetch:
|
||||
# Simulate 2 pages of results
|
||||
page1 = {
|
||||
"rowsFound": 45,
|
||||
"advertSummaryList": {"advertSummary": [
|
||||
{"id": f"ad{i}", "title": {"Value": f"Ad {i}"},
|
||||
"linkUrl": f"https://example.com/{i}",
|
||||
"attributes": [{"name": "PUBLISHED", "items": [
|
||||
{"name": "publishedString",
|
||||
"valueString": "2026-07-04T15:30:00+02:00"}]}]},
|
||||
] for i in range(3)}]
|
||||
}
|
||||
|
||||
page2 = {
|
||||
"rowsFound": 45,
|
||||
"advertSummaryList": {"advertSummary": [
|
||||
{"id": f"ad{i}", "title": {"Value": f"Ad {i}"},
|
||||
"linkUrl": f"https://example.com/{i}",
|
||||
"attributes": [{"name": "PUBLISHED", "items": [
|
||||
{"name": "publishedString",
|
||||
"valueString": "2026-07-04T15:30:00+02:00"}]}]},
|
||||
] for i in range(3, 6)}]
|
||||
}
|
||||
|
||||
mock_fetch.side_effect = [page1, page2]
|
||||
|
||||
ads, total = await scraper.fetch_ads("test keyword", max_pages=2)
|
||||
|
||||
assert len(ads) == 6 # 3 from each page
|
||||
|
||||
|
||||
class TestScraperBase:
|
||||
|
||||
def test_headers_default(self):
|
||||
scraper = WillhabenScraper()
|
||||
headers = scraper.headers
|
||||
|
||||
assert "Accept" in headers
|
||||
assert "User-Agent" in headers
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_with_retry_exhausts_retries(self):
|
||||
import httpx
|
||||
|
||||
scraper = WillhabenScraper()
|
||||
client = AsyncMock()
|
||||
client.get = AsyncMock(side_effect=httpx.ConnectError("connection refused"))
|
||||
|
||||
with pytest.raises(httpx.ConnectError):
|
||||
await scraper._fetch_with_retry(client, "http://example.com", max_retries=2)
|
||||
```
|
||||
|
||||
### 4. Create `worker/tests/test_filters.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
|
||||
class TestPriceFilters:
|
||||
|
||||
async def test_pass_when_no_min(self):
|
||||
"""Ad passes when no price minimum is set."""
|
||||
# Import the actual function being tested
|
||||
from notifier import _extract_price # or wherever it lives
|
||||
|
||||
kw_row = {"price_min": None, "price_max": None}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "PRICE", "items": [
|
||||
{"name": "priceString", "valueString": "50.00"}]}]
|
||||
}
|
||||
|
||||
# The check function (to be implemented in main.py)
|
||||
from ..main import _check_price_filters # actual implementation
|
||||
|
||||
result = await _check_price_filters(ad_dict, kw_row)
|
||||
assert result is True
|
||||
|
||||
async def test_pass_when_no_max(self):
|
||||
kw_row = {"price_min": 1000, "price_max": None}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "PRICE", "items": [
|
||||
{"name": "priceString", "valueString": "100.00"}]}]
|
||||
}
|
||||
|
||||
from ..main import _check_price_filters
|
||||
result = await _check_price_filters(ad_dict, kw_row)
|
||||
assert result is True
|
||||
|
||||
async def test_fail_below_min(self):
|
||||
kw_row = {"price_min": 50000, "price_max": None} # €500 min
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "PRICE", "items": [
|
||||
{"name": "priceString", "valueString": "10.00"}]}] # €10 — too cheap
|
||||
}
|
||||
|
||||
from ..main import _check_price_filters
|
||||
result = await _check_price_filters(ad_dict, kw_row)
|
||||
assert result is False
|
||||
|
||||
async def test_fail_above_max(self):
|
||||
kw_row = {"price_min": None, "price_max": 1000} # €10 max
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "PRICE", "items": [
|
||||
{"name": "priceString", "valueString": "500.00"}]}] # €500 — too expensive
|
||||
}
|
||||
|
||||
from ..main import _check_price_filters
|
||||
result = await _check_price_filters(ad_dict, kw_row)
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestPostcodeFilters:
|
||||
|
||||
async def test_pass_when_no_filter(self):
|
||||
kw_row = {"allowed_postcodes": None}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "LOCATION", "items": [
|
||||
{"name": "ZIP", "valueString": "1010"}]}]
|
||||
}
|
||||
|
||||
from ..main import _check_postcode_filter
|
||||
result = await _check_postcode_filter(ad_dict, kw_row)
|
||||
assert result is True
|
||||
|
||||
async def test_pass_when_matching(self):
|
||||
kw_row = {"allowed_postcodes": ["1010", "1020"]}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "LOCATION", "items": [
|
||||
{"name": "ZIP", "valueString": "1010"}]}]
|
||||
}
|
||||
|
||||
from ..main import _check_postcode_filter
|
||||
result = await _check_postcode_filter(ad_dict, kw_row)
|
||||
assert result is True
|
||||
|
||||
async def test_fail_when_not_matching(self):
|
||||
kw_row = {"allowed_postcodes": ["1010", "1020"]}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "LOCATION", "items": [
|
||||
{"name": "ZIP", "valueString": "8010"}]}] # Graz — not in allowed list
|
||||
}
|
||||
|
||||
from ..main import _check_postcode_filter
|
||||
result = await _check_postcode_filter(ad_dict, kw_row)
|
||||
assert result is False
|
||||
|
||||
async def test_fail_when_no_postcode_in_ad(self):
|
||||
kw_row = {"allowed_postcodes": ["1010", "1020"]}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [] # no location info
|
||||
}
|
||||
|
||||
from ..main import _check_postcode_filter
|
||||
result = await _check_postcode_filter(ad_dict, kw_row)
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestMuteHours:
|
||||
|
||||
async def test_no_mute_when_not_configured(self, mock_pool):
|
||||
mock_pool.fetchrow.return_value = None
|
||||
|
||||
from ..notifier import _is_in_mute_hours
|
||||
result = await _is_in_mute_hours("298181113", mock_pool)
|
||||
|
||||
assert result is False
|
||||
|
||||
async def test_no_mute_outside_window(self, mock_pool):
|
||||
from datetime import time
|
||||
|
||||
mock_pool.fetchrow.return_value = {
|
||||
"mute_start": time(22, 0), # 10 PM
|
||||
"mute_end": time(7, 0), # 7 AM
|
||||
}
|
||||
|
||||
# Mock current time to noon UTC (outside mute window)
|
||||
with patch("notifier.datetime") as mock_dt:
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_dt.now.return_value = datetime(2026, 1, 1, 12, 0, tzinfo=timezone.utc)
|
||||
|
||||
from ..notifier import _is_in_mute_hours
|
||||
result = await _is_in_mute_hours("298181113", mock_pool)
|
||||
|
||||
assert result is False
|
||||
|
||||
async def test_muted_during_window(self, mock_pool):
|
||||
from datetime import time
|
||||
|
||||
mock_pool.fetchrow.return_value = {
|
||||
"mute_start": time(22, 0),
|
||||
"mute_end": time(7, 0),
|
||||
}
|
||||
|
||||
with patch("notifier.datetime") as mock_dt:
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_dt.now.return_value = datetime(2026, 1, 1, 3, 0, tzinfo=timezone.utc)
|
||||
|
||||
from ..notifier import _is_in_mute_hours
|
||||
result = await _is_in_mute_hours("298181113", mock_pool)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestNotificationQueue:
|
||||
|
||||
async def test_enqueue_on_failure(self, mock_pool):
|
||||
from ..notifier import _enqueue_retry
|
||||
|
||||
await _enqueue_retry(
|
||||
ad_id="ad-uuid-here",
|
||||
telegram_id="298181113",
|
||||
message_text="Test notification",
|
||||
notif_type="new",
|
||||
error_msg="Telegram API timeout",
|
||||
)
|
||||
|
||||
mock_pool.execute.assert_called_once()
|
||||
|
||||
async def test_no_duplicate_enqueue(self, mock_pool):
|
||||
"""Same ad+user should not create duplicate queue entries."""
|
||||
mock_pool.fetchval.return_value = "already-exists-uuid"
|
||||
|
||||
from ..notifier import _enqueue_retry
|
||||
|
||||
await _enqueue_retry(
|
||||
ad_id="ad-uuid-here",
|
||||
telegram_id="298181113",
|
||||
message_text="Test",
|
||||
notif_type="new",
|
||||
error_msg="timeout",
|
||||
)
|
||||
|
||||
# fetchval called (to check), but execute NOT called (no insert)
|
||||
mock_pool.fetchval.assert_called_once()
|
||||
mock_pool.execute.assert_not_called()
|
||||
```
|
||||
|
||||
### 5. Create `worker/tests/test_scheduler.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
|
||||
class TestSchedulerFlow:
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_process_notification_queue_retries_pending(self, mock_pool):
|
||||
"""Pending items should be retried if backoff period has elapsed."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_pool.fetch.return_value = [
|
||||
{
|
||||
"id": "queue-item-1",
|
||||
"ad_id": "ad-uuid",
|
||||
"telegram_id": "298181113",
|
||||
"message_text": "Retry this ad",
|
||||
"type": "new",
|
||||
"attempts": 0,
|
||||
"max_attempts": 5,
|
||||
"last_error": "timeout",
|
||||
"updated_at": datetime(2026, 7, 4, 10, 0, tzinfo=timezone.utc),
|
||||
},
|
||||
]
|
||||
|
||||
with patch("main.get_application_bot") as mock_get_bot:
|
||||
mock_bot = AsyncMock()
|
||||
mock_bot.send_message = AsyncMock()
|
||||
mock_get_bot.return_value = mock_bot
|
||||
|
||||
from ..main import process_notification_queue
|
||||
result = await process_notification_queue()
|
||||
|
||||
assert result == 1
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_process_notification_queue_dead_after_max_attempts(self, mock_pool):
|
||||
"""Items exceeding max_attempts should be marked as dead."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_pool.fetch.return_value = [
|
||||
{
|
||||
"id": "queue-item-2",
|
||||
"ad_id": "ad-uuid",
|
||||
"telegram_id": "298181113",
|
||||
"message_text": "Will fail again",
|
||||
"type": "new",
|
||||
"attempts": 5, # already at max
|
||||
"max_attempts": 5,
|
||||
"last_error": "user blocked bot",
|
||||
"updated_at": datetime(2026, 7, 4, 10, 0, tzinfo=timezone.utc),
|
||||
},
|
||||
]
|
||||
|
||||
with patch("main.get_application_bot") as mock_get_bot:
|
||||
import telegram.error
|
||||
mock_bot = AsyncMock()
|
||||
mock_bot.send_message = AsyncMock(
|
||||
side_effect=telegram.error.TelegramError("blocked")
|
||||
)
|
||||
mock_get_bot.return_value = mock_bot
|
||||
|
||||
from ..main import process_notification_queue
|
||||
await process_notification_queue()
|
||||
|
||||
# Should have updated to 'dead' status
|
||||
calls = [c[0] for c in mock_pool.execute.call_args_list]
|
||||
assert any("status = 'dead'" in str(c) or "status=$2" in str(c)
|
||||
for c in calls), "Item should be marked as dead"
|
||||
|
||||
|
||||
class TestDigestFlushing:
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_flush_digest_buffers(self, mock_pool):
|
||||
"""Buffered items older than interval should be flushed."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_pool.fetch.side_effect = [
|
||||
# First fetch: get digest-enabled users
|
||||
[{"telegram_id": "298181113", "digest_interval": 60}],
|
||||
# Second fetch: get buffered items
|
||||
[
|
||||
{"id": "buf-1", "keyword": "rtx 3090",
|
||||
"title": "Ad 1", "price": 75000,
|
||||
"url": "https://...", "ad_id": "ad-uuid"},
|
||||
{"id": "buf-2", "keyword": "rtx 3090",
|
||||
"title": "Ad 2", "price": 68000,
|
||||
"url": "https://...", "ad_id": "ad-uuid-2"},
|
||||
],
|
||||
]
|
||||
|
||||
with patch("main.get_application_bot") as mock_get_bot:
|
||||
mock_bot = AsyncMock()
|
||||
mock_bot.send_message = AsyncMock()
|
||||
mock_get_bot.return_value = mock_bot
|
||||
|
||||
from ..main import flush_digest_buffers
|
||||
result = await flush_digest_buffers()
|
||||
|
||||
assert result == 1 # one digest sent
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_flush_empty_buffer(self, mock_pool):
|
||||
"""No buffered items should result in no action."""
|
||||
mock_pool.fetch.side_effect = [
|
||||
[{"telegram_id": "298181113", "digest_interval": 60}],
|
||||
[], # empty buffer
|
||||
]
|
||||
|
||||
from ..main import flush_digest_buffers
|
||||
result = await flush_digest_buffers()
|
||||
|
||||
assert result == 0
|
||||
|
||||
|
||||
class TestGracefulShutdown:
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cleanup_stops_scheduler_and_closes_pool(self):
|
||||
"""Cleanup should cancel scheduler, stop bot, close DB pool."""
|
||||
from telegram.ext import Application
|
||||
|
||||
app = AsyncMock(spec=Application)
|
||||
app.updater.running = True
|
||||
|
||||
with patch("main._scheduler_task") as mock_task:
|
||||
mock_task.done.return_value = False
|
||||
|
||||
from ..main import cleanup
|
||||
await cleanup(app)
|
||||
|
||||
mock_task.cancel.assert_called_once()
|
||||
app.updater.stop_polling.assert_called_once()
|
||||
```
|
||||
|
||||
### 6. Create `worker/tests/test_health.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_endpoint_ok():
|
||||
"""Health endpoint should return 200 when system is healthy."""
|
||||
from health import create_health_app
|
||||
|
||||
app = create_health_app()
|
||||
|
||||
with patch("health.get_pool") as mock_pool_get:
|
||||
mock_pool = AsyncMock()
|
||||
mock_pool.fetchval = AsyncMock(return_value=1)
|
||||
mock_pool_get.return_value = mock_pool
|
||||
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
|
||||
try:
|
||||
from aiohttp.test_utils import TestClient, TestServer
|
||||
client = TestClient(TestServer(runner))
|
||||
async with client:
|
||||
resp = await client.get("/health")
|
||||
assert resp.status == 200
|
||||
|
||||
data = await resp.json()
|
||||
assert data["status"] == "ok"
|
||||
assert "db_connected" in data
|
||||
finally:
|
||||
await runner.cleanup()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_endpoint_unhealthy_db():
|
||||
"""Health endpoint should return 503 when DB is unreachable."""
|
||||
from health import create_health_app
|
||||
|
||||
app = create_health_app()
|
||||
|
||||
with patch("health.get_pool") as mock_pool_get:
|
||||
mock_pool_get.side_effect = Exception("connection refused")
|
||||
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
|
||||
try:
|
||||
from aiohttp.test_utils import TestClient, TestServer
|
||||
client = TestClient(TestServer(runner))
|
||||
async with client:
|
||||
resp = await client.get("/health")
|
||||
assert resp.status == 503
|
||||
|
||||
data = await resp.json()
|
||||
assert data["status"] == "unhealthy"
|
||||
finally:
|
||||
await runner.cleanup()
|
||||
```
|
||||
|
||||
### 7. Create GitHub Actions workflow `.github/workflows/ci.yml`
|
||||
|
||||
```yaml
|
||||
name: CI — Lint & Test
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main, 'feat/*']
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
lint-and-test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install --upgrade pip
|
||||
pip install -r worker/requirements.txt
|
||||
pip install -r worker/requirements-test.txt
|
||||
|
||||
- name: Lint with flake8
|
||||
run: |
|
||||
flake8 worker/src/ --count --select=E9,F63,F7,F82 --show-source --statistics
|
||||
flake8 worker/tests/ --count --select=E9,F63,F7,F82 --show-source --statistics
|
||||
|
||||
- name: Test with pytest + coverage
|
||||
run: |
|
||||
cd worker
|
||||
python -m pytest tests/ --cov=src --cov-report=term-missing --cov-fail-under=80
|
||||
|
||||
build-docker:
|
||||
runs-on: ubuntu-latest
|
||||
needs: lint-and-test
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Build Docker image
|
||||
run: docker compose -f docker-compose.yml build worker
|
||||
|
||||
- name: Test healthcheck
|
||||
run: |
|
||||
docker compose up -d worker
|
||||
sleep 5
|
||||
docker inspect --format='{{.State.Health.Status}}' willhaben-tracker-worker-1 || true
|
||||
docker compose down
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `pytest` runs with 0 failures and ≥80% coverage on all source files
|
||||
- [ ] GitHub Actions pipeline passes on every push to `main` and feature branches
|
||||
- [ ] Tests cover: scraper parsing, pagination, price filters, postcode filters, mute hours, notification retry queue, digest flushing, graceful shutdown, healthcheck endpoint
|
||||
- [ ] Flake8 linting (error-level checks) passes in CI
|
||||
- [ ] Docker image builds successfully after all tests pass
|
||||
- [ ] Adding a new test file automatically includes it in the coverage report
|
||||
Reference in New Issue
Block a user