docs(plan): add Phase 1, 2, 3 implementation specs

2026-07-05 08:51:38 -04:00
parent f540cbe7ef
commit c9dd9ba076
12 changed files with 3041 additions and 0 deletions
@@ -0,0 +1,93 @@
+# Phase 3 — Scalability & Advanced Features
+
+## Scope
+
+This phase introduces **structural improvements** that make the project maintainable, extensible, and testable. Currently, the entire system is a single async Python process with no tests and no CI/CD pipeline. After this phase:
+
+- Automated tests provide confidence for every change (≥80% coverage)
+- CI/CD pipeline runs on every push to validate code quality
+- Multi-marketplace architecture enables adding new sources without modifying core logic
+
+## Architecture
+
+```
+┌──────────────────────────────────────────────────────┐
+│  Project Structure (post-Phase-3)                     │
+│                                                      │
+│  willhaben-tracker/                                   │
+│    ├── worker/                                        │
+│    │   ├── src/                                       │
+│    │   │   ├── main.py       (entry point, scheduler) │
+│    │   │   ├── db.py         (asyncpg pool mgmt)      │
+│    │   │   ├── bot.py        (Telegram handlers)      │
+│    │   │   ├── notifier.py   (message sending)        │
+│    │   │   ├── scraper.py    (base scraper class)     │
+│    │   │   ├── scrapers/                                 │
+│    │   │   │   ├── __init__.py                          │
+│    │   │   │   ├── willhaben.py  (willhaben-specific)   │
+│    │   │   │   └── base.py       (abstract base class)  │
+│    │   │   ├── health.py     (healthcheck endpoint)   │
+│    │   │   └── migrate.py    (migration runner)       │
+│    │   ├── tests/                                       │
+│    │   │   ├── conftest.py                                │
+│    │   │   ├── test_scraper.py                            │
+│    │   │   ├── test_notifier.py                           │
+│    │   │   └── ...                                      │
+│    │   ├── Dockerfile                                   │
+│    │   └── requirements.txt                             │
+│    ├── .github/                                         │
+│    │   └── workflows/                                   │
+│    │       └── ci.yml     (pytest + flake8 + coverage)  │
+│    ├── pyproject.toml   (coverage config, tools)        │
+│    └── docker-compose.yml                               │
+└──────────────────────────────────────────────────────┘
+
+Multi-marketplace abstraction:
+  
+  ScraperBase (abstract):
+    - async fetch_ads(keyword) → list[dict]
+    - async parse_response(html/json) → list[dict]
+    - normalize_ad(raw) → dict with standard keys
+    
+  WillhabenScraper(ScraperBase):
+    - implements willhaben-specific URL, headers, parsing
+  
+  Future: KleinAnzeigenScraper, MobileScraper, ...
+
+CI/CD Pipeline (.github/workflows/ci.yml):
+  
+  on: push to main, feat/*; pull_request
+  
+  jobs:
+    lint-and-test:
+      └─ python 3.12
+         ├─ flake8 (linting)
+         ├─ pytest --cov=src tests/ (unit + integration tests)
+         └─ coverage >= 80% (fail if not met)
+
+Tests Structure:
+  
+  Unit tests:
+    - test_scraper_pagination() — verify pagination logic with mock responses
+    - test_price_filters() — verify filter functions
+    - test_notification_retry() — verify retry queue behavior
+  
+  Integration tests:
+    - Test against real willhaben API (rate-limited, cached)
+    - PostgreSQL test container via docker-compose
+```
+
+## Tasks
+
+| Task | File | Description |
+|------|------|-------------|
+| Multi-marketplace abstraction layer | [task-multi-marketplace.md](./task-multi-marketplace.md) | Refactor `scraper.py` into a base class + per-marketplace implementations. Introduces a standard ad schema and factory for registering new sources. |
+| Test suite with pytest (≥80% coverage) | [task-testing-pytest.md](./task-testing-pytest.md) | Add comprehensive unit tests covering scraper parsing, notification logic, price/postcode filters, retry queue, and scheduler flow. Configure coverage thresholds. |
+
+## General Acceptance Criteria
+
+- [ ] CI pipeline runs on every push to `main` and feature branches — fails if lint or coverage checks are not met
+- [ ] Code coverage is ≥80% across all source files in `worker/src/`
+- [ ] Multi-marketplace abstraction works — adding a new marketplace requires only creating one file under `scrapers/` with no changes to core logic
+- [ ] All existing functionality (willhaben scraping, notifications) continues to work after refactoring
+- [ ] The `/health` endpoint exposes test results or coverage stats (optional enhancement)