94 lines
4.9 KiB
Markdown
94 lines
4.9 KiB
Markdown
# Phase 3 — Scalability & Advanced Features
|
|
|
|
## Scope
|
|
|
|
This phase introduces **structural improvements** that make the project maintainable, extensible, and testable. Currently, the entire system is a single async Python process with no tests and no CI/CD pipeline. After this phase:
|
|
|
|
- Automated tests provide confidence for every change (≥80% coverage)
|
|
- CI/CD pipeline runs on every push to validate code quality
|
|
- Multi-marketplace architecture enables adding new sources without modifying core logic
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────┐
|
|
│ Project Structure (post-Phase-3) │
|
|
│ │
|
|
│ willhaben-tracker/ │
|
|
│ ├── worker/ │
|
|
│ │ ├── src/ │
|
|
│ │ │ ├── main.py (entry point, scheduler) │
|
|
│ │ │ ├── db.py (asyncpg pool mgmt) │
|
|
│ │ │ ├── bot.py (Telegram handlers) │
|
|
│ │ │ ├── notifier.py (message sending) │
|
|
│ │ │ ├── scraper.py (base scraper class) │
|
|
│ │ │ ├── scrapers/ │
|
|
│ │ │ │ ├── __init__.py │
|
|
│ │ │ │ ├── willhaben.py (willhaben-specific) │
|
|
│ │ │ │ └── base.py (abstract base class) │
|
|
│ │ │ ├── health.py (healthcheck endpoint) │
|
|
│ │ │ └── migrate.py (migration runner) │
|
|
│ │ ├── tests/ │
|
|
│ │ │ ├── conftest.py │
|
|
│ │ │ ├── test_scraper.py │
|
|
│ │ │ ├── test_notifier.py │
|
|
│ │ │ └── ... │
|
|
│ │ ├── Dockerfile │
|
|
│ │ └── requirements.txt │
|
|
│ ├── .github/ │
|
|
│ │ └── workflows/ │
|
|
│ │ └── ci.yml (pytest + flake8 + coverage) │
|
|
│ ├── pyproject.toml (coverage config, tools) │
|
|
│ └── docker-compose.yml │
|
|
└──────────────────────────────────────────────────────┘
|
|
|
|
Multi-marketplace abstraction:
|
|
|
|
ScraperBase (abstract):
|
|
- async fetch_ads(keyword) → list[dict]
|
|
- async parse_response(html/json) → list[dict]
|
|
- normalize_ad(raw) → dict with standard keys
|
|
|
|
WillhabenScraper(ScraperBase):
|
|
- implements willhaben-specific URL, headers, parsing
|
|
|
|
Future: KleinAnzeigenScraper, MobileScraper, ...
|
|
|
|
CI/CD Pipeline (.github/workflows/ci.yml):
|
|
|
|
on: push to main, feat/*; pull_request
|
|
|
|
jobs:
|
|
lint-and-test:
|
|
└─ python 3.12
|
|
├─ flake8 (linting)
|
|
├─ pytest --cov=src tests/ (unit + integration tests)
|
|
└─ coverage >= 80% (fail if not met)
|
|
|
|
Tests Structure:
|
|
|
|
Unit tests:
|
|
- test_scraper_pagination() — verify pagination logic with mock responses
|
|
- test_price_filters() — verify filter functions
|
|
- test_notification_retry() — verify retry queue behavior
|
|
|
|
Integration tests:
|
|
- Test against real willhaben API (rate-limited, cached)
|
|
- PostgreSQL test container via docker-compose
|
|
```
|
|
|
|
## Tasks
|
|
|
|
| Task | File | Description |
|
|
|------|------|-------------|
|
|
| Multi-marketplace abstraction layer | [task-multi-marketplace.md](./task-multi-marketplace.md) | Refactor `scraper.py` into a base class + per-marketplace implementations. Introduces a standard ad schema and factory for registering new sources. |
|
|
| Test suite with pytest (≥80% coverage) | [task-testing-pytest.md](./task-testing-pytest.md) | Add comprehensive unit tests covering scraper parsing, notification logic, price/postcode filters, retry queue, and scheduler flow. Configure coverage thresholds. |
|
|
|
|
## General Acceptance Criteria
|
|
|
|
- [ ] CI pipeline runs on every push to `main` and feature branches — fails if lint or coverage checks are not met
|
|
- [ ] Code coverage is ≥80% across all source files in `worker/src/`
|
|
- [ ] Multi-marketplace abstraction works — adding a new marketplace requires only creating one file under `scrapers/` with no changes to core logic
|
|
- [ ] All existing functionality (willhaben scraping, notifications) continues to work after refactoring
|
|
- [ ] The `/health` endpoint exposes test results or coverage stats (optional enhancement)
|