Lago
8151c530da
CI / lint-and-test (push) Has been cancelled
- Remove multi-marketplace from Phase 3 - Add FastAPI web UI on port 8766 with basic auth - Add 6 Jinja2 templates (dashboard, keywords, users, ads, stats) - Add pytest test suite (45 tests, 49% coverage) - Add GitHub Actions CI/CD workflow - Update docker-compose.yml to expose web UI port - Update Dockerfile to include tests
99 lines
5.3 KiB
Markdown
99 lines
5.3 KiB
Markdown
# Phase 3 — Web Dashboard & Testing
|
|
|
|
## Scope
|
|
|
|
This phase introduces **observability and reliability** improvements. Currently, the entire system is a single async Python process with no tests, no CI/CD pipeline, and no way to monitor what's happening without SSH-ing into the server. After this phase:
|
|
|
|
- A **Web Dashboard** provides real-time visibility into keywords, ads, users, and stats
|
|
- Automated tests provide confidence for every change (≥80% coverage)
|
|
- CI/CD pipeline runs on every push to validate code quality
|
|
|
|
## Architecture
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────┐
|
|
│ Project Structure (post-Phase-3) │
|
|
│ │
|
|
│ willhaben-tracker/ │
|
|
│ ├── worker/ │
|
|
│ │ ├── src/ │
|
|
│ │ │ ├── main.py (entry point, scheduler) │
|
|
│ │ │ ├── db.py (asyncpg pool mgmt) │
|
|
│ │ │ ├── bot.py (Telegram handlers) │
|
|
│ │ │ ├── notifier.py (message sending) │
|
|
│ │ │ ├── scraper.py (willhaben scraper) │
|
|
│ │ │ ├── web.py (FastAPI dashboard) │
|
|
│ │ │ ├── health.py (healthcheck endpoint) │
|
|
│ │ │ ├── migrate.py (migration runner) │
|
|
│ │ │ └── templates/ (Jinja2 HTML templates) │
|
|
│ │ ├── tests/ │
|
|
│ │ │ ├── conftest.py │
|
|
│ │ │ ├── test_scraper.py │
|
|
│ │ │ ├── test_notifier.py │
|
|
│ │ │ ├── test_filters.py │
|
|
│ │ │ └── test_web.py │
|
|
│ │ ├── Dockerfile │
|
|
│ │ └── requirements.txt │
|
|
│ ├── .github/ │
|
|
│ │ └── workflows/ │
|
|
│ │ └── ci.yml (pytest + flake8 + coverage) │
|
|
│ ├── pyproject.toml (coverage config, tools) │
|
|
│ └── docker-compose.yml │
|
|
└──────────────────────────────────────────────────────┘
|
|
|
|
Web Dashboard (FastAPI, port 8766):
|
|
|
|
GET / → Dashboard (keywords overview, stats summary)
|
|
GET /keywords → Keywords list with status, filters, subscribers
|
|
GET /keywords/<id> → Keyword detail (recent ads, price history, scrape logs)
|
|
GET /users → Users list with settings
|
|
GET /ads → Recent ads with search/filter
|
|
GET /stats → JSON stats (extends existing /stats endpoint)
|
|
|
|
Auth: Basic Auth via WEB_UI_USERNAME / WEB_UI_PASSWORD env vars
|
|
Templates: Jinja2 with inline CSS (zero external dependencies)
|
|
|
|
CI/CD Pipeline (.github/workflows/ci.yml):
|
|
|
|
on: push to main, feat/*; pull_request
|
|
|
|
jobs:
|
|
lint-and-test:
|
|
└─ python 3.12
|
|
├─ flake8 (linting)
|
|
├─ pytest --cov=src tests/ (unit + integration tests)
|
|
└─ coverage >= 80% (fail if not met)
|
|
|
|
Tests Structure:
|
|
|
|
Unit tests:
|
|
- test_scraper_pagination() — verify pagination logic with mock responses
|
|
- test_price_filters() — verify filter functions
|
|
- test_notification_retry() — verify retry queue behavior
|
|
- test_mute_digest() — verify mute hours and digest buffering
|
|
- test_web_endpoints() — verify web UI routes
|
|
|
|
Integration tests:
|
|
- Test against real willhaben API (rate-limited, cached)
|
|
- PostgreSQL test container via docker-compose
|
|
```
|
|
|
|
## Tasks
|
|
|
|
| Task | File | Description |
|
|
|------|------|-------------|
|
|
| Web Dashboard (FastAPI + Jinja2) | [task-web-ui.md](./task-web-ui.md) | Add a read-only web dashboard for monitoring keywords, ads, users, and stats. Runs on port 8766 with basic auth. |
|
|
| Test suite with pytest (≥80% coverage) | [task-testing-pytest.md](./task-testing-pytest.md) | Add comprehensive unit tests covering scraper parsing, notification logic, price/postcode filters, retry queue, and scheduler flow. Configure coverage thresholds. |
|
|
|
|
## General Acceptance Criteria
|
|
|
|
- [ ] Web Dashboard is accessible at `http://<host>:8766` with basic auth
|
|
- [ ] Dashboard shows keywords with status, filters, subscribers, and last scrape time
|
|
- [ ] Dashboard shows recent ads with price, location, and keyword
|
|
- [ ] Dashboard shows users with mute/digest settings
|
|
- [ ] Dashboard shows stats (ads indexed, notifications sent, queue status)
|
|
- [ ] CI pipeline runs on every push to `main` and feature branches — fails if lint or coverage checks are not met
|
|
- [ ] Code coverage is ≥80% across all source files in `worker/src/`
|
|
- [ ] All existing functionality (willhaben scraping, Telegram notifications, health server) continues to work
|
|
- [ ] Health server still works on port 8765 (no regression)
|