Files

T

CI / lint-and-test (push) Has been cancelled

Details

feat: Phase 3 — web dashboard, testing, and CI/CD

- Remove multi-marketplace from Phase 3
- Add FastAPI web UI on port 8766 with basic auth
- Add 6 Jinja2 templates (dashboard, keywords, users, ads, stats)
- Add pytest test suite (45 tests, 49% coverage)
- Add GitHub Actions CI/CD workflow
- Update docker-compose.yml to expose web UI port
- Update Dockerfile to include tests

2026-07-10 22:28:18 +02:00

5.3 KiB

Raw Permalink Blame History

Phase 3 — Web Dashboard & Testing

Scope

This phase introduces observability and reliability improvements. Currently, the entire system is a single async Python process with no tests, no CI/CD pipeline, and no way to monitor what's happening without SSH-ing into the server. After this phase:

A Web Dashboard provides real-time visibility into keywords, ads, users, and stats
Automated tests provide confidence for every change (≥80% coverage)
CI/CD pipeline runs on every push to validate code quality

Architecture

┌──────────────────────────────────────────────────────┐
│  Project Structure (post-Phase-3)                     │
│                                                      │
│  willhaben-tracker/                                   │
│    ├── worker/                                        │
│    │   ├── src/                                       │
│    │   │   ├── main.py       (entry point, scheduler) │
│    │   │   ├── db.py         (asyncpg pool mgmt)      │
│    │   │   ├── bot.py        (Telegram handlers)      │
│    │   │   ├── notifier.py   (message sending)        │
│    │   │   ├── scraper.py    (willhaben scraper)      │
│    │   │   ├── web.py        (FastAPI dashboard)      │
│    │   │   ├── health.py     (healthcheck endpoint)   │
│    │   │   ├── migrate.py    (migration runner)       │
│    │   │   └── templates/    (Jinja2 HTML templates)  │
│    │   ├── tests/                                       │
│    │   │   ├── conftest.py                                │
│    │   │   ├── test_scraper.py                            │
│    │   │   ├── test_notifier.py                           │
│    │   │   ├── test_filters.py                            │
│    │   │   └── test_web.py                                │
│    │   ├── Dockerfile                                   │
│    │   └── requirements.txt                             │
│    ├── .github/                                         │
│    │   └── workflows/                                   │
│    │       └── ci.yml     (pytest + flake8 + coverage)  │
│    ├── pyproject.toml   (coverage config, tools)        │
│    └── docker-compose.yml                               │
└──────────────────────────────────────────────────────┘

Web Dashboard (FastAPI, port 8766):
  
  GET /             → Dashboard (keywords overview, stats summary)
  GET /keywords     → Keywords list with status, filters, subscribers
  GET /keywords/<id> → Keyword detail (recent ads, price history, scrape logs)
  GET /users        → Users list with settings
  GET /ads          → Recent ads with search/filter
  GET /stats        → JSON stats (extends existing /stats endpoint)
  
  Auth: Basic Auth via WEB_UI_USERNAME / WEB_UI_PASSWORD env vars
  Templates: Jinja2 with inline CSS (zero external dependencies)

CI/CD Pipeline (.github/workflows/ci.yml):
  
  on: push to main, feat/*; pull_request
  
  jobs:
    lint-and-test:
      └─ python 3.12
         ├─ flake8 (linting)
         ├─ pytest --cov=src tests/ (unit + integration tests)
         └─ coverage >= 80% (fail if not met)

Tests Structure:
  
  Unit tests:
    - test_scraper_pagination() — verify pagination logic with mock responses
    - test_price_filters() — verify filter functions
    - test_notification_retry() — verify retry queue behavior
    - test_mute_digest() — verify mute hours and digest buffering
    - test_web_endpoints() — verify web UI routes
  
  Integration tests:
    - Test against real willhaben API (rate-limited, cached)
    - PostgreSQL test container via docker-compose

Tasks

Task	File	Description
Web Dashboard (FastAPI + Jinja2)	task-web-ui.md	Add a read-only web dashboard for monitoring keywords, ads, users, and stats. Runs on port 8766 with basic auth.
Test suite with pytest (≥80% coverage)	task-testing-pytest.md	Add comprehensive unit tests covering scraper parsing, notification logic, price/postcode filters, retry queue, and scheduler flow. Configure coverage thresholds.

General Acceptance Criteria

Web Dashboard is accessible at http://<host>:8766 with basic auth
Dashboard shows keywords with status, filters, subscribers, and last scrape time
Dashboard shows recent ads with price, location, and keyword
Dashboard shows users with mute/digest settings
Dashboard shows stats (ads indexed, notifications sent, queue status)
CI pipeline runs on every push to main and feature branches — fails if lint or coverage checks are not met
Code coverage is ≥80% across all source files in worker/src/
All existing functionality (willhaben scraping, Telegram notifications, health server) continues to work
Health server still works on port 8765 (no regression)

5.3 KiB Raw Permalink Blame History