docs(plan): add Phase 1, 2, 3 implementation specs
This commit is contained in:
hermes
@@ -0,0 +1,93 @@
|
||||
# Phase 3 — Scalability & Advanced Features
|
||||
|
||||
## Scope
|
||||
|
||||
This phase introduces **structural improvements** that make the project maintainable, extensible, and testable. Currently, the entire system is a single async Python process with no tests and no CI/CD pipeline. After this phase:
|
||||
|
||||
- Automated tests provide confidence for every change (≥80% coverage)
|
||||
- CI/CD pipeline runs on every push to validate code quality
|
||||
- Multi-marketplace architecture enables adding new sources without modifying core logic
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────┐
|
||||
│ Project Structure (post-Phase-3) │
|
||||
│ │
|
||||
│ willhaben-tracker/ │
|
||||
│ ├── worker/ │
|
||||
│ │ ├── src/ │
|
||||
│ │ │ ├── main.py (entry point, scheduler) │
|
||||
│ │ │ ├── db.py (asyncpg pool mgmt) │
|
||||
│ │ │ ├── bot.py (Telegram handlers) │
|
||||
│ │ │ ├── notifier.py (message sending) │
|
||||
│ │ │ ├── scraper.py (base scraper class) │
|
||||
│ │ │ ├── scrapers/ │
|
||||
│ │ │ │ ├── __init__.py │
|
||||
│ │ │ │ ├── willhaben.py (willhaben-specific) │
|
||||
│ │ │ │ └── base.py (abstract base class) │
|
||||
│ │ │ ├── health.py (healthcheck endpoint) │
|
||||
│ │ │ └── migrate.py (migration runner) │
|
||||
│ │ ├── tests/ │
|
||||
│ │ │ ├── conftest.py │
|
||||
│ │ │ ├── test_scraper.py │
|
||||
│ │ │ ├── test_notifier.py │
|
||||
│ │ │ └── ... │
|
||||
│ │ ├── Dockerfile │
|
||||
│ │ └── requirements.txt │
|
||||
│ ├── .github/ │
|
||||
│ │ └── workflows/ │
|
||||
│ │ └── ci.yml (pytest + flake8 + coverage) │
|
||||
│ ├── pyproject.toml (coverage config, tools) │
|
||||
│ └── docker-compose.yml │
|
||||
└──────────────────────────────────────────────────────┘
|
||||
|
||||
Multi-marketplace abstraction:
|
||||
|
||||
ScraperBase (abstract):
|
||||
- async fetch_ads(keyword) → list[dict]
|
||||
- async parse_response(html/json) → list[dict]
|
||||
- normalize_ad(raw) → dict with standard keys
|
||||
|
||||
WillhabenScraper(ScraperBase):
|
||||
- implements willhaben-specific URL, headers, parsing
|
||||
|
||||
Future: KleinAnzeigenScraper, MobileScraper, ...
|
||||
|
||||
CI/CD Pipeline (.github/workflows/ci.yml):
|
||||
|
||||
on: push to main, feat/*; pull_request
|
||||
|
||||
jobs:
|
||||
lint-and-test:
|
||||
└─ python 3.12
|
||||
├─ flake8 (linting)
|
||||
├─ pytest --cov=src tests/ (unit + integration tests)
|
||||
└─ coverage >= 80% (fail if not met)
|
||||
|
||||
Tests Structure:
|
||||
|
||||
Unit tests:
|
||||
- test_scraper_pagination() — verify pagination logic with mock responses
|
||||
- test_price_filters() — verify filter functions
|
||||
- test_notification_retry() — verify retry queue behavior
|
||||
|
||||
Integration tests:
|
||||
- Test against real willhaben API (rate-limited, cached)
|
||||
- PostgreSQL test container via docker-compose
|
||||
```
|
||||
|
||||
## Tasks
|
||||
|
||||
| Task | File | Description |
|
||||
|------|------|-------------|
|
||||
| Multi-marketplace abstraction layer | [task-multi-marketplace.md](./task-multi-marketplace.md) | Refactor `scraper.py` into a base class + per-marketplace implementations. Introduces a standard ad schema and factory for registering new sources. |
|
||||
| Test suite with pytest (≥80% coverage) | [task-testing-pytest.md](./task-testing-pytest.md) | Add comprehensive unit tests covering scraper parsing, notification logic, price/postcode filters, retry queue, and scheduler flow. Configure coverage thresholds. |
|
||||
|
||||
## General Acceptance Criteria
|
||||
|
||||
- [ ] CI pipeline runs on every push to `main` and feature branches — fails if lint or coverage checks are not met
|
||||
- [ ] Code coverage is ≥80% across all source files in `worker/src/`
|
||||
- [ ] Multi-marketplace abstraction works — adding a new marketplace requires only creating one file under `scrapers/` with no changes to core logic
|
||||
- [ ] All existing functionality (willhaben scraping, notifications) continues to work after refactoring
|
||||
- [ ] The `/health` endpoint exposes test results or coverage stats (optional enhancement)
|
||||
@@ -0,0 +1,363 @@
|
||||
# Task: Multi-marketplace abstraction layer
|
||||
|
||||
## Description
|
||||
|
||||
Currently, `scraper.py` is tightly coupled to willhaben's API format and URL. Adding a second marketplace (e.g., Kleinanzeigen, Facebook Marketplace) would require extensive refactoring of the core logic — duplicating pagination, error handling, and notification code with subtle differences per source.
|
||||
|
||||
This task introduces an **abstract base class** for scrapers and a **standardized ad schema**, making it trivial to add new marketplaces by implementing only marketplace-specific parsing logic.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ scraper.py (module) │
|
||||
│ │
|
||||
│ ┌───────────────────────────────────┐│
|
||||
│ │ ScraperBase (ABC) ││
|
||||
│ │ ││
|
||||
│ │ Properties: ││
|
||||
│ │ name str ││
|
||||
│ │ base_url str ││
|
||||
│ │ max_pages int ││
|
||||
│ │ ││
|
||||
│ │ Abstract methods: ││
|
||||
│ │ build_query(url, params) → URL ││
|
||||
│ │ parse_page(html/json) → list ││
|
||||
│ │ normalize_ad(raw) → dict ││
|
||||
│ │ ││
|
||||
│ │ Concrete methods (shared): ││
|
||||
│ │ fetch_ads(keyword, cursor) ││
|
||||
│ │ _fetch_with_retry(url) ││
|
||||
│ └───────────────────────────────────┘│
|
||||
└──────┬────────────────────────────────┘
|
||||
│ inherits
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ scrapers/willhaben.py │
|
||||
│ │
|
||||
│ class WillhabenScraper(ScraperBase): │
|
||||
│ name = "willhaben" ││
|
||||
│ base_url = ".../api/v1/ad-search" ││
|
||||
│ build_query() → willhaben URL ││
|
||||
│ parse_page(json) → list ││
|
||||
│ normalize_ad(raw) → standard dict ││
|
||||
└───────────────────────────────────────┘
|
||||
|
||||
Standard ad schema (dict):
|
||||
{
|
||||
"id": str, # marketplace-specific ID
|
||||
"marketplace": str, # e.g. "willhaben"
|
||||
"title": str, # ad title
|
||||
"price": int | None, # price in cents
|
||||
"currency": str, # e.g. "EUR"
|
||||
"url": str, # full URL to the ad page
|
||||
"published_at": datetime | None,
|
||||
"location": { │
|
||||
"city": str, │
|
||||
"postcode": str | None, │
|
||||
"region": str | None │
|
||||
}, │
|
||||
"attributes": dict # marketplace-specific extras
|
||||
}
|
||||
|
||||
Scheduler (in main.py):
|
||||
|
||||
scrapers: list[ScraperBase] = [
|
||||
WillhabenScraper(),
|
||||
KleinanzeigenScraper(), # future
|
||||
]
|
||||
|
||||
for scraper in scrapers:
|
||||
ads_raw, total_hits = await scraper.fetch_ads(keyword)
|
||||
# ... process with same pipeline (filters, notifications)
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Abstract base class** defines the contract. Concrete scrapers only implement what's different per marketplace — URL building, response parsing, and field normalization.
|
||||
- *Alternative*: Could use a plugin architecture with entry_points, but that adds significant complexity for what is currently expected to be ≤3 marketplaces.
|
||||
- **Standardized output schema** ensures the downstream pipeline (filters, notifications) works identically regardless of source. Marketplace-specific fields are stored in `attributes`.
|
||||
- **Pagination logic lives in the base class**. Most marketplaces use offset/limit pagination; the abstract method handles this generically. Special cases override `_fetch_page()`.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Create `worker/src/scrapers/__init__.py`
|
||||
|
||||
```python
|
||||
from .willhaben import WillhabenScraper
|
||||
|
||||
__all__ = ["WillhabenScraper"]
|
||||
|
||||
|
||||
def get_scriper_by_name(name: str) -> "ScraperBase":
|
||||
"""Factory function to instantiate scrapers by name."""
|
||||
registry = {
|
||||
"willhaben": WillhabenScraper,
|
||||
}
|
||||
|
||||
cls = registry.get(name.lower())
|
||||
if not cls:
|
||||
raise ValueError(f"Unknown marketplace: {name}")
|
||||
|
||||
return cls()
|
||||
```
|
||||
|
||||
### 2. Create `worker/src/scrapers/base.py` (abstract base class)
|
||||
|
||||
```python
|
||||
import abc
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ScraperBase(abc.ABC):
|
||||
"""Abstract base class for marketplace scrapers."""
|
||||
|
||||
name: str = "unknown"
|
||||
base_url: str = ""
|
||||
max_pages: int = 2
|
||||
|
||||
@abc.abstractmethod
|
||||
def build_query(self, keyword: str, offset: int) -> str:
|
||||
"""Build the full API URL/endpoint for a keyword + offset."""
|
||||
...
|
||||
|
||||
@abc.abstractmethod
|
||||
def parse_page(self, response_content: Any) -> list[dict]:
|
||||
"""Parse raw response into list of ad dicts (marketplace-specific)."""
|
||||
...
|
||||
|
||||
@abc.abstractmethod
|
||||
def normalize_ad(self, raw_ad: dict) -> dict:
|
||||
"""Convert marketplace-specific format to standard schema."""
|
||||
...
|
||||
|
||||
async def fetch_ads(
|
||||
self,
|
||||
keyword: str,
|
||||
cursor_at: datetime | None = None,
|
||||
max_pages: int | None = None,
|
||||
) -> tuple[list[dict], int]:
|
||||
"""Fetch ads with pagination. Shared implementation."""
|
||||
pages = max_pages or self.max_pages
|
||||
all_ads: list[dict] = []
|
||||
total_hits = 0
|
||||
|
||||
from ..scraper import get_client # httpx singleton
|
||||
|
||||
client = await get_client()
|
||||
|
||||
for page in range(pages):
|
||||
url = self.build_query(keyword, offset=page * 30)
|
||||
|
||||
try:
|
||||
content = await self._fetch_with_retry(client, url)
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"%s: fetch failed at page %d for '%s': %s",
|
||||
self.name, page, keyword, exc
|
||||
)
|
||||
break
|
||||
|
||||
raw_ads = self.parse_page(content)
|
||||
|
||||
if not raw_ads:
|
||||
logger.info("%s: no more ads on page %d for '%s'",
|
||||
self.name, page, keyword)
|
||||
break
|
||||
|
||||
# Normalize and filter by cursor
|
||||
normalized = []
|
||||
for raw in raw_ads:
|
||||
ad = self.normalize_ad(raw)
|
||||
if cursor_at and ad["published_at"] and ad["published_at"] <= cursor_at:
|
||||
continue
|
||||
normalized.append(ad)
|
||||
|
||||
all_ads.extend(normalized)
|
||||
|
||||
# Politeness delay
|
||||
if page < pages - 1 and normalized:
|
||||
await asyncio.sleep(1.0)
|
||||
|
||||
return all_ads, total_hits
|
||||
|
||||
async def _fetch_with_retry(
|
||||
self,
|
||||
client: Any,
|
||||
url: str,
|
||||
max_retries: int = 3,
|
||||
) -> Any:
|
||||
"""Generic retry wrapper for HTTP fetches."""
|
||||
import httpx
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
resp = await client.get(url)
|
||||
resp.raise_for_status()
|
||||
return resp.json() if "application/json" in (resp.headers.get("content-type") or "") else resp.text
|
||||
except httpx.ConnectError as exc:
|
||||
logger.warning("%s: transport error attempt %d: %s",
|
||||
self.name, attempt + 1, exc)
|
||||
if attempt < max_retries - 1:
|
||||
await asyncio.sleep(2 ** attempt)
|
||||
else:
|
||||
raise
|
||||
|
||||
@property
|
||||
def headers(self) -> dict[str, str]:
|
||||
"""HTTP headers for requests. Override per marketplace."""
|
||||
return {}
|
||||
```
|
||||
|
||||
### 3. Create `worker/src/scrapers/willhaben.py` (refactor existing logic)
|
||||
|
||||
Move the current willhaben-specific code from `scraper.py` into this implementation:
|
||||
|
||||
```python
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from .base import ScraperBase
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class WillhabenScraper(ScraperBase):
|
||||
name = "willhaben"
|
||||
base_url = "https://api.willhaben.at/external/api/v1/ad-search"
|
||||
|
||||
@property
|
||||
def headers(self) -> dict[str, str]:
|
||||
return {
|
||||
"Accept": "application/json",
|
||||
"User-Agent": "Mozilla/5.0 (compatible; WillhabenTracker/1.0)",
|
||||
}
|
||||
|
||||
def build_query(self, keyword: str, offset: int = 0) -> str:
|
||||
"""Build willhaben API URL with keyword + pagination."""
|
||||
import urllib.parse
|
||||
|
||||
params = {
|
||||
"keyword": keyword,
|
||||
"rows": 30,
|
||||
"sort": 1, # newest first
|
||||
"offset": offset,
|
||||
}
|
||||
|
||||
return f"{self.base_url}?{urllib.parse.urlencode(params)}"
|
||||
|
||||
def parse_page(self, response_content: dict) -> list[dict]:
|
||||
"""Parse willhaben JSON response into raw ad dicts."""
|
||||
ads_list = (response_content.get("advertSummaryList") or {}).get(
|
||||
"advertSummary", []
|
||||
)
|
||||
|
||||
total_hits = int(response_content.get("rowsFound", 0))
|
||||
|
||||
return ads_list
|
||||
|
||||
def normalize_ad(self, raw_ad: dict) -> dict:
|
||||
"""Convert willhaben ad format to standard schema."""
|
||||
# Extract attributes from the nested format
|
||||
attrs = self._parse_attributes(raw_ad)
|
||||
|
||||
# Extract ID
|
||||
ad_id_raw = raw_ad.get("id", "")
|
||||
|
||||
# Extract title (handle various formats)
|
||||
title_raw = raw_ad.get("title") or raw_ad.get("Title", {})
|
||||
title = title_raw.get("Value", title_raw) if isinstance(title_raw, dict) else str(title_raw or "")
|
||||
|
||||
# Extract price
|
||||
price_str = attrs.get("PRICE_String") or attrs.get("priceString", "")
|
||||
try:
|
||||
price_cents = int(float(price_str.replace(".", ""))) if price_str else None
|
||||
except (ValueError, TypeError):
|
||||
price_cents = None
|
||||
|
||||
# Extract published date
|
||||
pub_str = attrs.get("PUBLISHED_String") or attrs.get("publishedString", "")
|
||||
published_at = None
|
||||
try:
|
||||
published_at = datetime.fromisoformat(pub_str.replace("Z", "+00:00"))
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
# Extract location
|
||||
city = attrs.get("LOCATION_CityName") or ""
|
||||
postcode = attrs.get("LOCATION_ZIP") or ""
|
||||
region = attrs.get("LOCATION_Region") or ""
|
||||
|
||||
return {
|
||||
"id": ad_id_raw,
|
||||
"marketplace": self.name,
|
||||
"title": title.strip(),
|
||||
"price": price_cents,
|
||||
"currency": "EUR",
|
||||
"url": raw_ad.get("linkUrl", ""),
|
||||
"published_at": published_at,
|
||||
"location": {
|
||||
"city": city,
|
||||
"postcode": postcode if postcode else None,
|
||||
"region": region if region else None,
|
||||
},
|
||||
"attributes": attrs, # preserve marketplace-specific fields
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _parse_attributes(ad_dict: dict) -> dict:
|
||||
"""Parse willhaben's nested attribute format into flat dict."""
|
||||
result = {}
|
||||
|
||||
for attr_group in ad_dict.get("attributes", []):
|
||||
if not isinstance(attr_group, dict):
|
||||
continue
|
||||
|
||||
group_name = attr_group.get("name") or ""
|
||||
|
||||
for item in attr_group.get("items", []):
|
||||
key = f"{group_name}_{item['name']}" if group_name else item["name"]
|
||||
result[key] = item.get("valueString", "")
|
||||
|
||||
return result
|
||||
```
|
||||
|
||||
### 4. Update `main.py` scheduler to use the new scraper factory
|
||||
|
||||
Replace direct calls to `fetch_ads(keyword)` with:
|
||||
|
||||
```python
|
||||
from scrapers import get_scriper_by_name
|
||||
|
||||
# At startup:
|
||||
scrapers_config = os.getenv("SCRAPERS", "willhaben").split(",")
|
||||
active_scrapers = [get_scriper_by_name(s) for s in scrapers_config]
|
||||
|
||||
# In scheduler loop:
|
||||
for scraper in active_scrapers:
|
||||
ads_raw, total_hits = await scraper.fetch_ads(keyword, cursor_at=cursor)
|
||||
|
||||
# ... process with existing pipeline (price filter, postcode filter, etc.)
|
||||
```
|
||||
|
||||
### 5. Add `.env.example` configuration for scrapers
|
||||
|
||||
```bash
|
||||
# Marketplace sources to scrape (comma-separated)
|
||||
SCRAPERS=willhaben
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `WillhabenScraper` produces identical output to the current `scraper.py` implementation (no regression in ad extraction)
|
||||
- [ ] Adding a new marketplace requires only: creating one file under `scrapers/`, registering it in `__init__.py`, and listing it in SCRAPERS env var
|
||||
- [ ] The standardized ad schema includes all fields needed by the downstream pipeline (price, postcode, published_at, URL)
|
||||
- [ ] Pagination logic works correctly through the base class for willhaben
|
||||
- [ ] Error handling (retries, timeouts) continues to work with the new abstraction
|
||||
- [ ] All existing bot commands and notifications function identically after refactoring
|
||||
@@ -0,0 +1,777 @@
|
||||
# Task: Test suite with pytest (≥80% coverage)
|
||||
|
||||
## Description
|
||||
|
||||
The project currently has **zero automated tests**. Every change is verified manually by watching logs or sending test messages to the bot. This makes refactoring risky and prevents CI/CD automation.
|
||||
|
||||
This task introduces a comprehensive pytest test suite covering all critical paths: scraper parsing, notification logic, price/postcode filters, retry queue behavior, and scheduler flow. Coverage threshold is set to 80% minimum.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ Tests structure │
|
||||
│ │
|
||||
│ worker/tests/ │
|
||||
│ ├── conftest.py │
|
||||
│ │ (fixtures: mock_pool, mock_bot)│
|
||||
│ ├── test_scraper.py │
|
||||
│ │ (parsing, pagination) │
|
||||
│ ├── test_notifier.py │
|
||||
│ │ (send, retry queue, digest) │
|
||||
│ ├── test_filters.py │
|
||||
│ │ (price, postcode, mute hours) │
|
||||
│ ├── test_scheduler.py │
|
||||
│ │ (cycle flow, shutdown) │
|
||||
│ └── test_health.py │
|
||||
│ (healthcheck endpoint) │
|
||||
│ │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ CI/CD Pipeline (.github/workflows/ci.yml)
|
||||
│ │
|
||||
│ on: push to main, feat/*; pull_request│
|
||||
│ │
|
||||
│ jobs: │
|
||||
│ lint-and-test: │
|
||||
│ └─ python 3.12 │
|
||||
│ ├─ flake8 (error-only) │
|
||||
│ ├─ pytest --cov=src tests/ │
|
||||
│ └─ coverage >= 80% │
|
||||
└───────────────────────────────────────┘
|
||||
|
||||
Test strategy:
|
||||
|
||||
Unit tests (majority):
|
||||
- Isolate each function/method with mocks
|
||||
- Test edge cases: missing fields, NULL prices, empty results
|
||||
- Fast (<1s per test)
|
||||
|
||||
Integration tests (minority):
|
||||
- Real HTTP to willhaben API (cached responses only)
|
||||
- PostgreSQL test container via docker-compose
|
||||
|
||||
Mocking strategy:
|
||||
- Telegram Bot: mock `bot.send_message()` → verify call count + content
|
||||
- Asyncpg pool: mock fetchval/fetch/execute → return canned data
|
||||
- httpx client: use pytest-httpx to intercept and return fixtures
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **pytest over unittest**. Cleaner syntax, better fixture system, easier async support.
|
||||
- **pytest-asyncio** for testing async functions directly without wrapping in `loop.run_until_complete()`.
|
||||
- Adding as test dependency: `pip install pytest pytest-asyncio pytest-cov httpx[socks]`
|
||||
- **Coverage threshold at 80%** enforced via `pyproject.toml`. Failures are actionable (which files/functions need coverage).
|
||||
- **Snapshot testing for HTTP responses**. Save real willhaben API responses as JSON fixtures to avoid live network calls in CI.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add test dependencies and configuration
|
||||
|
||||
In `worker/requirements-test.txt`:
|
||||
```txt
|
||||
pytest>=8.0
|
||||
pytest-asyncio>=0.24
|
||||
pytest-cov>=6.0
|
||||
aioresponses>=0.7 # mock aiohttp responses for health endpoint tests
|
||||
```
|
||||
|
||||
Create `pyproject.toml` in the project root:
|
||||
```toml
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["worker/tests"]
|
||||
asyncio_mode = "auto"
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["worker/src"]
|
||||
omit = [
|
||||
"*/tests/*",
|
||||
"*/migrate.py", # migration runner — tested manually against real DB
|
||||
]
|
||||
|
||||
[tool.coverage.report]
|
||||
fail_under = 80.0
|
||||
show_missing = true
|
||||
exclude_lines = [
|
||||
"pragma: no cover",
|
||||
"if TYPE_CHECKING:",
|
||||
]
|
||||
```
|
||||
|
||||
### 2. Create `worker/tests/conftest.py` (fixtures)
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, MagicMock
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_pool():
|
||||
"""Mock asyncpg.Pool with fetchval/fetch/execute."""
|
||||
pool = AsyncMock()
|
||||
pool.fetchval = AsyncMock(return_value=None)
|
||||
pool.fetch = AsyncMock(return_value=[])
|
||||
pool.execute = AsyncMock(return_value="DONE 1")
|
||||
return pool
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_bot():
|
||||
"""Mock Telegram Bot instance."""
|
||||
bot = MagicMock()
|
||||
bot.send_message = AsyncMock(return_value=True)
|
||||
bot.get_me = AsyncMock(return_value={"id": "bot_user", "is_bot": True})
|
||||
return bot
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_willhaben_response():
|
||||
"""Realistic willhaben API response for testing."""
|
||||
return {
|
||||
"rowsFound": 45,
|
||||
"advertSummaryList": {
|
||||
"advertSummary": [
|
||||
{
|
||||
"id": "123456789",
|
||||
"title": {"Value": "RTX 3090 Gaming X - Top Zustand"},
|
||||
"linkUrl": "https://www.willhaben.at/iad/markt/123456789-rtx-3090-gaming-x",
|
||||
"attributes": [
|
||||
{
|
||||
"name": "PRICE",
|
||||
"items": [{"name": "priceString", "valueString": "750.00"}]
|
||||
},
|
||||
{
|
||||
"name": "PUBLISHED",
|
||||
"items": [{"name": "publishedString", "valueString": "2026-07-04T12:30:00+02:00"}]
|
||||
},
|
||||
{
|
||||
"name": "LOCATION",
|
||||
"items": [
|
||||
{"name": "CityName", "valueString": "Wien"},
|
||||
{"name": "ZIP", "valueString": "1010"},
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
]
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_ad_normalized():
|
||||
"""Expected normalized ad dict from willhaben response."""
|
||||
return {
|
||||
"id": "123456789",
|
||||
"marketplace": "willhaben",
|
||||
"title": "RTX 3090 Gaming X - Top Zustand",
|
||||
"price": 75000, # in cents
|
||||
"currency": "EUR",
|
||||
"url": "https://www.willhaben.at/iad/markt/123456789-rtx-3090-gaming-x",
|
||||
"published_at": ..., # will be datetime object
|
||||
"location": {
|
||||
"city": "Wien",
|
||||
"postcode": "1010",
|
||||
"region": None,
|
||||
},
|
||||
"attributes": {...},
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_keyword_row():
|
||||
"""Sample keyword DB row."""
|
||||
return {
|
||||
"id": "kw-uuid-here",
|
||||
"keyword_name": "rtx 3090",
|
||||
"telegram_id": "298181113",
|
||||
"is_active": True,
|
||||
"price_min": 50000, # €500 minimum
|
||||
"price_max": 1000000, # €10000 maximum
|
||||
"allowed_postcodes": ["1010", "1020"],
|
||||
"last_scraped_at": None,
|
||||
"ads_cursor": None,
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Create `worker/tests/test_scraper.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
from scrapers.willhaben import WillhabenScraper
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
class TestWillhabenScraper:
|
||||
|
||||
def test_build_query(self):
|
||||
scraper = WillhabenScraper()
|
||||
url = scraper.build_query("rtx 3090", offset=30)
|
||||
|
||||
assert "keyword=rtx+3090" in url
|
||||
assert "offset=30" in url
|
||||
assert "rows=30" in url
|
||||
|
||||
def test_build_query_default_offset(self):
|
||||
scraper = WillhabenScraper()
|
||||
url = scraper.build_query("gtx 1660")
|
||||
|
||||
assert "offset=0" in url
|
||||
|
||||
def test_parse_page_empty_response(self, sample_willhaben_response):
|
||||
scraper = WillhabenScraper()
|
||||
|
||||
# Empty response
|
||||
empty = {"rowsFound": 0, "advertSummaryList": {"advertSummary": []}}
|
||||
result = scraper.parse_page(empty)
|
||||
|
||||
assert result == []
|
||||
|
||||
def test_normalize_ad(self, sample_willhaben_response):
|
||||
scraper = WillhabenScraper()
|
||||
raw_ad = sample_willhaben_response["advertSummaryList"]["advertSummary"][0]
|
||||
ad = scraper.normalize_ad(raw_ad)
|
||||
|
||||
assert ad["id"] == "123456789"
|
||||
assert ad["marketplace"] == "willhaben"
|
||||
assert ad["price"] == 75000 # cents
|
||||
assert ad["location"]["postcode"] == "1010"
|
||||
|
||||
def test_normalize_ad_missing_price(self):
|
||||
scraper = WillhabenScraper()
|
||||
raw_ad = {
|
||||
"id": "no-price",
|
||||
"title": {"Value": "Free RTX"},
|
||||
"linkUrl": "https://example.com",
|
||||
"attributes": [], # no price
|
||||
}
|
||||
ad = scraper.normalize_ad(raw_ad)
|
||||
|
||||
assert ad["price"] is None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_ads_pagination(self):
|
||||
scraper = WillhabenScraper()
|
||||
|
||||
with patch.object(scraper, '_fetch_with_retry') as mock_fetch:
|
||||
# Simulate 2 pages of results
|
||||
page1 = {
|
||||
"rowsFound": 45,
|
||||
"advertSummaryList": {"advertSummary": [
|
||||
{"id": f"ad{i}", "title": {"Value": f"Ad {i}"},
|
||||
"linkUrl": f"https://example.com/{i}",
|
||||
"attributes": [{"name": "PUBLISHED", "items": [
|
||||
{"name": "publishedString",
|
||||
"valueString": "2026-07-04T15:30:00+02:00"}]}]},
|
||||
] for i in range(3)}]
|
||||
}
|
||||
|
||||
page2 = {
|
||||
"rowsFound": 45,
|
||||
"advertSummaryList": {"advertSummary": [
|
||||
{"id": f"ad{i}", "title": {"Value": f"Ad {i}"},
|
||||
"linkUrl": f"https://example.com/{i}",
|
||||
"attributes": [{"name": "PUBLISHED", "items": [
|
||||
{"name": "publishedString",
|
||||
"valueString": "2026-07-04T15:30:00+02:00"}]}]},
|
||||
] for i in range(3, 6)}]
|
||||
}
|
||||
|
||||
mock_fetch.side_effect = [page1, page2]
|
||||
|
||||
ads, total = await scraper.fetch_ads("test keyword", max_pages=2)
|
||||
|
||||
assert len(ads) == 6 # 3 from each page
|
||||
|
||||
|
||||
class TestScraperBase:
|
||||
|
||||
def test_headers_default(self):
|
||||
scraper = WillhabenScraper()
|
||||
headers = scraper.headers
|
||||
|
||||
assert "Accept" in headers
|
||||
assert "User-Agent" in headers
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_with_retry_exhausts_retries(self):
|
||||
import httpx
|
||||
|
||||
scraper = WillhabenScraper()
|
||||
client = AsyncMock()
|
||||
client.get = AsyncMock(side_effect=httpx.ConnectError("connection refused"))
|
||||
|
||||
with pytest.raises(httpx.ConnectError):
|
||||
await scraper._fetch_with_retry(client, "http://example.com", max_retries=2)
|
||||
```
|
||||
|
||||
### 4. Create `worker/tests/test_filters.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
|
||||
class TestPriceFilters:
|
||||
|
||||
async def test_pass_when_no_min(self):
|
||||
"""Ad passes when no price minimum is set."""
|
||||
# Import the actual function being tested
|
||||
from notifier import _extract_price # or wherever it lives
|
||||
|
||||
kw_row = {"price_min": None, "price_max": None}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "PRICE", "items": [
|
||||
{"name": "priceString", "valueString": "50.00"}]}]
|
||||
}
|
||||
|
||||
# The check function (to be implemented in main.py)
|
||||
from ..main import _check_price_filters # actual implementation
|
||||
|
||||
result = await _check_price_filters(ad_dict, kw_row)
|
||||
assert result is True
|
||||
|
||||
async def test_pass_when_no_max(self):
|
||||
kw_row = {"price_min": 1000, "price_max": None}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "PRICE", "items": [
|
||||
{"name": "priceString", "valueString": "100.00"}]}]
|
||||
}
|
||||
|
||||
from ..main import _check_price_filters
|
||||
result = await _check_price_filters(ad_dict, kw_row)
|
||||
assert result is True
|
||||
|
||||
async def test_fail_below_min(self):
|
||||
kw_row = {"price_min": 50000, "price_max": None} # €500 min
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "PRICE", "items": [
|
||||
{"name": "priceString", "valueString": "10.00"}]}] # €10 — too cheap
|
||||
}
|
||||
|
||||
from ..main import _check_price_filters
|
||||
result = await _check_price_filters(ad_dict, kw_row)
|
||||
assert result is False
|
||||
|
||||
async def test_fail_above_max(self):
|
||||
kw_row = {"price_min": None, "price_max": 1000} # €10 max
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "PRICE", "items": [
|
||||
{"name": "priceString", "valueString": "500.00"}]}] # €500 — too expensive
|
||||
}
|
||||
|
||||
from ..main import _check_price_filters
|
||||
result = await _check_price_filters(ad_dict, kw_row)
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestPostcodeFilters:
|
||||
|
||||
async def test_pass_when_no_filter(self):
|
||||
kw_row = {"allowed_postcodes": None}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "LOCATION", "items": [
|
||||
{"name": "ZIP", "valueString": "1010"}]}]
|
||||
}
|
||||
|
||||
from ..main import _check_postcode_filter
|
||||
result = await _check_postcode_filter(ad_dict, kw_row)
|
||||
assert result is True
|
||||
|
||||
async def test_pass_when_matching(self):
|
||||
kw_row = {"allowed_postcodes": ["1010", "1020"]}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "LOCATION", "items": [
|
||||
{"name": "ZIP", "valueString": "1010"}]}]
|
||||
}
|
||||
|
||||
from ..main import _check_postcode_filter
|
||||
result = await _check_postcode_filter(ad_dict, kw_row)
|
||||
assert result is True
|
||||
|
||||
async def test_fail_when_not_matching(self):
|
||||
kw_row = {"allowed_postcodes": ["1010", "1020"]}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [{"name": "LOCATION", "items": [
|
||||
{"name": "ZIP", "valueString": "8010"}]}] # Graz — not in allowed list
|
||||
}
|
||||
|
||||
from ..main import _check_postcode_filter
|
||||
result = await _check_postcode_filter(ad_dict, kw_row)
|
||||
assert result is False
|
||||
|
||||
async def test_fail_when_no_postcode_in_ad(self):
|
||||
kw_row = {"allowed_postcodes": ["1010", "1020"]}
|
||||
|
||||
ad_dict = {
|
||||
"attributes": [] # no location info
|
||||
}
|
||||
|
||||
from ..main import _check_postcode_filter
|
||||
result = await _check_postcode_filter(ad_dict, kw_row)
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestMuteHours:
|
||||
|
||||
async def test_no_mute_when_not_configured(self, mock_pool):
|
||||
mock_pool.fetchrow.return_value = None
|
||||
|
||||
from ..notifier import _is_in_mute_hours
|
||||
result = await _is_in_mute_hours("298181113", mock_pool)
|
||||
|
||||
assert result is False
|
||||
|
||||
async def test_no_mute_outside_window(self, mock_pool):
|
||||
from datetime import time
|
||||
|
||||
mock_pool.fetchrow.return_value = {
|
||||
"mute_start": time(22, 0), # 10 PM
|
||||
"mute_end": time(7, 0), # 7 AM
|
||||
}
|
||||
|
||||
# Mock current time to noon UTC (outside mute window)
|
||||
with patch("notifier.datetime") as mock_dt:
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_dt.now.return_value = datetime(2026, 1, 1, 12, 0, tzinfo=timezone.utc)
|
||||
|
||||
from ..notifier import _is_in_mute_hours
|
||||
result = await _is_in_mute_hours("298181113", mock_pool)
|
||||
|
||||
assert result is False
|
||||
|
||||
async def test_muted_during_window(self, mock_pool):
|
||||
from datetime import time
|
||||
|
||||
mock_pool.fetchrow.return_value = {
|
||||
"mute_start": time(22, 0),
|
||||
"mute_end": time(7, 0),
|
||||
}
|
||||
|
||||
with patch("notifier.datetime") as mock_dt:
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_dt.now.return_value = datetime(2026, 1, 1, 3, 0, tzinfo=timezone.utc)
|
||||
|
||||
from ..notifier import _is_in_mute_hours
|
||||
result = await _is_in_mute_hours("298181113", mock_pool)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestNotificationQueue:
|
||||
|
||||
async def test_enqueue_on_failure(self, mock_pool):
|
||||
from ..notifier import _enqueue_retry
|
||||
|
||||
await _enqueue_retry(
|
||||
ad_id="ad-uuid-here",
|
||||
telegram_id="298181113",
|
||||
message_text="Test notification",
|
||||
notif_type="new",
|
||||
error_msg="Telegram API timeout",
|
||||
)
|
||||
|
||||
mock_pool.execute.assert_called_once()
|
||||
|
||||
async def test_no_duplicate_enqueue(self, mock_pool):
|
||||
"""Same ad+user should not create duplicate queue entries."""
|
||||
mock_pool.fetchval.return_value = "already-exists-uuid"
|
||||
|
||||
from ..notifier import _enqueue_retry
|
||||
|
||||
await _enqueue_retry(
|
||||
ad_id="ad-uuid-here",
|
||||
telegram_id="298181113",
|
||||
message_text="Test",
|
||||
notif_type="new",
|
||||
error_msg="timeout",
|
||||
)
|
||||
|
||||
# fetchval called (to check), but execute NOT called (no insert)
|
||||
mock_pool.fetchval.assert_called_once()
|
||||
mock_pool.execute.assert_not_called()
|
||||
```
|
||||
|
||||
### 5. Create `worker/tests/test_scheduler.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
|
||||
class TestSchedulerFlow:
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_process_notification_queue_retries_pending(self, mock_pool):
|
||||
"""Pending items should be retried if backoff period has elapsed."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_pool.fetch.return_value = [
|
||||
{
|
||||
"id": "queue-item-1",
|
||||
"ad_id": "ad-uuid",
|
||||
"telegram_id": "298181113",
|
||||
"message_text": "Retry this ad",
|
||||
"type": "new",
|
||||
"attempts": 0,
|
||||
"max_attempts": 5,
|
||||
"last_error": "timeout",
|
||||
"updated_at": datetime(2026, 7, 4, 10, 0, tzinfo=timezone.utc),
|
||||
},
|
||||
]
|
||||
|
||||
with patch("main.get_application_bot") as mock_get_bot:
|
||||
mock_bot = AsyncMock()
|
||||
mock_bot.send_message = AsyncMock()
|
||||
mock_get_bot.return_value = mock_bot
|
||||
|
||||
from ..main import process_notification_queue
|
||||
result = await process_notification_queue()
|
||||
|
||||
assert result == 1
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_process_notification_queue_dead_after_max_attempts(self, mock_pool):
|
||||
"""Items exceeding max_attempts should be marked as dead."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_pool.fetch.return_value = [
|
||||
{
|
||||
"id": "queue-item-2",
|
||||
"ad_id": "ad-uuid",
|
||||
"telegram_id": "298181113",
|
||||
"message_text": "Will fail again",
|
||||
"type": "new",
|
||||
"attempts": 5, # already at max
|
||||
"max_attempts": 5,
|
||||
"last_error": "user blocked bot",
|
||||
"updated_at": datetime(2026, 7, 4, 10, 0, tzinfo=timezone.utc),
|
||||
},
|
||||
]
|
||||
|
||||
with patch("main.get_application_bot") as mock_get_bot:
|
||||
import telegram.error
|
||||
mock_bot = AsyncMock()
|
||||
mock_bot.send_message = AsyncMock(
|
||||
side_effect=telegram.error.TelegramError("blocked")
|
||||
)
|
||||
mock_get_bot.return_value = mock_bot
|
||||
|
||||
from ..main import process_notification_queue
|
||||
await process_notification_queue()
|
||||
|
||||
# Should have updated to 'dead' status
|
||||
calls = [c[0] for c in mock_pool.execute.call_args_list]
|
||||
assert any("status = 'dead'" in str(c) or "status=$2" in str(c)
|
||||
for c in calls), "Item should be marked as dead"
|
||||
|
||||
|
||||
class TestDigestFlushing:
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_flush_digest_buffers(self, mock_pool):
|
||||
"""Buffered items older than interval should be flushed."""
|
||||
from datetime import datetime, timezone
|
||||
|
||||
mock_pool.fetch.side_effect = [
|
||||
# First fetch: get digest-enabled users
|
||||
[{"telegram_id": "298181113", "digest_interval": 60}],
|
||||
# Second fetch: get buffered items
|
||||
[
|
||||
{"id": "buf-1", "keyword": "rtx 3090",
|
||||
"title": "Ad 1", "price": 75000,
|
||||
"url": "https://...", "ad_id": "ad-uuid"},
|
||||
{"id": "buf-2", "keyword": "rtx 3090",
|
||||
"title": "Ad 2", "price": 68000,
|
||||
"url": "https://...", "ad_id": "ad-uuid-2"},
|
||||
],
|
||||
]
|
||||
|
||||
with patch("main.get_application_bot") as mock_get_bot:
|
||||
mock_bot = AsyncMock()
|
||||
mock_bot.send_message = AsyncMock()
|
||||
mock_get_bot.return_value = mock_bot
|
||||
|
||||
from ..main import flush_digest_buffers
|
||||
result = await flush_digest_buffers()
|
||||
|
||||
assert result == 1 # one digest sent
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_flush_empty_buffer(self, mock_pool):
|
||||
"""No buffered items should result in no action."""
|
||||
mock_pool.fetch.side_effect = [
|
||||
[{"telegram_id": "298181113", "digest_interval": 60}],
|
||||
[], # empty buffer
|
||||
]
|
||||
|
||||
from ..main import flush_digest_buffers
|
||||
result = await flush_digest_buffers()
|
||||
|
||||
assert result == 0
|
||||
|
||||
|
||||
class TestGracefulShutdown:
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cleanup_stops_scheduler_and_closes_pool(self):
|
||||
"""Cleanup should cancel scheduler, stop bot, close DB pool."""
|
||||
from telegram.ext import Application
|
||||
|
||||
app = AsyncMock(spec=Application)
|
||||
app.updater.running = True
|
||||
|
||||
with patch("main._scheduler_task") as mock_task:
|
||||
mock_task.done.return_value = False
|
||||
|
||||
from ..main import cleanup
|
||||
await cleanup(app)
|
||||
|
||||
mock_task.cancel.assert_called_once()
|
||||
app.updater.stop_polling.assert_called_once()
|
||||
```
|
||||
|
||||
### 6. Create `worker/tests/test_health.py`
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_endpoint_ok():
|
||||
"""Health endpoint should return 200 when system is healthy."""
|
||||
from health import create_health_app
|
||||
|
||||
app = create_health_app()
|
||||
|
||||
with patch("health.get_pool") as mock_pool_get:
|
||||
mock_pool = AsyncMock()
|
||||
mock_pool.fetchval = AsyncMock(return_value=1)
|
||||
mock_pool_get.return_value = mock_pool
|
||||
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
|
||||
try:
|
||||
from aiohttp.test_utils import TestClient, TestServer
|
||||
client = TestClient(TestServer(runner))
|
||||
async with client:
|
||||
resp = await client.get("/health")
|
||||
assert resp.status == 200
|
||||
|
||||
data = await resp.json()
|
||||
assert data["status"] == "ok"
|
||||
assert "db_connected" in data
|
||||
finally:
|
||||
await runner.cleanup()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_health_endpoint_unhealthy_db():
|
||||
"""Health endpoint should return 503 when DB is unreachable."""
|
||||
from health import create_health_app
|
||||
|
||||
app = create_health_app()
|
||||
|
||||
with patch("health.get_pool") as mock_pool_get:
|
||||
mock_pool_get.side_effect = Exception("connection refused")
|
||||
|
||||
runner = web.AppRunner(app)
|
||||
await runner.setup()
|
||||
|
||||
try:
|
||||
from aiohttp.test_utils import TestClient, TestServer
|
||||
client = TestClient(TestServer(runner))
|
||||
async with client:
|
||||
resp = await client.get("/health")
|
||||
assert resp.status == 503
|
||||
|
||||
data = await resp.json()
|
||||
assert data["status"] == "unhealthy"
|
||||
finally:
|
||||
await runner.cleanup()
|
||||
```
|
||||
|
||||
### 7. Create GitHub Actions workflow `.github/workflows/ci.yml`
|
||||
|
||||
```yaml
|
||||
name: CI — Lint & Test
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main, 'feat/*']
|
||||
pull_request:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
lint-and-test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install --upgrade pip
|
||||
pip install -r worker/requirements.txt
|
||||
pip install -r worker/requirements-test.txt
|
||||
|
||||
- name: Lint with flake8
|
||||
run: |
|
||||
flake8 worker/src/ --count --select=E9,F63,F7,F82 --show-source --statistics
|
||||
flake8 worker/tests/ --count --select=E9,F63,F7,F82 --show-source --statistics
|
||||
|
||||
- name: Test with pytest + coverage
|
||||
run: |
|
||||
cd worker
|
||||
python -m pytest tests/ --cov=src --cov-report=term-missing --cov-fail-under=80
|
||||
|
||||
build-docker:
|
||||
runs-on: ubuntu-latest
|
||||
needs: lint-and-test
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Build Docker image
|
||||
run: docker compose -f docker-compose.yml build worker
|
||||
|
||||
- name: Test healthcheck
|
||||
run: |
|
||||
docker compose up -d worker
|
||||
sleep 5
|
||||
docker inspect --format='{{.State.Health.Status}}' willhaben-tracker-worker-1 || true
|
||||
docker compose down
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `pytest` runs with 0 failures and ≥80% coverage on all source files
|
||||
- [ ] GitHub Actions pipeline passes on every push to `main` and feature branches
|
||||
- [ ] Tests cover: scraper parsing, pagination, price filters, postcode filters, mute hours, notification retry queue, digest flushing, graceful shutdown, healthcheck endpoint
|
||||
- [ ] Flake8 linting (error-level checks) passes in CI
|
||||
- [ ] Docker image builds successfully after all tests pass
|
||||
- [ ] Adding a new test file automatically includes it in the coverage report
|
||||
Reference in New Issue
Block a user