willhaben-tracker/docs/phase-0/task-healthcheck.md

# Task: Healthcheck endpoint + docker-compose configuration

## Description

The worker container currently has no healthcheck configured. Docker only relies on `restart: unless-stopped` which reacts to process crashes but **not** to logical failures (e.g., DB connection lost, Telegram API rate-limited indefinitely, scheduler deadlocked).

This task adds a lightweight HTTP `/health` endpoint inside the worker that reports the status of critical subsystems (DB connectivity, scheduler last run time), and configures Docker to use it for automatic restart decisions.

## Architecture

```
┌───────────────────────────────────────────────┐
│  worker container                              │
│                                               │
│  ┌───────────────────────────────────────┐    │
│  │ aiohttp web server (port 8765)        │    │
│  │                                       │    │
│  │ GET /health                           │    │
│  │   → check DB pool connectivity        │    │
│  │   → check scheduler last run time     │    │
│  │   → return JSON status                │    │
│  │                                       │    │
│  │ GET /stats                            │    │
│  │   → extended metrics (optional)       │    │
│  └───────────────────────────────────────┘    │
│                                               │
└───────────────────────────────────────────────┘

docker healthcheck:
  curl -f http://localhost:8765/health || exit 1
  interval=30s  timeout=5s  retries=3  start_period=10s
```

### Health check response format

**Healthy (200 OK):**
```json
{
    "status": "ok",
    "uptime_seconds": 3600,
    "scheduler_last_run": "2026-07-04T12:34:56+00:00",
    "db_connected": true,
    "telegram_polling": true
}
```

**Unhealthy (503 Service Unavailable):**
```json
{
    "status": "unhealthy",
    "reason": "scheduler_stale_last_run_120s"
}
```

### Key design decisions

- **aiohttp over http.server**: aiohttp is already async-native and integrates naturally with the existing asyncio event loop. No blocking I/O.
  - *Alternative*: Could use `websockets` or just `asyncio.start_server`, but aiohttp provides proper HTTP/JSON handling out of the box.
  - Adding `aiohttp` to requirements.txt (~500KB overhead).
- **DB check**: Execute a lightweight `SELECT 1` against the pool, not a full connection creation. Fast and accurate.
- **Scheduler staleness threshold**: If last_run > 2 * max_keyword_interval + 60s, mark unhealthy. This prevents false positives from slow scrape cycles.

## Implementation Details

### 1. Add `aiohttp` to requirements.txt

```txt
python-telegram-bot==21.4
asyncpg==0.30.0
httpx==0.27.2
python-dotenv==1.0.1
aiohttp==3.9.5          # ← new: healthcheck HTTP server
```

### 2. Create `worker/src/health.py`

```python
import time
import logging
from aiohttp import web

logger = logging.getLogger(__name__)

_start_time = time.time()
_last_scheduler_run: float | None = None


def record_scheduler_run():
    """Call this at the start of each scheduler cycle."""
    global _last_scheduler_run
    _last_scheduler_run = time.time()


async def health_handler(request: web.Request) -> web.Response:
    from db import get_pool
    
    status = "ok"
    checks = {
        "uptime_seconds": int(time.time() - _start_time),
        "db_connected": False,
        "telegram_polling": True,  # will be set by main.py
    }

    if _last_scheduler_run:
        elapsed = time.time() - _last_scheduler_run
        checks["scheduler_last_run_seconds_ago"] = round(elapsed)
    
    # Check DB connectivity
    try:
        pool = await get_pool()
        await pool.fetchval("SELECT 1")
        checks["db_connected"] = True
    except Exception as e:
        status = "unhealthy"
        checks["db_error"] = str(e)

    # Check scheduler staleness (threshold: 300s configurable)
    stale_threshold = int(os.getenv("HEALTHCHECK_SCHEDULER_STALE_S", "300"))
    if _last_scheduler_run and (time.time() - _last_scheduler_run) > stale_threshold:
        status = "unhealthy"
        checks["scheduler_stale"] = True

    body = {"status": status, **checks}
    
    status_code = 200 if status == "ok" else 503
    return web.json_response(body, status=status_code)


async def stats_handler(request: web.Request) -> web.Response:
    """Extended stats endpoint (for future dashboard)."""
    from db import get_pool
    
    pool = await get_pool()
    
    kw_count = await pool.fetchval("SELECT COUNT(*) FROM keywords") or 0
    active_kw = await pool.fetchval(
        "SELECT COUNT(*) FROM keywords WHERE is_active"
    ) or 0
    ad_count = await pool.fetchval("SELECT COUNT(*) FROM ads") or 0
    notif_count = await pool.fetchval("SELECT COUNT(*) FROM notifications") or 0
    
    return web.json_response({
        "keywords": kw_count,
        "active_keywords": active_kw,
        "ads_indexed": ad_count,
        "notifications_sent": notif_count,
    })


def create_health_app() -> web.Application:
    app = web.Application()
    app.router.add_get("/health", health_handler)
    app.router.add_get("/stats", stats_handler)
    return app
```

### 3. Start health server in `main.py`

Add after pool initialization, before scheduler starts:

```python
from health import create_health_app, record_scheduler_run

# ... in main():
health_app = create_health_app()
runner = web.AppRunner(health_app)
await runner.setup()
site = web.TCPSite(runner, "0.0.0.0", 8765)
await site.start()
logger.info("Health check server listening on :8765")
```

### 4. Call `record_scheduler_run()` at the top of each scheduler loop iteration in `main.py`

### 5. Update `docker-compose.yml` with healthcheck

```yaml
services:
  worker:
    build: ./worker
    restart: unless-stopped
    env_file:
      - .env
    networks:
      - supabase_default
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

networks:
  supabase_default:
    external: true
```

### 6. Add `curl` to the Docker image (not included in slim)

Update `worker/Dockerfile`:

```dockerfile
FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ .
ENTRYPOINT ["/app/entrypoint.sh"]
CMD ["python", "main.py"]
```

## Acceptance Criteria

- [ ] `curl http://localhost:8765/health` returns 200 with JSON status when the worker is healthy
- [ ] Simulating a DB failure (e.g., disconnecting from Supabase) causes `/health` to return 503 within one polling cycle
- [ ] Docker reports the container as `healthy` after the start period (~10-60 seconds)
- [ ] When the healthcheck fails 3 consecutive times, Docker automatically restarts the container
- [ ] The health server does not interfere with the scheduler or bot loop (no blocking I/O)
- [ ] `/stats` endpoint returns accurate counts of keywords, ads, and notifications