willhaben-tracker/docs/phase-0/task-healthcheck.md

Task: Healthcheck endpoint + docker-compose configuration

Description

The worker container currently has no healthcheck configured. Docker only relies on restart: unless-stopped which reacts to process crashes but not to logical failures (e.g., DB connection lost, Telegram API rate-limited indefinitely, scheduler deadlocked).

This task adds a lightweight HTTP /health endpoint inside the worker that reports the status of critical subsystems (DB connectivity, scheduler last run time), and configures Docker to use it for automatic restart decisions.

Architecture

┌───────────────────────────────────────────────┐
│  worker container                              │
│                                               │
│  ┌───────────────────────────────────────┐    │
│  │ aiohttp web server (port 8765)        │    │
│  │                                       │    │
│  │ GET /health                           │    │
│  │   → check DB pool connectivity        │    │
│  │   → check scheduler last run time     │    │
│  │   → return JSON status                │    │
│  │                                       │    │
│  │ GET /stats                            │    │
│  │   → extended metrics (optional)       │    │
│  └───────────────────────────────────────┘    │
│                                               │
└───────────────────────────────────────────────┘

docker healthcheck:
  curl -f http://localhost:8765/health || exit 1
  interval=30s  timeout=5s  retries=3  start_period=10s

Health check response format

Healthy (200 OK):

{
    "status": "ok",
    "uptime_seconds": 3600,
    "scheduler_last_run": "2026-07-04T12:34:56+00:00",
    "db_connected": true,
    "telegram_polling": true
}

Unhealthy (503 Service Unavailable):

{
    "status": "unhealthy",
    "reason": "scheduler_stale_last_run_120s"
}

Key design decisions

• aiohttp over http.server: aiohttp is already async-native and integrates naturally with the existing asyncio event loop. No blocking I/O.
  • Alternative: Could use websockets or just asyncio.start_server, but aiohttp provides proper HTTP/JSON handling out of the box.
  • Adding aiohttp to requirements.txt (~500KB overhead).
• DB check: Execute a lightweight SELECT 1 against the pool, not a full connection creation. Fast and accurate.
• Scheduler staleness threshold: If last_run > 2 * max_keyword_interval + 60s, mark unhealthy. This prevents false positives from slow scrape cycles.

Implementation Details

1. Add aiohttp to requirements.txt

python-telegram-bot==21.4
asyncpg==0.30.0
httpx==0.27.2
python-dotenv==1.0.1
aiohttp==3.9.5          # ← new: healthcheck HTTP server

2. Create worker/src/health.py

import time
import logging
from aiohttp import web

logger = logging.getLogger(__name__)

_start_time = time.time()
_last_scheduler_run: float | None = None


def record_scheduler_run():
    """Call this at the start of each scheduler cycle."""
    global _last_scheduler_run
    _last_scheduler_run = time.time()


async def health_handler(request: web.Request) -> web.Response:
    from db import get_pool
    
    status = "ok"
    checks = {
        "uptime_seconds": int(time.time() - _start_time),
        "db_connected": False,
        "telegram_polling": True,  # will be set by main.py
    }

    if _last_scheduler_run:
        elapsed = time.time() - _last_scheduler_run
        checks["scheduler_last_run_seconds_ago"] = round(elapsed)
    
    # Check DB connectivity
    try:
        pool = await get_pool()
        await pool.fetchval("SELECT 1")
        checks["db_connected"] = True
    except Exception as e:
        status = "unhealthy"
        checks["db_error"] = str(e)

    # Check scheduler staleness (threshold: 300s configurable)
    stale_threshold = int(os.getenv("HEALTHCHECK_SCHEDULER_STALE_S", "300"))
    if _last_scheduler_run and (time.time() - _last_scheduler_run) > stale_threshold:
        status = "unhealthy"
        checks["scheduler_stale"] = True

    body = {"status": status, **checks}
    
    status_code = 200 if status == "ok" else 503
    return web.json_response(body, status=status_code)


async def stats_handler(request: web.Request) -> web.Response:
    """Extended stats endpoint (for future dashboard)."""
    from db import get_pool
    
    pool = await get_pool()
    
    kw_count = await pool.fetchval("SELECT COUNT(*) FROM keywords") or 0
    active_kw = await pool.fetchval(
        "SELECT COUNT(*) FROM keywords WHERE is_active"
    ) or 0
    ad_count = await pool.fetchval("SELECT COUNT(*) FROM ads") or 0
    notif_count = await pool.fetchval("SELECT COUNT(*) FROM notifications") or 0
    
    return web.json_response({
        "keywords": kw_count,
        "active_keywords": active_kw,
        "ads_indexed": ad_count,
        "notifications_sent": notif_count,
    })


def create_health_app() -> web.Application:
    app = web.Application()
    app.router.add_get("/health", health_handler)
    app.router.add_get("/stats", stats_handler)
    return app

3. Start health server in main.py

Add after pool initialization, before scheduler starts:

from health import create_health_app, record_scheduler_run

# ... in main():
health_app = create_health_app()
runner = web.AppRunner(health_app)
await runner.setup()
site = web.TCPSite(runner, "0.0.0.0", 8765)
await site.start()
logger.info("Health check server listening on :8765")

4. Call record_scheduler_run() at the top of each scheduler loop iteration in main.py

5. Update docker-compose.yml with healthcheck

services:
  worker:
    build: ./worker
    restart: unless-stopped
    env_file:
      - .env
    networks:
      - supabase_default
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

networks:
  supabase_default:
    external: true

6. Add curl to the Docker image (not included in slim)

Update worker/Dockerfile:

FROM python:3.12-slim
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ .
ENTRYPOINT ["/app/entrypoint.sh"]
CMD ["python", "main.py"]

Acceptance Criteria

• curl http://localhost:8765/health returns 200 with JSON status when the worker is healthy
• Simulating a DB failure (e.g., disconnecting from Supabase) causes /health to return 503 within one polling cycle
• Docker reports the container as healthy after the start period (~10-60 seconds)
• When the healthcheck fails 3 consecutive times, Docker automatically restarts the container
• The health server does not interfere with the scheduler or bot loop (no blocking I/O)
• /stats endpoint returns accurate counts of keywords, ads, and notifications