feat(phase-0): stability hardening — search path fix, auto-migrations, graceful shutdown, health endpoint
- Remove SET search_path from db.py and migration SQL (Supabase uses public schema) - Add migrate.py with tracking table for forward-only SQL migrations - Add entrypoint.sh: waits for DB, runs migrations, then starts app - Copy 01-schema.sql + zz-seed.sql to worker/src/migrations/ - Add health.py: /health endpoint (200/503) with DB connectivity + scheduler staleness checks - /stats endpoint with keyword/ad/notification counts - Rewrite main.py shutdown sequence: signal handler, 5s grace for scheduler, ordered cleanup - Update Dockerfile: HEALTHCHECK directive, entrypoint, COPY migrations - Update docker-compose.yml: stop_grace_period=15s, healthcheck config, env vars - Add aiohttp>=3.9 to requirements.txt for health server
This commit is contained in:
hermes
@@ -0,0 +1,220 @@
|
||||
# Task: Healthcheck endpoint + docker-compose configuration
|
||||
|
||||
## Description
|
||||
|
||||
The worker container currently has no healthcheck configured. Docker only relies on `restart: unless-stopped` which reacts to process crashes but **not** to logical failures (e.g., DB connection lost, Telegram API rate-limited indefinitely, scheduler deadlocked).
|
||||
|
||||
This task adds a lightweight HTTP `/health` endpoint inside the worker that reports the status of critical subsystems (DB connectivity, scheduler last run time), and configures Docker to use it for automatic restart decisions.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────┐
|
||||
│ worker container │
|
||||
│ │
|
||||
│ ┌───────────────────────────────────────┐ │
|
||||
│ │ aiohttp web server (port 8765) │ │
|
||||
│ │ │ │
|
||||
│ │ GET /health │ │
|
||||
│ │ → check DB pool connectivity │ │
|
||||
│ │ → check scheduler last run time │ │
|
||||
│ │ → return JSON status │ │
|
||||
│ │ │ │
|
||||
│ │ GET /stats │ │
|
||||
│ │ → extended metrics (optional) │ │
|
||||
│ └───────────────────────────────────────┘ │
|
||||
│ │
|
||||
└───────────────────────────────────────────────┘
|
||||
|
||||
docker healthcheck:
|
||||
curl -f http://localhost:8765/health || exit 1
|
||||
interval=30s timeout=5s retries=3 start_period=10s
|
||||
```
|
||||
|
||||
### Health check response format
|
||||
|
||||
**Healthy (200 OK):**
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"uptime_seconds": 3600,
|
||||
"scheduler_last_run": "2026-07-04T12:34:56+00:00",
|
||||
"db_connected": true,
|
||||
"telegram_polling": true
|
||||
}
|
||||
```
|
||||
|
||||
**Unhealthy (503 Service Unavailable):**
|
||||
```json
|
||||
{
|
||||
"status": "unhealthy",
|
||||
"reason": "scheduler_stale_last_run_120s"
|
||||
}
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **aiohttp over http.server**: aiohttp is already async-native and integrates naturally with the existing asyncio event loop. No blocking I/O.
|
||||
- *Alternative*: Could use `websockets` or just `asyncio.start_server`, but aiohttp provides proper HTTP/JSON handling out of the box.
|
||||
- Adding `aiohttp` to requirements.txt (~500KB overhead).
|
||||
- **DB check**: Execute a lightweight `SELECT 1` against the pool, not a full connection creation. Fast and accurate.
|
||||
- **Scheduler staleness threshold**: If last_run > 2 * max_keyword_interval + 60s, mark unhealthy. This prevents false positives from slow scrape cycles.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add `aiohttp` to requirements.txt
|
||||
|
||||
```txt
|
||||
python-telegram-bot==21.4
|
||||
asyncpg==0.30.0
|
||||
httpx==0.27.2
|
||||
python-dotenv==1.0.1
|
||||
aiohttp==3.9.5 # ← new: healthcheck HTTP server
|
||||
```
|
||||
|
||||
### 2. Create `worker/src/health.py`
|
||||
|
||||
```python
|
||||
import time
|
||||
import logging
|
||||
from aiohttp import web
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_start_time = time.time()
|
||||
_last_scheduler_run: float | None = None
|
||||
|
||||
|
||||
def record_scheduler_run():
|
||||
"""Call this at the start of each scheduler cycle."""
|
||||
global _last_scheduler_run
|
||||
_last_scheduler_run = time.time()
|
||||
|
||||
|
||||
async def health_handler(request: web.Request) -> web.Response:
|
||||
from db import get_pool
|
||||
|
||||
status = "ok"
|
||||
checks = {
|
||||
"uptime_seconds": int(time.time() - _start_time),
|
||||
"db_connected": False,
|
||||
"telegram_polling": True, # will be set by main.py
|
||||
}
|
||||
|
||||
if _last_scheduler_run:
|
||||
elapsed = time.time() - _last_scheduler_run
|
||||
checks["scheduler_last_run_seconds_ago"] = round(elapsed)
|
||||
|
||||
# Check DB connectivity
|
||||
try:
|
||||
pool = await get_pool()
|
||||
await pool.fetchval("SELECT 1")
|
||||
checks["db_connected"] = True
|
||||
except Exception as e:
|
||||
status = "unhealthy"
|
||||
checks["db_error"] = str(e)
|
||||
|
||||
# Check scheduler staleness (threshold: 300s configurable)
|
||||
stale_threshold = int(os.getenv("HEALTHCHECK_SCHEDULER_STALE_S", "300"))
|
||||
if _last_scheduler_run and (time.time() - _last_scheduler_run) > stale_threshold:
|
||||
status = "unhealthy"
|
||||
checks["scheduler_stale"] = True
|
||||
|
||||
body = {"status": status, **checks}
|
||||
|
||||
status_code = 200 if status == "ok" else 503
|
||||
return web.json_response(body, status=status_code)
|
||||
|
||||
|
||||
async def stats_handler(request: web.Request) -> web.Response:
|
||||
"""Extended stats endpoint (for future dashboard)."""
|
||||
from db import get_pool
|
||||
|
||||
pool = await get_pool()
|
||||
|
||||
kw_count = await pool.fetchval("SELECT COUNT(*) FROM keywords") or 0
|
||||
active_kw = await pool.fetchval(
|
||||
"SELECT COUNT(*) FROM keywords WHERE is_active"
|
||||
) or 0
|
||||
ad_count = await pool.fetchval("SELECT COUNT(*) FROM ads") or 0
|
||||
notif_count = await pool.fetchval("SELECT COUNT(*) FROM notifications") or 0
|
||||
|
||||
return web.json_response({
|
||||
"keywords": kw_count,
|
||||
"active_keywords": active_kw,
|
||||
"ads_indexed": ad_count,
|
||||
"notifications_sent": notif_count,
|
||||
})
|
||||
|
||||
|
||||
def create_health_app() -> web.Application:
|
||||
app = web.Application()
|
||||
app.router.add_get("/health", health_handler)
|
||||
app.router.add_get("/stats", stats_handler)
|
||||
return app
|
||||
```
|
||||
|
||||
### 3. Start health server in `main.py`
|
||||
|
||||
Add after pool initialization, before scheduler starts:
|
||||
|
||||
```python
|
||||
from health import create_health_app, record_scheduler_run
|
||||
|
||||
# ... in main():
|
||||
health_app = create_health_app()
|
||||
runner = web.AppRunner(health_app)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, "0.0.0.0", 8765)
|
||||
await site.start()
|
||||
logger.info("Health check server listening on :8765")
|
||||
```
|
||||
|
||||
### 4. Call `record_scheduler_run()` at the top of each scheduler loop iteration in `main.py`
|
||||
|
||||
### 5. Update `docker-compose.yml` with healthcheck
|
||||
|
||||
```yaml
|
||||
services:
|
||||
worker:
|
||||
build: ./worker
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- .env
|
||||
networks:
|
||||
- supabase_default
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8765/health"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
networks:
|
||||
supabase_default:
|
||||
external: true
|
||||
```
|
||||
|
||||
### 6. Add `curl` to the Docker image (not included in slim)
|
||||
|
||||
Update `worker/Dockerfile`:
|
||||
|
||||
```dockerfile
|
||||
FROM python:3.12-slim
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY src/ .
|
||||
ENTRYPOINT ["/app/entrypoint.sh"]
|
||||
CMD ["python", "main.py"]
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `curl http://localhost:8765/health` returns 200 with JSON status when the worker is healthy
|
||||
- [ ] Simulating a DB failure (e.g., disconnecting from Supabase) causes `/health` to return 503 within one polling cycle
|
||||
- [ ] Docker reports the container as `healthy` after the start period (~10-60 seconds)
|
||||
- [ ] When the healthcheck fails 3 consecutive times, Docker automatically restarts the container
|
||||
- [ ] The health server does not interfere with the scheduler or bot loop (no blocking I/O)
|
||||
- [ ] `/stats` endpoint returns accurate counts of keywords, ads, and notifications
|
||||
Reference in New Issue
Block a user