docs(plan): add Phase 1, 2, 3 implementation specs
This commit is contained in:
hermes
@@ -0,0 +1,82 @@
|
||||
# Phase 2 — User Experience & Advanced Filtering
|
||||
|
||||
## Scope
|
||||
|
||||
This phase introduces **user-facing features** that significantly improve the experience of keyword tracking. Currently, every matching ad triggers an instant notification regardless of price, location, or time of day — leading to noise for popular keywords.
|
||||
|
||||
After this phase:
|
||||
- Users can configure price ranges and postcodes per keyword
|
||||
- Notifications respect mute hours (no alerts at 3 AM)
|
||||
- Users opt into digest mode (bundled summaries instead of individual pings)
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ User Interaction Layer │
|
||||
│ │
|
||||
│ Telegram Bot Commands: │
|
||||
│ /set_price_min <kw> <€> │
|
||||
│ /set_price_max <kw> <€> │
|
||||
│ /set_postcode <kw> <list> │
|
||||
│ /mute_hours <start>-<end> │
|
||||
│ /digest on|off │
|
||||
│ │
|
||||
└──────────┬───────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ Database Schema (extended) │
|
||||
│ │
|
||||
│ keywords table: │
|
||||
│ + price_min int │
|
||||
│ + price_max int │
|
||||
│ + allowed_postcodes text[] │
|
||||
│ │
|
||||
│ user_settings table (new): │
|
||||
│ telegram_id text PK │
|
||||
│ mute_start time │
|
||||
│ mute_end time │
|
||||
│ digest_mode bool DEFAULT false │
|
||||
│ digest_interval int DEFAULT 60 │
|
||||
└──────────┬───────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ Notification Pipeline (modified) │
|
||||
│ │
|
||||
│ For each new ad: │
|
||||
│ ├─ filter by price_min/max? → skip │
|
||||
│ ├─ filter by allowed_postcodes? → skip │
|
||||
│ ├─ user in mute hours? │
|
||||
│ │ digest_on → buffer to digest_table │
|
||||
│ │ digest_off→ skip notification │
|
||||
│ └─ normal → send now │
|
||||
│ │
|
||||
│ Digest scheduler (separate task): │
|
||||
│ every digest_interval: │
|
||||
│ collect buffered notifications per user │
|
||||
│ format as summary message │
|
||||
│ send single message │
|
||||
│ clear buffer │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Tasks
|
||||
|
||||
| Task | File | Description |
|
||||
|------|------|-------------|
|
||||
| Price range filters per keyword | [task-price-filters.md](./task-price-filters.md) | Add `price_min` and `price_max` columns to the keywords table; filter ads during processing based on these thresholds. Bot commands to set/unset. |
|
||||
| Location / postcode filters per keyword | [task-postcode-filters.md](./task-postcode-filters.md) | Add `allowed_postcodes` text[] column to keywords; only notify if an ad's location matches any allowed postcode. |
|
||||
| Mute hours per user | [task-mute-hours.md](./task-mute-hours.md) | Create `user_settings` table with configurable mute window (start/end time in UTC); suppress notifications during this window. |
|
||||
| Digest / summary notifications | [task-digest-notifications.md](./task-digest-notifications.md) | Buffer notifications for users with digest mode enabled; send a bundled summary at configured intervals instead of individual alerts. |
|
||||
|
||||
## General Acceptance Criteria
|
||||
|
||||
- [ ] Users can set price min/max on any keyword and only receive notifications within that range
|
||||
- [ ] Postcode filtering works — ads outside allowed postcodes are silently skipped (not counted as new)
|
||||
- [ ] Mute hours suppress all notifications to a user during the configured window, regardless of keyword
|
||||
- [ ] Digest mode buffers individual alerts and sends one summary message at the configured interval
|
||||
- [ ] All filters combine correctly: an ad is only notified if it passes price + postcode checks AND the user is not muted (or digest mode active)
|
||||
- [ ] The bot provides clear feedback when a filter setting is changed ("Keyword X: price range set to €100–€500")
|
||||
- [ ] Admin can view all keyword filters and user settings via `/keywords` command output
|
||||
@@ -0,0 +1,355 @@
|
||||
# Task: Digest / summary notifications
|
||||
|
||||
## Description
|
||||
|
||||
For popular keywords that generate many matches per cycle, users may receive 10–20 individual notifications in quick succession. This task introduces **digest mode** — instead of immediate alerts, notifications are buffered and sent as a single summary message at configurable intervals.
|
||||
|
||||
Digest mode is complementary to mute hours: during mute hours, all messages are suppressed; with digest mode ON, messages are collected and sent as a batch at the configured interval (default: every 60 minutes).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ user_settings table │
|
||||
│ digest_mode bool DEFAULT false │
|
||||
│ digest_interval int DEFAULT 60 │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Notification Pipeline (modified) │
|
||||
│ │
|
||||
│ For each new ad: │
|
||||
│ if user.digest_mode == false: │
|
||||
│ → send immediately (current) │
|
||||
│ elif in mute hours: │
|
||||
│ → discard (already handled) │
|
||||
│ else: │
|
||||
│ → insert into digest_buffer │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ digest_buffer table (new) │
|
||||
│ │
|
||||
│ id uuid PK │
|
||||
│ telegram_id text │
|
||||
│ ad_id uuid REFERENCES ads │
|
||||
│ keyword text │
|
||||
│ title text │
|
||||
│ price int │
|
||||
│ url text │
|
||||
│ created_at timestamptz │
|
||||
│ │
|
||||
│ INDEX: telegram_id, created_at │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Digest Scheduler (separate task) │
|
||||
│ │
|
||||
│ Runs every digest_interval per user │
|
||||
│ ├─ SELECT all buffered items │
|
||||
│ ├─ GROUP BY telegram_id │
|
||||
│ ├─ Format summary message │
|
||||
│ └─ DELETE buffered items │
|
||||
│ │
|
||||
│ Summary format: │
|
||||
│ 📋 Digest — 5 new ads (14:30 UTC) │
|
||||
│ │
|
||||
│ 🔑 "rtx 3090" (3 ads): │
|
||||
│ • RTX 3090 Ti - €750 [link] │
|
||||
│ • ASUS RTX 3090 - €680 [link] │
|
||||
│ • MSI RTX 3090 Gaming X - €720 │
|
||||
│ │
|
||||
│ 🔑 "gtx 1660" (2 ads): │
|
||||
│ • GTX 1660 Super - €120 [link] │
|
||||
│ • EVGA GTX 1660 - €95 [link] │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Separate buffer table** instead of in-memory only. Survives restarts, visible via pgAdmin for debugging.
|
||||
- **Per-user interval**: Each user configures their own digest frequency (default 60 min). Implemented with a single scheduler task that checks all users' intervals on each cycle.
|
||||
- **Group by keyword** in the summary message. Makes it easy to scan relevant categories without digging through unrelated listings.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration
|
||||
|
||||
In `worker/src/migrations/04-user-settings.sql`:
|
||||
|
||||
```sql
|
||||
-- digest_buffer table for accumulating notifications
|
||||
|
||||
CREATE TABLE IF NOT EXISTS digest_buffer (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
telegram_id text NOT NULL,
|
||||
ad_id uuid REFERENCES ads(id) ON DELETE CASCADE,
|
||||
keyword text NOT NULL,
|
||||
title text NOT NULL,
|
||||
price int, -- in cents
|
||||
url text,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_digest_buffer
|
||||
ON digest_buffer(telegram_id, created_at DESC);
|
||||
|
||||
COMMENT ON TABLE digest_buffer IS
|
||||
'Buffer for digest-mode notifications. Flushed to Telegram at intervals.';
|
||||
```
|
||||
|
||||
### 2. Buffer notifications instead of sending immediately
|
||||
|
||||
In `main.py` scheduler loop, after all filters pass and mute check passes:
|
||||
|
||||
```python
|
||||
# Check if user has digest mode enabled
|
||||
settings = await pool.fetchrow(
|
||||
"SELECT digest_mode FROM user_settings WHERE telegram_id = $1",
|
||||
telegram_id_str,
|
||||
)
|
||||
|
||||
if settings and settings["digest_mode"]:
|
||||
# Buffer for digest
|
||||
await pool.execute(
|
||||
"""INSERT INTO digest_buffer (telegram_id, ad_id, keyword, title, price, url)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)""",
|
||||
telegram_id_str,
|
||||
ad_id,
|
||||
kw_row["keyword"],
|
||||
ad_dict.get("title", "Unknown"),
|
||||
_extract_price(ad_dict),
|
||||
ad_dict.get("url"),
|
||||
)
|
||||
|
||||
# Still log the notification (for stats)
|
||||
await log_notify(pool, ad_id, telegram_id, "new")
|
||||
else:
|
||||
# Send immediately (current behavior)
|
||||
await notify_new(bot, pool, kw_row["keyword"],
|
||||
telegram_id, ad_dict, ad_id)
|
||||
```
|
||||
|
||||
### 3. Add digest flushing task to scheduler
|
||||
|
||||
In `main.py`, add a new async function and call it at the start of each cycle:
|
||||
|
||||
```python
|
||||
async def flush_digest_buffers() -> int:
|
||||
"""Process pending digest buffers for users whose interval has elapsed."""
|
||||
from db import get_pool
|
||||
|
||||
pool = await get_pool()
|
||||
|
||||
# Get all users with digest mode ON
|
||||
users = await pool.fetch("""
|
||||
SELECT telegram_id, digest_interval
|
||||
FROM user_settings
|
||||
WHERE digest_mode = true
|
||||
""")
|
||||
|
||||
sent_count = 0
|
||||
|
||||
for user in users:
|
||||
interval_min = user["digest_interval"] or 60
|
||||
cutoff = datetime.now(tz=timezone.utc) - timedelta(minutes=interval_min)
|
||||
|
||||
# Get buffered items older than the interval
|
||||
buffered = await pool.fetch("""
|
||||
SELECT db.id, db.keyword, db.title, db.price, db.url
|
||||
FROM digest_buffer db
|
||||
WHERE db.telegram_id = $1
|
||||
AND db.created_at <= $2
|
||||
ORDER BY db.keyword, db.created_at DESC
|
||||
""", user["telegram_id"], cutoff)
|
||||
|
||||
if not buffered:
|
||||
continue
|
||||
|
||||
# Group by keyword
|
||||
from collections import defaultdict
|
||||
groups: dict[str, list] = defaultdict(list)
|
||||
|
||||
for item in buffered:
|
||||
price_str = f"€{item['price']/100:.2f}" if item['price'] else "Free"
|
||||
entry = f"• {item['title']} - {price_str}"
|
||||
groups[item["keyword"]].append(entry)
|
||||
|
||||
# Build summary message
|
||||
lines = [f"📋 Digest — {len(buffered)} new ads ({cutoff:%H:%M}–{datetime.now(tz=timezone.utc):%H:%M} UTC)\n"]
|
||||
|
||||
for kw_name, entries in groups.items():
|
||||
lines.append(f"\n🔑 \"{kw_name}\" ({len(entries)} ads):")
|
||||
# Limit to 10 entries per keyword to avoid spam
|
||||
for entry in entries[:10]:
|
||||
lines.append(entry)
|
||||
if len(entries) > 10:
|
||||
lines.append(f" ... and {len(entries)-10} more")
|
||||
|
||||
message_text = "\n".join(lines)
|
||||
|
||||
# Send the digest
|
||||
try:
|
||||
from bot import get_application_bot
|
||||
bot = get_application_bot()
|
||||
|
||||
telegram_id_int = int(user["telegram_id"])
|
||||
await bot.send_message(chat_id=telegram_id_int, text=message_text)
|
||||
|
||||
sent_count += 1
|
||||
|
||||
# Log all buffered notifications as delivered
|
||||
buffer_ids = [item["id"] for item in buffered]
|
||||
for ad_item in buffered:
|
||||
await log_notify(pool, ad_item["ad_id"], telegram_id_int, "new")
|
||||
|
||||
except TelegramError as e:
|
||||
logger.error("Digest send failed for %s: %s", user["telegram_id"], e)
|
||||
|
||||
finally:
|
||||
# Clear the buffer (whether sent or not — if it failed, log entries remain in DB)
|
||||
await pool.execute(
|
||||
"""DELETE FROM digest_buffer
|
||||
WHERE telegram_id = $1 AND created_at <= $2""",
|
||||
user["telegram_id"], cutoff,
|
||||
)
|
||||
|
||||
return sent_count
|
||||
|
||||
|
||||
# Call at the start of run_scheduler():
|
||||
async def run_scheduler() -> None:
|
||||
while True:
|
||||
try:
|
||||
record_scheduler_run()
|
||||
|
||||
# Flush digest buffers first
|
||||
digests_sent = await flush_digest_buffers()
|
||||
if digests_sent:
|
||||
logger.info("Sent %d digest summaries", digests_sent)
|
||||
|
||||
# Process notification queue...
|
||||
processed = await process_notification_queue()
|
||||
if processed:
|
||||
logger.info("Retried %d queued notifications", processed)
|
||||
|
||||
# ... existing keyword iteration ...
|
||||
```
|
||||
|
||||
### 4. Add bot commands in `bot.py`
|
||||
|
||||
```python
|
||||
async def cmd_digest_on(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Enable digest mode with optional interval."""
|
||||
|
||||
interval = 60 # default minutes
|
||||
if len(context.args) > 0:
|
||||
try:
|
||||
interval = int(context.args[0])
|
||||
if interval < 5 or interval > 1440:
|
||||
raise ValueError
|
||||
except ValueError:
|
||||
await update.message.reply_text(
|
||||
"Usage: /digest_on [minutes]\n"
|
||||
"Interval must be between 5 and 1440 minutes (24h)."
|
||||
)
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO user_settings (telegram_id, digest_mode, digest_interval)
|
||||
VALUES ($1, true, $2)
|
||||
ON CONFLICT (telegram_id)
|
||||
DO UPDATE SET digest_mode = EXCLUDED.digest_mode,
|
||||
digest_interval = EXCLUDED.digest_interval""",
|
||||
telegram_id, interval,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Digest mode ENABLED\n"
|
||||
f"Digests will be sent every {interval} minutes.\n"
|
||||
"Use /digest_off to return to instant notifications."
|
||||
)
|
||||
|
||||
|
||||
async def cmd_digest_off(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Disable digest mode."""
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO user_settings (telegram_id, digest_mode)
|
||||
VALUES ($1, false)
|
||||
ON CONFLICT (telegram_id)
|
||||
DO UPDATE SET digest_mode = EXCLUDED.digest_mode""",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
# Flush any remaining buffered items immediately
|
||||
await pool.execute(
|
||||
"""DELETE FROM digest_buffer WHERE telegram_id = $1""",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
"✅ Digest mode DISABLED — notifications are now instant."
|
||||
)
|
||||
|
||||
|
||||
async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Show current user settings (extended from mute hours task)."""
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
settings = await pool.fetchrow(
|
||||
"SELECT mute_start, mute_end, digest_mode, digest_interval "
|
||||
"FROM user_settings WHERE telegram_id = $1",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
reply_parts = []
|
||||
|
||||
if not settings or not settings["mute_start"]:
|
||||
reply_parts.append("🔕 Mute hours: OFF")
|
||||
else:
|
||||
reply_parts.append(
|
||||
f"🔕 Mute hours: {settings['mute_start']} — {settings['mute_end']} UTC"
|
||||
)
|
||||
|
||||
if settings and settings["digest_mode"]:
|
||||
reply_parts.append(
|
||||
f"📋 Digest: ON (every {settings['digest_interval']} min)"
|
||||
)
|
||||
else:
|
||||
reply_parts.append("📋 Digest: OFF (instant notifications)")
|
||||
|
||||
await update.message.reply_text("\n".join(reply_parts))
|
||||
```
|
||||
|
||||
Register handlers:
|
||||
```python
|
||||
dp.add_handler(CommandHandler("digest_on", cmd_digest_on))
|
||||
dp.add_handler(CommandHandler("digest_off", cmd_digest_off))
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/digest_on` enables digest mode with 60-minute default interval
|
||||
- [ ] `/digest_on 30` sets digest interval to 30 minutes
|
||||
- [ ] New ads are inserted into `digest_buffer` instead of being sent immediately when digest is ON
|
||||
- [ ] At the configured interval, all buffered items are flushed as a single summary message
|
||||
- [ ] The summary groups ads by keyword and includes price information
|
||||
- [ ] After flushing, buffered items are deleted from the table
|
||||
- [ ] `/digest_off` disables digest mode and sends any remaining buffered items immediately
|
||||
- [ ] Mute hours take precedence over digest — muted notifications are discarded, not buffered
|
||||
@@ -0,0 +1,211 @@
|
||||
# Task: Mute hours per user
|
||||
|
||||
## Description
|
||||
|
||||
Users may add keywords that are popular enough to trigger notifications at any hour. Currently, there's no way to suppress alerts during sleeping hours — the bot sends notifications 24/7.
|
||||
|
||||
This task creates a `user_settings` table with configurable mute windows (start/end time). During the mute window, notifications for that user are suppressed entirely. The notification is not lost — it's still logged in `log_notifications`, but the Telegram message is not sent.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ user_settings table (new) │
|
||||
│ │
|
||||
│ telegram_id text PRIMARY KEY │
|
||||
│ mute_start time │
|
||||
│ mute_end time │
|
||||
│ digest_mode bool DEFAULT false │
|
||||
│ digest_interval int DEFAULT 60 │
|
||||
│ │
|
||||
│ Example: │
|
||||
│ telegram_id = '298181113' │
|
||||
│ mute_start = '22:00:00' │
|
||||
│ mute_end = '07:00:00' │
|
||||
│ → no alerts between 10PM-7AM UTC │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Notification pipeline (in main.py) │
|
||||
│ │
|
||||
│ For each new ad that passes filters: │
|
||||
│ user_settings = get from DB │
|
||||
│ if in_mute_hours(user_settings): │
|
||||
│ log_notify() │
|
||||
│ → skip Telegram send │
|
||||
│ else: │
|
||||
│ notify_new() / notify_drop() │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Time stored as `time` type in PostgreSQL** — native, efficient for range checks. Default is NULL (no mute window).
|
||||
- *Alternative*: Could store as integer hours (e.g., 22, 7), but `time` type gives flexibility for minute-level precision and clearer UI.
|
||||
- **UTC timezone**: The bot operates in UTC internally. Users should be informed that mute times are in UTC. Adding timezone support per-user is a Phase 3 consideration.
|
||||
- **Mute window can cross midnight** — start > end means the window wraps around midnight (e.g., 22:00–07:00). The check handles this correctly.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration
|
||||
|
||||
In `worker/src/migrations/04-user-settings.sql`:
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS user_settings (
|
||||
telegram_id text PRIMARY KEY,
|
||||
mute_start time,
|
||||
mute_end time,
|
||||
digest_mode bool NOT NULL DEFAULT false,
|
||||
digest_interval int NOT NULL DEFAULT 60, -- minutes
|
||||
|
||||
CONSTRAINT chk_mute_hours CHECK (
|
||||
mute_start IS NULL AND mute_end IS NULL
|
||||
OR mute_start IS NOT NULL AND mute_end IS NOT NULL
|
||||
)
|
||||
);
|
||||
|
||||
COMMENT ON TABLE user_settings IS
|
||||
'User-specific settings for notification behavior';
|
||||
```
|
||||
|
||||
### 2. Add mute hours check in `notifier.py` or `main.py`
|
||||
|
||||
```python
|
||||
async def _is_in_mute_hours(
|
||||
telegram_id: str,
|
||||
pool: asyncpg.Pool
|
||||
) -> bool:
|
||||
"""Check if the current time is within the user's mute window."""
|
||||
|
||||
settings = await pool.fetchrow(
|
||||
"SELECT mute_start, mute_end FROM user_settings WHERE telegram_id = $1",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
if not settings or not settings["mute_start"] or not settings["mute_end"]:
|
||||
return False # no mute configured
|
||||
|
||||
now_utc = datetime.now(tz=timezone.utc).time()
|
||||
start = settings["mute_start"]
|
||||
end = settings["mute_end"]
|
||||
|
||||
if start < end:
|
||||
# Normal window (e.g., 22:00–07:00 → actually wraps, so this is rare)
|
||||
return start <= now_utc <= end
|
||||
else:
|
||||
# Window crosses midnight (e.g., 22:00 to 07:00 next day)
|
||||
return now_utc >= start or now_utc <= end
|
||||
|
||||
|
||||
# In main.py scheduler loop, before calling notify_new():
|
||||
telegram_id_str = str(telegram_id)
|
||||
|
||||
in_mute = await _is_in_mute_hours(telegram_id_str, pool)
|
||||
if in_mute:
|
||||
logger.debug("Muted notification for user %s (mute window active)", telegram_id)
|
||||
# Still log it but don't send Telegram message
|
||||
await log_notify(pool, ad_id, telegram_id, "new")
|
||||
return
|
||||
|
||||
# Proceed with normal notification...
|
||||
```
|
||||
|
||||
### 3. Add bot commands in `bot.py`
|
||||
|
||||
```python
|
||||
async def cmd_set_mute(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Set mute hours window (HH:MM-HH:MM format)."""
|
||||
if len(context.args) < 1:
|
||||
await update.message.reply_text(
|
||||
"Usage: /mute_hours HH:MM-HH:MM\n"
|
||||
"Example: /mute_hours 22:00-07:00 (mutes from 10PM to 7AM UTC)\n"
|
||||
"Use /mute_off to disable."
|
||||
)
|
||||
return
|
||||
|
||||
try:
|
||||
start_str, end_str = context.args[0].split("-")
|
||||
mute_start = datetime.strptime(start_str.strip(), "%H:%M").time()
|
||||
mute_end = datetime.strptime(end_str.strip(), "%H:%M").time()
|
||||
except (ValueError, TypeError) as e:
|
||||
await update.message.reply_text(
|
||||
f"Invalid format. Use HH:MM-HH:MM.\nExample: /mute_hours 22:00-07:00"
|
||||
)
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO user_settings (telegram_id, mute_start, mute_end)
|
||||
VALUES ($1, $2, $3)
|
||||
ON CONFLICT (telegram_id)
|
||||
DO UPDATE SET mute_start = EXCLUDED.mute_start, mute_end = EXCLUDED.mute_end""",
|
||||
telegram_id, mute_start, mute_end,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Mute hours set: {mute_start} — {mute_end} UTC\n"
|
||||
"No notifications will be sent during this window.\n"
|
||||
"Use /mute_off to disable or change."
|
||||
)
|
||||
|
||||
|
||||
async def cmd_mute_off(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Disable mute hours."""
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
await pool.execute(
|
||||
"""INSERT INTO user_settings (telegram_id, mute_start, mute_end)
|
||||
VALUES ($1, NULL, NULL)
|
||||
ON CONFLICT (telegram_id)
|
||||
DO UPDATE SET mute_start = EXCLUDED.mute_start, mute_end = EXCLUDED.mute_end""",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
await update.message.reply_text("✅ Mute hours disabled.")
|
||||
|
||||
|
||||
async def cmd_status(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Show current user settings."""
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
settings = await pool.fetchrow(
|
||||
"SELECT mute_start, mute_end FROM user_settings WHERE telegram_id = $1",
|
||||
telegram_id,
|
||||
)
|
||||
|
||||
if not settings or not settings["mute_start"]:
|
||||
reply = "🔕 Mute hours: OFF (notifications sent 24/7)"
|
||||
else:
|
||||
reply = f"🔕 Mute hours: {settings['mute_start']} — {settings['mute_end']} UTC"
|
||||
|
||||
await update.message.reply_text(reply)
|
||||
```
|
||||
|
||||
Register handlers:
|
||||
```python
|
||||
dp.add_handler(MessageHandler(REGEX(r"^/mute_hours"), cmd_set_mute))
|
||||
dp.add_handler(CommandHandler("mute_off", cmd_mute_off))
|
||||
dp.add_handler(CommandHandler("status", cmd_status))
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/mute_hours 22:00-07:00` sets mute window from 10 PM to 7 AM UTC
|
||||
- [ ] Notifications during the mute window are logged but NOT sent via Telegram
|
||||
- [ ] Notifications outside the mute window work normally (no regression)
|
||||
- [ ] Mute windows that cross midnight (start > end) are handled correctly
|
||||
- [ ] `/mute_off` clears both start and end times, restoring 24/7 notifications
|
||||
- [ ] `/status` shows current mute settings clearly
|
||||
- [ ] Users without any settings in `user_settings` table receive all notifications (default behavior unchanged)
|
||||
@@ -0,0 +1,199 @@
|
||||
# Task: Location / postcode filters per keyword
|
||||
|
||||
## Description
|
||||
|
||||
Ads can match a user's keyword interest but be located in a completely different region (e.g., "rtx 3090" in Graz when the user only cares about Vienna). This task adds optional postcode filtering so users receive alerts only for ads in their desired locations.
|
||||
|
||||
The willhaben API returns location data in `LOCATION_CityName`, `LOCATION_ZIP` (postcode), and similar fields. We'll match against these fields.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ keywords table (extended) │
|
||||
│ │
|
||||
│ ... │
|
||||
│ allowed_postcodes text[] │
|
||||
│ ... │
|
||||
│ │
|
||||
│ Example: │
|
||||
│ allowed_postcodes = {'1010','1020'} │
|
||||
│ → only ads in Vienna 1st/2nd dist. │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Processing filter (in main.py) │
|
||||
│ │
|
||||
│ for ad in ads_raw: │
|
||||
│ ad_zip = _extract_postcode(ad) │
|
||||
│ if kw.allowed_postcodes and │
|
||||
│ ad_zip not in postcodes: │
|
||||
│ skip │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Bot commands (in bot.py): │
|
||||
│ │
|
||||
│ /postcode <keyword> p1,p2,p3 │
|
||||
│ → sets allowed_postcodes = {'p1', │
|
||||
│ 'p2','p3'} │
|
||||
│ /clear_postcode <keyword> │
|
||||
│ → removes filter (NULL) │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Text array** (`text[]`) instead of a separate lookup table. Simple, efficient for the typical case (<10 postcodes per keyword), and leverages PostgreSQL's native array support.
|
||||
- *Alternative*: A `keyword_postcodes` junction table allows individual postcode management but adds unnecessary complexity for this use case.
|
||||
- **Match against willhaben's ZIP code field** (`LOCATION_ZIP` in the ad attributes). This is the most reliable location identifier and works across all Austrian postcodes (4 digits, e.g., "1010", "8010").
|
||||
- **Empty or missing postcode = skip if filter active**. If `allowed_postcodes` is set but an ad has no ZIP code, it's excluded. This prevents noise from unlocated ads.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration
|
||||
|
||||
In `worker/src/migrations/03-keyword-filters.sql`:
|
||||
|
||||
```sql
|
||||
ALTER TABLE keywords ADD COLUMN IF NOT EXISTS allowed_postcodes text[];
|
||||
|
||||
COMMENT ON COLUMN keywords.allowed_postcodes IS
|
||||
'Austrian postcodes (4-digit strings). Only ads matching these are notified.';
|
||||
```
|
||||
|
||||
### 2. Extract postcode from ad data in `notifier.py` or `scraper.py`
|
||||
|
||||
Add helper function:
|
||||
|
||||
```python
|
||||
def _extract_postcode(ad_dict: dict) -> str | None:
|
||||
"""Extract the postal code (ZIP) from a willhaben ad."""
|
||||
attrs = _parse_attributes(ad_dict)
|
||||
|
||||
# Try multiple field names that willhaben might use
|
||||
for key in ("LOCATION_ZIP", "LocationZip", "postalcode"):
|
||||
val = attrs.get(key) or attrs.get(f"{key}_String")
|
||||
if val:
|
||||
return str(val).strip()
|
||||
|
||||
return None
|
||||
```
|
||||
|
||||
### 3. Add postcode filter check in `main.py` scheduler loop
|
||||
|
||||
```python
|
||||
async def _check_postcode_filter(
|
||||
ad_dict: dict,
|
||||
kw_row: dict
|
||||
) -> bool:
|
||||
"""Return True if the ad passes the postcode filter."""
|
||||
|
||||
postcodes = kw_row.get("allowed_postcodes") # list or None
|
||||
|
||||
if not postcodes:
|
||||
return True # no filter active
|
||||
|
||||
from notifier import _extract_postcode # or wherever it lives
|
||||
|
||||
ad_zip = _extract_postcode(ad_dict)
|
||||
|
||||
if not ad_zip:
|
||||
logger.debug("No postcode found in ad, skipping")
|
||||
return False
|
||||
|
||||
# Normalize: willhaben returns "1010" as string, we store same way
|
||||
return ad_zip in postcodes
|
||||
|
||||
|
||||
# In the scheduler loop (after price check):
|
||||
for ad_dict in ads_raw:
|
||||
if not await _check_price_filters(ad_dict, kw_row):
|
||||
continue
|
||||
|
||||
if not await _check_postcode_filter(ad_dict, kw_row):
|
||||
continue
|
||||
|
||||
# ... rest of processing
|
||||
```
|
||||
|
||||
### 4. Add bot commands in `bot.py`
|
||||
|
||||
```python
|
||||
async def cmd_set_postcode(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Set allowed postcodes for a keyword."""
|
||||
if len(context.args) < 2:
|
||||
await update.message.reply_text("Usage: /postcode <keyword> <p1,p2,p3>")
|
||||
return
|
||||
|
||||
kw_name = context.args[0]
|
||||
postcode_strs = [pc.strip() for pc in context.args[1].split(",")]
|
||||
|
||||
# Validate format (4-digit Austrian postcodes)
|
||||
invalid = [pc for pc in postcode_strs if not re.match(r"^\d{3,5}$", pc)]
|
||||
if invalid:
|
||||
await update.message.reply_text(
|
||||
f"Invalid postcode(s): {', '.join(invalid)}. "
|
||||
"Use 4-digit format like 1010, 8010."
|
||||
)
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
kw_id = await pool.fetchval(
|
||||
"""SELECT id FROM keywords
|
||||
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
|
||||
kw_name.lower(), telegram_id,
|
||||
)
|
||||
|
||||
if not kw_id:
|
||||
await update.message.reply_text(f"Keyword '{kw_name}' not found.")
|
||||
return
|
||||
|
||||
await pool.execute(
|
||||
"UPDATE keywords SET allowed_postcodes = $1 WHERE id = $2",
|
||||
postcode_strs, kw_id, # asyncpg handles text[] natively
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Keyword '{kw_name}': postcodes set to {', '.join(postcode_strs)}"
|
||||
)
|
||||
|
||||
|
||||
async def cmd_clear_postcode(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Remove postcode filter for a keyword."""
|
||||
if len(context.args) < 1:
|
||||
await update.message.reply_text("Usage: /clear_postcode <keyword>")
|
||||
return
|
||||
|
||||
# ... same pattern, set allowed_postcodes = NULL
|
||||
```
|
||||
|
||||
Register handlers:
|
||||
```python
|
||||
dp.add_handler(MessageHandler(REGEX(r"^/postcode"), cmd_set_postcode))
|
||||
dp.add_handler(MessageHandler(REGEX(r"^/clear_postcode"), cmd_clear_postcode))
|
||||
```
|
||||
|
||||
### 5. Update `/keywords` command output
|
||||
|
||||
Add postcode info to the listing:
|
||||
```python
|
||||
if row["allowed_postcodes"]:
|
||||
line += f"\n 📍 {', '.join(row['allowed_postcodes'])}"
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/postcode keyword 1010,1020` sets allowed postcodes to `{'1010', '1020'}` for that keyword
|
||||
- [ ] Ads with ZIP codes NOT in the allowed list are skipped during processing
|
||||
- [ ] Ads with no ZIP code at all are skipped when a filter is active
|
||||
- [ ] `/clear_postcode keyword` removes the filter (NULL)
|
||||
- [ ] Invalid postcodes (non-numeric or wrong length) are rejected by the bot
|
||||
- [ ] The `/keywords` command shows active postcodes next to each keyword
|
||||
- [ ] Both price AND postcode filters work correctly together (ad must pass both to be notified)
|
||||
@@ -0,0 +1,254 @@
|
||||
# Task: Price range filters per keyword
|
||||
|
||||
## Description
|
||||
|
||||
Currently, every ad that matches a keyword triggers a notification regardless of price. For keywords like "rtx" or "3090", this means users receive alerts for €5 listings (accessories) alongside relevant hardware deals.
|
||||
|
||||
This task adds `price_min` and `price_max` columns to the `keywords` table and implements server-side filtering during ad processing. Users set these via bot commands, and ads outside the range are silently skipped without being stored in the `ads` table or triggering notifications.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────┐
|
||||
│ keywords table (extended) │
|
||||
│ │
|
||||
│ ... │
|
||||
│ price_min int │
|
||||
│ price_max int │
|
||||
│ ... │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Processing pipeline (in main.py) │
|
||||
│ │
|
||||
│ for ad in ads_raw: │
|
||||
│ price = _extract_price(ad) │
|
||||
│ if price is None: │
|
||||
│ continue │
|
||||
│ if kw.price_min and price < min: │
|
||||
│ skip (below minimum) │
|
||||
│ if kw.price_max and price > max: │
|
||||
│ skip (above maximum) │
|
||||
│ → process ad normally │
|
||||
└──────────┬────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Bot commands (in bot.py): │
|
||||
│ │
|
||||
│ /price_min <keyword> <€amount> │
|
||||
│ /price_max <keyword> <€amount> │
|
||||
│ /clear_price <keyword> │
|
||||
│ /keywords → shows price ranges │
|
||||
└───────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Filter during processing, not at DB time** — the filter is applied in Python before inserting into `ads`. This keeps the `ads` table clean (only relevant ads are stored) and avoids extra WHERE clauses on every scrape cycle.
|
||||
- *Alternative*: Could use a computed column or trigger, but adds complexity for simple numeric comparison.
|
||||
- **Both filters optional** — NULL = no limit. Users can set only min, only max, both, or neither.
|
||||
- **Price extraction from ad data**: Use the existing `_extract_price()` function in `notifier.py`. If price is not available (free items), treat as €0 for filtering purposes.
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### 1. Add migration
|
||||
|
||||
In `worker/src/migrations/03-keyword-filters.sql`:
|
||||
|
||||
```sql
|
||||
ALTER TABLE keywords
|
||||
ADD COLUMN IF NOT EXISTS price_min int,
|
||||
ADD COLUMN IF NOT EXISTS price_max int;
|
||||
|
||||
COMMENT ON COLUMN keywords.price_min IS 'Minimum price in cents (e.g. 5000 = €50). NULL = no limit.';
|
||||
COMMENT ON COLUMN keywords.price_max IS 'Maximum price in cents (e.g. 500000 = €5000). NULL = no limit.';
|
||||
```
|
||||
|
||||
### 2. Update `main.py` — filter ads by price during processing
|
||||
|
||||
In the scheduler's keyword loop, before calling `_process_ad()`:
|
||||
|
||||
```python
|
||||
# After extracting ad info in _process_ad():
|
||||
async def _check_price_filters(
|
||||
ad_dict: dict,
|
||||
kw_row: dict # from keywords table
|
||||
) -> bool:
|
||||
"""Return True if the ad passes price filters for this keyword."""
|
||||
|
||||
from notifier import _extract_price
|
||||
|
||||
price = _extract_price(ad_dict)
|
||||
|
||||
if price is None:
|
||||
# No price found — include it (could be "free" or missing data)
|
||||
return True
|
||||
|
||||
price_min = kw_row.get("price_min")
|
||||
price_max = kw_row.get("price_max")
|
||||
|
||||
if price_min is not None and price < price_min:
|
||||
logger.debug(
|
||||
"Price filter skip: ad %s price=%d min=%d",
|
||||
ad_id, price, price_min
|
||||
)
|
||||
return False
|
||||
|
||||
if price_max is not None and price > price_max:
|
||||
logger.debug(
|
||||
"Price filter skip: ad %s price=%d max=%d",
|
||||
ad_id, price, price_max
|
||||
)
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
# In the scheduler loop:
|
||||
for kw_row in keywords:
|
||||
if not kw_row["is_active"]:
|
||||
continue
|
||||
|
||||
ads_raw = await fetch_ads(kw_row["keyword"])
|
||||
|
||||
for ad_dict in ads_raw:
|
||||
# Check price filters BEFORE processing
|
||||
if not await _check_price_filters(ad_dict, kw_row):
|
||||
continue
|
||||
|
||||
# Existing _process_ad logic continues here...
|
||||
```
|
||||
|
||||
### 3. Add bot commands in `bot.py`
|
||||
|
||||
```python
|
||||
from telegram import Message
|
||||
from telegram.ext import ContextTypes
|
||||
|
||||
|
||||
async def cmd_set_price_min(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Set minimum price for a keyword."""
|
||||
if len(context.args) < 2:
|
||||
await update.message.reply_text("Usage: /price_min <keyword> <amount_in_euro>")
|
||||
return
|
||||
|
||||
kw_name = context.args[0]
|
||||
try:
|
||||
amount_eur = float(context.args[1])
|
||||
price_cents = int(amount_eur * 100)
|
||||
except ValueError:
|
||||
await update.message.reply_text("Invalid amount. Use a number like 50 or 99.99")
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
# Find keyword by name for this user
|
||||
kw_id = await pool.fetchval(
|
||||
"""SELECT id FROM keywords
|
||||
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
|
||||
kw_name.lower(), telegram_id,
|
||||
)
|
||||
|
||||
if not kw_id:
|
||||
await update.message.reply_text(f"Keyword '{kw_name}' not found for your account.")
|
||||
return
|
||||
|
||||
await pool.execute(
|
||||
"UPDATE keywords SET price_min = $1 WHERE id = $2",
|
||||
price_cents, kw_id,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Keyword '{kw_name}': minimum price set to €{amount_eur:.2f}"
|
||||
)
|
||||
|
||||
|
||||
async def cmd_set_price_max(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Set maximum price for a keyword."""
|
||||
# Same pattern as cmd_set_price_min but sets price_max
|
||||
|
||||
if len(context.args) < 2:
|
||||
await update.message.reply_text("Usage: /price_max <keyword> <amount_in_euro>")
|
||||
return
|
||||
|
||||
kw_name = context.args[0]
|
||||
try:
|
||||
amount_eur = float(context.args[1])
|
||||
price_cents = int(amount_eur * 100)
|
||||
except ValueError:
|
||||
await update.message.reply_text("Invalid amount. Use a number like 50 or 99.99")
|
||||
return
|
||||
|
||||
telegram_id = str(update.effective_user.id)
|
||||
|
||||
from db import get_pool
|
||||
pool = await get_pool()
|
||||
|
||||
kw_id = await pool.fetchval(
|
||||
"""SELECT id FROM keywords
|
||||
WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
|
||||
kw_name.lower(), telegram_id,
|
||||
)
|
||||
|
||||
if not kw_id:
|
||||
await update.message.reply_text(f"Keyword '{kw_name}' not found.")
|
||||
return
|
||||
|
||||
await pool.execute(
|
||||
"UPDATE keywords SET price_max = $1 WHERE id = $2",
|
||||
price_cents, kw_id,
|
||||
)
|
||||
|
||||
await update.message.reply_text(
|
||||
f"✅ Keyword '{kw_name}': maximum price set to €{amount_eur:.2f}"
|
||||
)
|
||||
|
||||
|
||||
async def cmd_clear_price(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
|
||||
"""Clear price filters for a keyword."""
|
||||
if len(context.args) < 1:
|
||||
await update.message.reply_text("Usage: /clear_price <keyword>")
|
||||
return
|
||||
|
||||
# ... similar pattern, set both to NULL
|
||||
```
|
||||
|
||||
Register handlers in `register_handlers()`:
|
||||
```python
|
||||
dp.add_handler(MessageHandler(
|
||||
REGEX(r"^/price_min"), cmd_set_price_min))
|
||||
dp.add_handler(MessageHandler(
|
||||
REGEX(r"^/price_max"), cmd_set_price_max))
|
||||
dp.add_handler(MessageHandler(
|
||||
REGEX(r"^/clear_price"), cmd_clear_price))
|
||||
```
|
||||
|
||||
### 4. Update `/keywords` command output to show price ranges
|
||||
|
||||
In the existing `/keywords` handler in `bot.py`, add:
|
||||
|
||||
```python
|
||||
price_info = ""
|
||||
if row["price_min"] is not None:
|
||||
price_info += f"min €{row['price_min']/100:.2f} "
|
||||
if row["price_max"] is not None:
|
||||
price_info += f"max €{row['price_max']/100:.2f}"
|
||||
if price_info:
|
||||
line += f"\n 💰 {price_info.strip()}"
|
||||
```
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `/price_min keyword 50` sets the minimum price to €50.00 (stored as 5000 cents) for that keyword
|
||||
- [ ] `/price_max keyword 1000` sets the maximum price to €1000.00
|
||||
- [ ] Ads with price below `price_min` are skipped during processing and not inserted into `ads` table
|
||||
- [ ] Ads with price above `price_max` are skipped during processing
|
||||
- [ ] `/clear_price keyword` removes both limits (sets to NULL)
|
||||
- [ ] Keywords without price filters continue to work as before (no regression)
|
||||
- [ ] The `/keywords` command displays the active price range for each keyword
|
||||
- [ ] Non-admin users can only modify their own keywords' price filters
|
||||
Reference in New Issue
Block a user