willhaben-tracker/docs/phase-2/task-postcode-filters.md

Task: Location / postcode filters per keyword

Description

Ads can match a user's keyword interest but be located in a completely different region (e.g., "rtx 3090" in Graz when the user only cares about Vienna). This task adds optional postcode filtering so users receive alerts only for ads in their desired locations.

The willhaben API returns location data in LOCATION_CityName, LOCATION_ZIP (postcode), and similar fields. We'll match against these fields.

Architecture

┌───────────────────────────────────────┐
│  keywords table (extended)            │
│                                       │
│    ...                                 │
│    allowed_postcodes text[]           │
│    ...                                 │
│                                       │
│  Example:                              │
│    allowed_postcodes = {'1010','1020'} │
│    → only ads in Vienna 1st/2nd dist. │
└──────────┬────────────────────────────┘
           │
           ▼
┌───────────────────────────────────────┐
│  Processing filter (in main.py)      │
│                                       │
│  for ad in ads_raw:                   │
│    ad_zip = _extract_postcode(ad)     │
│    if kw.allowed_postcodes and       │
│       ad_zip not in postcodes:        │
│      skip                             │
└──────────┬────────────────────────────┘
           │
           ▼
┌───────────────────────────────────────┐
│  Bot commands (in bot.py):            │
│                                       │
│  /postcode <keyword> p1,p2,p3         │
│    → sets allowed_postcodes = {'p1',  │
│      'p2','p3'}                       │
│  /clear_postcode <keyword>           │
│    → removes filter (NULL)            │
└───────────────────────────────────────┘

Key design decisions

• Text array (text[]) instead of a separate lookup table. Simple, efficient for the typical case (<10 postcodes per keyword), and leverages PostgreSQL's native array support.
  • Alternative: A keyword_postcodes junction table allows individual postcode management but adds unnecessary complexity for this use case.
• Match against willhaben's ZIP code field (LOCATION_ZIP in the ad attributes). This is the most reliable location identifier and works across all Austrian postcodes (4 digits, e.g., "1010", "8010").
• Empty or missing postcode = skip if filter active. If allowed_postcodes is set but an ad has no ZIP code, it's excluded. This prevents noise from unlocated ads.

Implementation Details

1. Add migration

In worker/src/migrations/03-keyword-filters.sql:

ALTER TABLE keywords ADD COLUMN IF NOT EXISTS allowed_postcodes text[];

COMMENT ON COLUMN keywords.allowed_postcodes IS 
    'Austrian postcodes (4-digit strings). Only ads matching these are notified.';

2. Extract postcode from ad data in notifier.py or scraper.py

Add helper function:

def _extract_postcode(ad_dict: dict) -> str | None:
    """Extract the postal code (ZIP) from a willhaben ad."""
    attrs = _parse_attributes(ad_dict)
    
    # Try multiple field names that willhaben might use
    for key in ("LOCATION_ZIP", "LocationZip", "postalcode"):
        val = attrs.get(key) or attrs.get(f"{key}_String")
        if val:
            return str(val).strip()
    
    return None

3. Add postcode filter check in main.py scheduler loop

async def _check_postcode_filter(
    ad_dict: dict, 
    kw_row: dict
) -> bool:
    """Return True if the ad passes the postcode filter."""
    
    postcodes = kw_row.get("allowed_postcodes")  # list or None
    
    if not postcodes:
        return True  # no filter active
    
    from notifier import _extract_postcode  # or wherever it lives
    
    ad_zip = _extract_postcode(ad_dict)
    
    if not ad_zip:
        logger.debug("No postcode found in ad, skipping")
        return False
    
    # Normalize: willhaben returns "1010" as string, we store same way
    return ad_zip in postcodes


# In the scheduler loop (after price check):
for ad_dict in ads_raw:
    if not await _check_price_filters(ad_dict, kw_row):
        continue
    
    if not await _check_postcode_filter(ad_dict, kw_row):
        continue
    
    # ... rest of processing

4. Add bot commands in bot.py

async def cmd_set_postcode(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    """Set allowed postcodes for a keyword."""
    if len(context.args) < 2:
        await update.message.reply_text("Usage: /postcode <keyword> <p1,p2,p3>")
        return
    
    kw_name = context.args[0]
    postcode_strs = [pc.strip() for pc in context.args[1].split(",")]
    
    # Validate format (4-digit Austrian postcodes)
    invalid = [pc for pc in postcode_strs if not re.match(r"^\d{3,5}$", pc)]
    if invalid:
        await update.message.reply_text(
            f"Invalid postcode(s): {', '.join(invalid)}. "
            "Use 4-digit format like 1010, 8010."
        )
        return
    
    telegram_id = str(update.effective_user.id)
    
    from db import get_pool
    pool = await get_pool()
    
    kw_id = await pool.fetchval(
        """SELECT id FROM keywords 
           WHERE LOWER(keyword_name) = $1 AND telegram_id = $2""",
        kw_name.lower(), telegram_id,
    )
    
    if not kw_id:
        await update.message.reply_text(f"Keyword '{kw_name}' not found.")
        return
    
    await pool.execute(
        "UPDATE keywords SET allowed_postcodes = $1 WHERE id = $2",
        postcode_strs, kw_id,  # asyncpg handles text[] natively
    )
    
    await update.message.reply_text(
        f"✅ Keyword '{kw_name}': postcodes set to {', '.join(postcode_strs)}"
    )


async def cmd_clear_postcode(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
    """Remove postcode filter for a keyword."""
    if len(context.args) < 1:
        await update.message.reply_text("Usage: /clear_postcode <keyword>")
        return
    
    # ... same pattern, set allowed_postcodes = NULL

Register handlers:

dp.add_handler(MessageHandler(REGEX(r"^/postcode"), cmd_set_postcode))
dp.add_handler(MessageHandler(REGEX(r"^/clear_postcode"), cmd_clear_postcode))

5. Update /keywords command output

Add postcode info to the listing:

if row["allowed_postcodes"]:
    line += f"\n         📍 {', '.join(row['allowed_postcodes'])}"

Acceptance Criteria

• /postcode keyword 1010,1020 sets allowed postcodes to {'1010', '1020'} for that keyword
• Ads with ZIP codes NOT in the allowed list are skipped during processing
• Ads with no ZIP code at all are skipped when a filter is active
• /clear_postcode keyword removes the filter (NULL)
• Invalid postcodes (non-numeric or wrong length) are rejected by the bot
• The /keywords command shows active postcodes next to each keyword
• Both price AND postcode filters work correctly together (ad must pass both to be notified)