Provider-Crawl/crawlers/PIPELINE.md

# Provider Discovery & Enrichment Pipeline

## Architecture: Multi-Step Enrichment

The pipeline builds provider profiles progressively, never relying on
competitor data. Each step adds richer detail from more authoritative sources.

```
STEP 1: DISCOVER               STEP 2: FIND WEBSITE           STEP 3: ENRICH
─────────────────               ────────────────────           ──────────────

VIC Register ─────┐                                           ┌─ Fetch homepage
NFDA Directory ───┼─▶ Basic     Google Places API ──┐         │  Find /pricing page
Funerals AU ──────┘   Provider  ABN Lookup ─────────┼─▶ URL ──┤  Download PDFs
                      Record    Search engines ─────┘         │  AI extract packages
                                                              └─▶ Structured data
                      name      website URL                      description
                      address   Google rating                    packages[]
                      phone     Google reviews                   inclusions[]
                      email     place_id                         pricing
                      state     ABN (validated)
```

## Step 1: Discovery (DONE — all modules built and tested)

Sources:
- VIC Consumer Affairs Register (796 records, VIC only) → `crawl_vic_register.py`
- Funerals Australia AJAX API (997 records, national) → `crawl_funerals_australia.py`
- NFDA WPSL API (209 records, national) → `crawl_nfda.py`

Orchestrator: `crawl_all.py`
Deduplication: `dedup.py` (fuzzy name + postcode + ABN matching)

Output: ~1,463 unique providers with basic contact info.
Stored in: funeral_brand + location tables in `database/providers.db`.

## Step 2: Website Discovery (DONE — module built and tested)

Module: `discover_websites.py`
Test result: 50% success rate on initial batch (DDG search + URL guessing)
Can be improved with Google Places API for higher hit rate.

For each provider that lacks a website URL:

### 2a. Serper.dev — Google search API (PRIMARY)
- Input: "{business name} {suburb} {state}"
- Returns: Google organic search results as JSON (title, link, snippet)
- Cost: **2,500 free queries** (no CC needed), then $1/1K
- Covers our entire 1,463 providers for $0
- Filters out directories/aggregators, validates first result
- Module: `discover_websites.py` with `search_serper()`

### 2b. DuckDuckGo lite (FALLBACK)
- Free, no API key, but aggressive rate limiting
- Used when Serper key not configured or quota exhausted
- Module: `discover_websites.py` with `search_ddg()`

### 2c. URL pattern guessing (SUPPLEMENTARY)
- Generates candidate domains from business name (e.g. smithfunerals.com.au)
- HTTP HEAD to check if live, then validate content
- Module: `discover_websites.py` with `guess_urls()`

### 2d. ABN Lookup — Australian Business Register (ENRICHMENT)
- Input: business name + state
- Returns: ABN, entity status, registered state/postcode
- Cost: **FREE** (government API, requires GUID registration)
- Validates business is active, gives strongest dedup key
- Does NOT return website URLs
- Module: `lookup_abn.py`
- Register for GUID: https://abr.business.gov.au/Tools/WebServices

### 2e. Google Places API (OPTIONAL PREMIUM)
- Input: "{business name}, {suburb} {state}"
- Returns: website, rating, review count, place_id, formatted phone
- Cost: 1,000 free/month (Enterprise tier), then ~$25/1K
- Best data quality but most expensive
- Not yet implemented — add when budget allows

### 2f. URL validation
- Fetch discovered URL, verify it loads
- Check page title/content mentions the business name
- Reject generic directories (yellowpages, truelocal, etc.)
- Mark confidence level: confirmed / probable / unverified

## Step 3: Website Enrichment (DONE — module built and tested)

Module: `enrich_websites.py`
- Finds pricing pages via 20+ URL patterns + link following
- Extracts description from meta tags
- Extracts contact info (phone, email, address)
- Stores cleaned pricing page text for AI extraction
- Detects PDF links for PDF-based pricing extraction

For each provider with a confirmed website:

### 3a. Homepage crawl
- Fetch homepage HTML
- Extract: description/about text, contact details
- Look for links to pricing/services pages

### 3b. Pricing page discovery
Try common URL patterns:
  /pricing, /prices, /packages, /services, /our-services,
  /funeral-costs, /funeral-packages, /service-options,
  /price-list, /transparency

Also:
- Parse sitemap.xml if available
- Follow links containing "pric", "packag", "cost", "service"
- Check for PDF links on pricing pages

### 3c. AI extraction (Claude Haiku)
- Send pricing page HTML to Haiku
- Extract: package names, funeral types, prices, inclusions
- Map to known inclusion types where possible
- Return confidence score

### 3d. PDF extraction (for InvoCare-type sites)
- Download compliance PDFs
- Extract text (pdftotext or similar)
- Send to Haiku for structured extraction
- ~25% of sites are PDF-only for pricing

## Listing Tiers

Providers are assigned a `listing_tier` based on data quality. Computed
automatically by `compute_tiers.py` after each enrichment run.

| Tier | Label | Criteria | Display |
|------|-------|----------|---------|
| `verified` | Full partner | `verified = true` (signed up) | Full branding, packages, arrangements |
| `priced` | Full pricing | 2+ packages with itemized inclusion prices | Package comparison, line-item detail |
| `estimated` | Some pricing | At least 1 package with a total price | Package prices shown, "Contact for details" on breakdowns |
| `listed` | Contact only | Name + location + phone, no pricing | "Contact for pricing" CTA, upgrade prompt |

Each tier below `verified` motivates the provider to sign up:
- `listed` → "Publish your pricing to attract more families"
- `estimated` → "Add detailed breakdowns to stand out"
- `priced` → "Sign up to enable online arrangements"

## Enrichment Status Flow

```
pending ──▶ website_found ──▶ partial ──▶ complete
   │              │               │
   └──▶ no_website_found    failed (retry later)
```

## N8N Workflow Design

### Workflow 1: Weekly Discovery
Cron → Run all source crawlers → Dedup into DB → Queue new providers

### Workflow 2: Daily Website Discovery
Cron → Fetch providers with no website → Google Places lookup
     → ABN lookup → Search fallback → Update DB

### Workflow 3: Daily Enrichment
Cron → Fetch providers with website but no packages
     → Crawl website → AI extract → Update DB

### Workflow 4: Monthly Re-check
Cron → Re-crawl enriched providers → Update pricing if changed

---

## Module Inventory

| Module | Purpose | N8N Workflow |
|--------|---------|-------------|
| `base.py` | Shared HTTP, DB, normalization utils | Used by all |
| `crawl_vic_register.py` | VIC government register (796 records) | Workflow 1 |
| `crawl_funerals_australia.py` | Funerals Australia API (997 records) | Workflow 1 |
| `crawl_nfda.py` | NFDA directory API (209 records) | Workflow 1 |
| `crawl_all.py` | Orchestrates all source crawlers | Workflow 1 |
| `dedup.py` | Cross-source dedup & merge engine | Workflow 1 |
| `discover_websites.py` | Find provider websites (Serper/DDG/guess) | Workflow 2 |
| `lookup_abn.py` | ABN validation via ABR API (free) | Workflow 2 |
| `enrich_websites.py` | Crawl provider sites, find pricing pages | Workflow 3 |
| `compute_tiers.py` | Compute listing_tier from data quality | After enrichment |
| `config.example.json` | API key template | — |

## API Keys Required

| Service | Key | Cost | Register |
|---------|-----|------|----------|
| Serper.dev | `serper_api_key` | 2,500 free, then $1/1K | https://serper.dev |
| ABR (ABN Lookup) | `abr_guid` | Free | https://abr.business.gov.au/Tools/WebServices |
| Anthropic (Haiku) | `anthropic_api_key` | ~$2/full run | https://console.anthropic.com |

## Quick Start

```bash
# 1. Configure API keys
cp config.example.json config.json
# Edit config.json with your keys

# 2. Reset database
cd ../database
sqlite3 providers.db < schema_sqlite.sql

# 3. Run full discovery pipeline
cd ../crawlers
python3 crawl_all.py          # Step 1: Discover from registries
python3 dedup.py              # Deduplicate across sources
python3 lookup_abn.py         # Step 2a: Get ABNs (free)
python3 discover_websites.py  # Step 2b: Find websites
python3 enrich_websites.py    # Step 3: Crawl for pricing
python3 compute_tiers.py      # Assign listing tiers

# Test mode (limited records)
python3 crawl_all.py --test
python3 discover_websites.py --limit=10 --state=VIC
python3 enrich_websites.py --limit=5
```