Provider-Crawl/n8n/PROCESS.md

Provider Discovery Pipeline — End-to-End Process

Plain-English walkthrough of what the n8n workflows do, in what order, and how the data they produce lands in the database.

The four workflows in workflows/ together form a continuous pipeline: Discover → Find websites → Enrich with pricing → Refresh periodically. Each workflow is an n8n schedule that shells out to Python scripts in /opt/crawlers (the crawlers/ folder, mounted into the n8n container).

---

The big picture

We're trying to populate the site with every funeral director in Australia, even before they've signed up with us. A provider starts life as a name and phone number from a public register and progressively gets enriched — website, description, packages, prices — until it either has enough data to be useful, or we've exhausted what's publicly available.

All discovered providers are hidden by default (funeral_brand.hidden = 1) and unverified (verified = 0) until an admin reviews them. The pipeline never modifies a provider that has signed up (verified = 1) — those are treated as authoritative.

A provider's data quality is summarised by a listing_tier:

Tier	Means
listed	Contact details only — we know the business exists
estimated	At least one package with a total price
priced	Two or more packages with itemised line items
verified	Signed-up partner (set manually, not by the pipeline)

The tier is recomputed after every enrichment pass and drives what the frontend shows.

---

Workflow 1 — Weekly Discovery

Runs: Mondays at 02:00 AEST File: workflows/1_weekly_discovery.json

What it does

Three source crawlers run in parallel against public registers:

1. VIC Consumer Affairs Register (crawl_vic_register.py) — ~796 Victorian funeral directors, scraped from the government register HTML.
2. Funerals Australia (crawl_funerals_australia.py) — ~997 members, fetched from their AJAX member-search API.
3. NFDA (crawl_nfda.py) — ~209 records from their WordPress store-locator API.

Each crawler writes its raw response to source_record and logs the run to source_log. Then the merge step waits for all three to finish and dedup.py runs, which is the interesting part: it matches records across sources by a combination of fuzzy name + postcode + (when available) ABN, merges duplicates into a single funeral_brand row, and attaches the per-source records to it.

Finally n8n queries how many new listed-tier providers appeared in the last 7 days and emits a summary.

Where the data lands

• source_log — one row per crawler run (start/finish, counts, errors).
• source_record — one row per raw record pulled from each source (e.g. a VIC Register entry). raw_data is the JSON as retrieved; normalized_data is the cleaned version.
• funeral_brand — one row per unique business (post-dedup). Receives title, phone, email, website (if the source provided one), business_address, business_suburb, business_state, business_postcode, source_key, source_url. hidden = 1, verified = 0, enrichment_status = 'pending', listing_tier = 'listed'.
• location — one or more rows per brand (multi-location providers). Receives title, address, suburb, state, postcode, lat/lng where the source provides them.
• source_record.matched_brand_id — back-pointer to the funeral_brand row that each raw record was merged into, with match_type indicating how (e.g. abn, name_postcode, fuzzy_name).

---

Workflow 2 — Daily Website Discovery

Runs: Every day at 04:00 AEST File: workflows/2_daily_website_discovery.json

What it does

For providers where funeral_brand.website IS NULL, tries to find a website in two passes:

1. ABN Lookup (lookup_abn.py) — calls the free Australian Business Register API to validate the business is real and attach a verified ABN + registered state/postcode. This doesn't find websites, but it strengthens the dedup key and marks the business as active.
2. Website discovery (discover_websites.py) — uses three strategies in order:
   • Serper.dev — Google-backed search ("{business name} {suburb} {state}"), takes the first non-directory result. 2,500 free queries.
   • DuckDuckGo lite — free fallback when Serper isn't configured or exhausted.
   • URL guessing — generates plausible domains from the business name (e.g. smithfunerals.com.au) and checks if they're live.

Each candidate URL is fetched and validated: the page must load, the title/body must mention the business name, and the domain must not be a known directory (Yellow Pages, True Local, etc.). A confidence level (confirmed/probable/unverified) is recorded.

Each run processes a batch of 100 providers. With ~469 needing websites, a fresh dataset fills up in ~5 days.

Where the data lands

• funeral_brand.abn — from ABR lookup.
• funeral_brand.website — the validated URL, if found.
• funeral_brand.business_state / business_postcode — overwritten with ABR values if they were missing or lower-quality.
• source_record — a new row with source_name = 'website_discovery' capturing the search query, all candidates considered, and why each was rejected. Useful for audit.

---

Workflow 3 — Daily Enrichment

Runs: Every day at 06:00 AEST File: workflows/3_daily_enrichment.json

This is the most complex workflow and the one that produces pricing data. It has two phases.

Phase A — Crawl websites (Python)

enrich_websites.py --limit=50 runs first, picking up providers where website IS NOT NULL AND enrichment_status = 'pending'. For each:

1. Fetch the homepage; extract meta description into funeral_brand.description.
2. Try ~20 common pricing URL patterns (/pricing, /packages, /funeral-costs, /transparency, etc.), parse the sitemap, and follow any link whose text contains "pric", "packag", "cost", or "service".
3. If a pricing page is found, save the cleaned body text. If a pricing PDF is linked, record its URL.
4. Write the result to source_record as source_name = 'website_crawl' — raw_data includes pricing_text, pricing_url, pdf_links, has_pricing flag.

At this point we have raw pricing text but no structured packages yet.

Phase B — AI extraction (n8n + Claude Haiku)

n8n then queries source_record for unprocessed website crawls that have pricing text (>100 chars):

1. For each, it pulls the full pricing text (up to 5000 chars).
2. Sends it to Claude Haiku with a strict JSON schema prompt asking for packages, funeral types, prices, and inclusions. The prompt constrains funeralType to the five allowed enum values and nudges toward the 16 standard inclusion type names.
3. Parses the JSON response (tolerant of markdown wrapping).
4. Inserts the packages and inclusions back into the DB.
5. Marks the source record processed and the brand as enrichment_status = 'complete'.

Finally compute_tiers.py runs and promotes brands whose new data now meets the estimated or priced thresholds.

Batch size is 20 AI extractions per run. At ~$0.002 per call, a full 469-provider pass costs ~$1.

Where the data lands

• funeral_brand.description — from meta tags on the homepage.
• funeral_brand.enrichment_status — 'complete' on success, 'partial' or 'failed' otherwise.
• funeral_brand.last_enriched_at — timestamp, used by Workflow 4.
• source_record — source_name = 'website_crawl' with raw_data.pricing_text, pricing_url, pdf_links, has_pricing. processed_at is set once AI extraction completes.
• package — one row per package found. title, funeral_type (constrained enum), brand_id, source_url = 'ai_extraction', extraction_confidence = 0.7.
• package_inclusion — one row per line item inside each package. price, optional, complimentary, inclusion_type_title, package_id.
• funeral_brand.listing_tier — recomputed by compute_tiers.py.

How the listing tier gets computed

compute_tiers.py looks at each brand's packages:

• 2+ packages, each with at least one priced inclusion → priced.
• 1+ packages with a total price → estimated.
• Everything else → listed.
• verified = 1 always beats the computed tier.

---

Workflow 4 — Monthly Refresh

Runs: 1st of each month at 03:00 AEST File: workflows/4_monthly_refresh.json

What it does

Pricing changes. Providers update their sites, add packages, drop services. This workflow keeps the dataset fresh:

1. Find providers where verified = 0 AND website IS NOT NULL AND last_enriched_at < 30 days ago.
2. Set their enrichment_status back to 'pending'.
3. Re-run enrich_websites.py --limit=200 against them — this re-crawls pricing pages and writes fresh source_record rows (old ones are kept for audit/history).
4. Workflow 3 will then pick them up over the following days for AI re-extraction.
5. compute_tiers.py runs to catch any tier changes.

New packages are inserted alongside old ones; compute_tiers looks at the current set. (A cleanup of stale packages isn't wired up yet — noted in crawlers/PIPELINE.md as a future improvement.)

Where the data lands

Same tables as Workflow 3, but you'll see multiple source_record rows per brand over time, which forms a change history.

---

Schema summary

funeral_brand (the provider — one per business)
  ├─ location (1..n — physical premises with lat/lng)
  ├─ package (0..n — a pricing offering)
  │    └─ package_inclusion (0..n — line items inside the package)
  ├─ known_for (0..n — descriptive tags, not yet populated by pipeline)
  └─ brand_funeral_area (many-to-many → funeral_area — service coverage, not yet populated)

source_log (one per crawler run)
source_record (one per raw record from a source, linked back to funeral_brand)

Pipeline never touches funeral_home (the parent corporation, e.g. InvoCare) or funeral_area (service area definitions) — those are populated manually or from other processes.

Columns the pipeline writes vs. leaves alone

Column	Written by	Notes
funeral_brand.title	WF1	From source registries
funeral_brand.phone, email	WF1	From source registries
funeral_brand.website	WF1 or WF2	Source registry if given, else discovered
funeral_brand.abn	WF2	From ABR
funeral_brand.description	WF3	Meta tags
funeral_brand.business_*	WF1/WF2	Preferring ABR values where available
funeral_brand.enrichment_status	WF3/WF4	State machine: pending → partial → complete, failed on error
funeral_brand.last_enriched_at	WF3	Used by WF4 for staleness check
funeral_brand.listing_tier	compute_tiers.py	After WF3/WF4
funeral_brand.source_key, source_url	WF1	Immutable once set
funeral_brand.verified, hidden	Never written by pipeline	Admin-only
funeral_brand.background_colour, foreground_colour, modal_description, funeral_home_id	Never written by pipeline	Admin/branding concern
package.*	WF3 (Claude Haiku)	source_url = 'ai_extraction', confidence 0.7
package_inclusion.*	WF3 (Claude Haiku)	inclusion_type_title pulled from a 16-item vocabulary
location.*	WF1	lat/lng only when source provides; google_place_key/rating require Places API (not yet wired)

The admin review flow (out of pipeline scope)

A provider stays hidden = 1 until an admin reviews it. The intended flow (not yet built — listed under "What's left to do" in the memory) is:

1. Admin UI lists newly enriched brands, sorted by tier.
2. Admin sets hidden = 0 to publish. They can also set verified = 1 if the provider has signed on as a partner — this protects them from future pipeline updates.

---

Running manually vs. via n8n

Everything n8n does can be reproduced with shell commands. The crawlers/run_overnight.sh script is effectively a single-pass equivalent of Workflows 1–3 back-to-back, useful for local testing or if n8n isn't available.

The n8n workflows are the production scheduler — they batch smaller chunks, run them at sensible hours (keeping server load and external API rate limits in mind), and handle the Claude Haiku HTTP calls natively (the Python scripts don't do AI extraction; they only prepare the text for n8n to send).

See README.md in this folder for setup.