101 lines
4.0 KiB
Markdown
101 lines
4.0 KiB
Markdown
# Provider Crawl
|
||
|
||
Automated pipeline for discovering Australian funeral directors from public
|
||
registries, finding their websites, and extracting pricing data. Feeds the
|
||
Funeral Arranger platform with a seed of unverified providers that an admin
|
||
reviews before publishing.
|
||
|
||
## What's in this repo
|
||
|
||
```
|
||
crawlers/ Python modules — one per data source + shared pipeline utilities
|
||
n8n/ n8n workflows that orchestrate the crawlers on a schedule
|
||
database/ SQLite + Postgres schemas, a seed dump, and a pre-populated dev DB
|
||
```
|
||
|
||
Four documents explain how it works, in increasing depth:
|
||
|
||
1. **[ONBOARDING.md](ONBOARDING.md)** — Human context: what the project is
|
||
for, ground rules, open work, how to get unblocked. **Start here if you're
|
||
new to the project.**
|
||
2. **[n8n/PROCESS.md](n8n/PROCESS.md)** — Plain-English end-to-end walkthrough
|
||
of the four workflows and how their output maps to database tables.
|
||
3. **[crawlers/PIPELINE.md](crawlers/PIPELINE.md)** — Architecture of the
|
||
Python modules, source-by-source notes, listing-tier logic.
|
||
4. **[n8n/README.md](n8n/README.md)** — How to stand up n8n locally with
|
||
Docker and import the workflow JSONs.
|
||
|
||
## Quick start (local, no n8n)
|
||
|
||
```bash
|
||
# 1. Install Python deps (requests, beautifulsoup4, rapidfuzz, pdfplumber)
|
||
python3 -m venv .venv && source .venv/bin/activate
|
||
pip install requests beautifulsoup4 rapidfuzz pdfplumber
|
||
|
||
# 2. Configure API keys
|
||
cp crawlers/config.example.json crawlers/config.json
|
||
# Edit crawlers/config.json and add your Serper key (free 2,500/month from serper.dev)
|
||
# Optionally add an Anthropic API key for AI pricing extraction
|
||
|
||
# 3. Inspect the included dev database (1,463 providers, 121 with pricing)
|
||
sqlite3 database/providers.db ".tables"
|
||
sqlite3 database/providers.db "SELECT listing_tier, COUNT(*) FROM funeral_brand GROUP BY listing_tier"
|
||
|
||
# 4. Or start fresh and run the pipeline end-to-end
|
||
rm database/providers.db
|
||
sqlite3 database/providers.db < database/schema_sqlite.sql
|
||
cd crawlers
|
||
./run_overnight.sh # equivalent to workflows 1–3 back to back
|
||
```
|
||
|
||
## API keys needed
|
||
|
||
| Service | Purpose | Cost | Where |
|
||
|---|---|---|---|
|
||
| Serper.dev | Website discovery via Google search | 2,500 free/mo | https://serper.dev |
|
||
| ABR (Australian Business Register) | ABN validation | Free (GUID required) | https://abr.business.gov.au/Tools/WebServices |
|
||
| Anthropic | AI pricing extraction (Claude Haiku) | ~$2 / full run | https://console.anthropic.com |
|
||
|
||
Only Serper is required to run the full pipeline. Anthropic is only needed for
|
||
the AI extraction step in Workflow 3 (can be skipped — Python crawl still
|
||
populates pricing text into `source_record` for manual review).
|
||
|
||
## Database
|
||
|
||
The included `database/providers.db` is a live SQLite snapshot from the last
|
||
overnight run: 1,463 unique providers, 994 with websites, 121 with pricing.
|
||
|
||
Schema files:
|
||
|
||
- `database/schema_sqlite.sql` — SQLite schema, used for local dev
|
||
- `database/schema.sql` — Postgres schema (production target)
|
||
- `database/seed_verified.sql` — seed data for partner (`verified = 1`) providers
|
||
- `database/PROVIDER-SCHEMA-SPEC.md` — schema commentary and design notes
|
||
- `database/IMAGE-MAPPING.md` — asset conventions for verified providers
|
||
|
||
Per `n8n/PROCESS.md`, the pipeline only writes to a defined subset of columns —
|
||
`verified`, `hidden`, branding, and `funeral_home_id` are admin-only.
|
||
|
||
## Status
|
||
|
||
What's built and working:
|
||
|
||
- All three source crawlers (VIC Register, Funerals Australia, NFDA)
|
||
- Cross-source deduplication with fuzzy name/postcode/ABN matching
|
||
- Website discovery via Serper + DDG + URL guessing
|
||
- ABN lookup via ABR
|
||
- Website enrichment: meta descriptions, pricing page discovery, PDF detection
|
||
- Listing tier computation
|
||
- Four n8n workflow JSONs ready to import
|
||
|
||
What's not yet built (open to pick up):
|
||
|
||
- Google Places API integration for richer location data (ratings, place_id)
|
||
- Playwright-based enrichment for JS-rendered sites (~37% of sites)
|
||
- Admin review UI for approving hidden providers before they go live
|
||
- Stale package cleanup in the monthly refresh
|
||
|
||
## Contact
|
||
|
||
Richie — richie@tensordesign.com.au
|