EquiTable

Atlanta's food-insecurity problem was as much about connection as supply. Hours, eligibility, and ID requirements were scattered or missing. EquiTable finds pantries near any location, extracts that structured data with an LLM, and streams it onto a live map as each source resolves. The hard part is keeping it fresh, so a curator agent re-checks the stalest pantries first, and a crashed run resumes where it stopped instead of starting over.

Architecture

The refresh agent is a LangGraph pipeline: a curator picks the stalest pantries, then a per-source subgraph scrapes, extracts, and validates each one, stepping up a three-tier Gemini ladder when extraction fails and stopping at a per-run budget. I migrated the API and the agent to Kubernetes to make that pipeline recoverable and observable.

• Crash-resume that actually resumes. Checkpoints were being written but never read back, because every restart minted a new thread ID. Anchoring the MongoDB checkpointer to the Kubernetes Job name fixed it; killing a pod mid-crawl, the retry skipped the work already done.
• Readiness checks MongoDB and liveness doesn't, so an Atlas blip pulls pods out of rotation instead of restarting all of them in a loop.
• The crawl runs as a CronJob with Forbid concurrency and retry and deadline limits, behind default-deny NetworkPolicies enforced by Calico.
• The API is scraped by Prometheus; the short-lived crawl pushes to a Pushgateway, since a 30-second pull would miss a pod that lives a few minutes.
• A version-controlled Grafana dashboard and 4 alert rules. The job-failure alert ignores jobs that later succeed, so a successful crash-resume doesn't page, and the dashboard caught a scheduled run that had been silently dropped.
• Crawl4AI handles most pages, with Jina Reader as a free fallback, so scraping costs nothing. Terraform codifies the namespace, service accounts, and Atlas setup.

Results

Pantries scraped by Crawl4AI

~74% of 128

Scraping cost

$0

Automated tests

362

Architecture decision records

31

Stack

• React 19 + Vite
• FastAPI
• LangGraph
• Gemini
• LangSmith
• MongoDB Atlas
• Kubernetes
• Prometheus + Grafana
• Terraform