Case Study · Firestorm Internet
Scaling site production 5 → 32 in 4 weeks.
A multi-agent content pipeline on an async queue that took site production from a manual, days-per-site crawl to 32 published sites in a month — while keeping a human in the loop where the AI was least trustworthy.
- 5 → 32
- sites in 4 weeks
- ₹1L+
- commission, month one
- 4–10 days → hours
- per-site build time
- 1,200–1,800
- pages / month
§ 01 — The problem
The business runs on volume: many niche WordPress sites, each needing a configured theme, plugins, design, and a body of written content. Built by hand, a single site took 4–10 days — provisioning, plugin setup, design, then writing and publishing dozens of pages. That didn't scale to the targets, and most of the work was mechanical.
§ 02 — The pipeline
I built an end-to-end pipeline (Python + Bash + WP-CLI) that takes a blank WordPress install and produces a configured, content-populated, publish-ready site: path discovery and SSH checks, plugin cleanup and install, premium license activation, design customization (logo, colors, typography) via WP-CLI, layout and elements import, indexing settings, AI article generation over templates, image sourcing and interlinking, then publishing. The only human step left is a final image review.
§ 03 — The multi-agent turn
My first version used a single agent to do both the writing and the HTML/template generation. It was unreliable — the model juggling two jobs produced worse output on both. So I split it: one agent (Claude) owns content creation, a second path (Python + Claude) owns HTML/template generation. Quality jumped once each agent had a single responsibility. That "one agent, one job" lesson is the core of how I design agent systems now.
§ 04 — Treating the LLM as untrusted
LLM output is untrusted by default in this system. Every model response is validated against the structure I expect; malformed responses are retried before anything moves downstream. And I kept a deliberate human gate exactly where the AI was least reliable and a wrong output going live was most costly — image selection. Knowing when not to automate is as much the design as the automation.
§ 05 — The async backbone
Generation, image fetching, and publishing are long-running and failure-prone, so they don't run inline. I built an async job queue on Celery + Redis: every task is a durable worker with retry policies, exponential backoff, and dead-letter queues, so a transient API failure costs a retry, not a run. The same backbone powers an internal contact-management tool that aggregates enquiries across 40+ sites.
§ 06 — Outcome
The pipeline scaled production from 5 to 32 sites in 4 weeks and drove ₹1L+ in affiliate commission in the first month, with per-site build time dropping from days to a few hours and throughput reaching 1,200–1,800 pages a month. The whole thing is containerized and shipped through Google Cloud Build on GCP.
§ 07 — What I'd do next
Add an eval layer that scores generated content before publish (not just schema-validates it), so quality is measured, not assumed; widen the human gate into a lightweight review queue with quality signals; and add per-task observability — token cost, latency, retry rates — to tune the pipeline on data rather than intuition.