Pillar
Failure Modes / 2 AM Pager
What breaks at 2 AM and why. Incident-shaped lessons from production AI systems.
2 AM PAGER
Airflow 3.3.2 shipped an upgrade trap, and the error message lies.
pandas 3 renames the DataFrame class path and XComs record the old one. Upgrade Airflow on every worker first, or the error message will send you chasing your config.
FAILURE MODES
Your demo worked. Your AI product won't.
Your demo ran on ten hand-picked examples. Production is the long tail of real inputs. Four things break, and none of them are the model.
2 AM PAGER
Your support bot is hallucinating. Blame the dumpster, not the model.
2:14 AM. The bot is inventing refund policies. The retriever is fine and the prompt is fine. The problem is the fourteen thousand tokens of dumpster you fed it.