Three AI Providers Went Down at Once. Would Your Product Keep Working?

On 3 September, ChatGPT and Codex, Claude, and Grok suffered overlapping outages. OpenAI’s status history recorded elevated errors across ChatGPT and Codex. Anthropic recorded elevated errors across several Claude models. Reports spread quickly enough for X to create dedicated trending topics.

The incidents did not necessarily share one cause. Their timing alone does not prove a common infrastructure failure, and production APIs, coding tools, and consumer chat interfaces were not affected in exactly the same way.

That uncertainty does not weaken the lesson. It makes the lesson more useful.

If an important workflow depends on an AI provider, “we can switch to another model” is not yet a resilience strategy. Several providers can fail in the same operational window. A fallback can be technically available but functionally unsuitable. It can also violate the data, security, latency, or cost assumptions of the original system.

The production question is therefore not “Which AI provider has the best uptime?”

It is:

“What does our product do when the AI capability is unavailable, slow, wrong, or only partly available?”

Serious software needs a precise answer before the next incident.

An AI Feature Can Become a Business-Critical Dependency

Many AI features begin as optional assistance. A model drafts a reply, summarises a document, proposes a code change, or classifies an incoming request. If it stops, a user can wait or complete the task manually.

The risk changes as the feature moves into the critical path.

Consider an AI component that:

• decides where customer requests are routed;
• extracts fields before an order can continue;
• reviews transactions for fraud;
• generates instructions used by an operations team;
• controls a coding or deployment workflow;
• turns unstructured documents into records required by another system.

When that component is unavailable, the business process may stop. When it returns malformed or low-quality output, the process may continue incorrectly. The second failure mode is often more dangerous because dashboards can remain green while bad decisions move downstream.

Production readiness therefore includes more than model accuracy. It includes availability, bounded latency, observable failure, recoverability, and a safe degraded mode.

Start With the Degraded Mode, Not the Fallback Model

Before choosing a second provider, define what the product should do without AI.

There are four common answers:

1. Continue without the enhancement. Search still works, but the AI summary disappears. A support form still submits, but automatic categorisation waits.
2. Queue the work. The request is stored safely and processed when the dependency recovers. This fits asynchronous work where delay is acceptable.
3. Route to a human. A person handles high-value or ambiguous cases using the original data, not a half-completed model output.
4. Stop safely. For high-risk actions, the correct fallback is no action. The system explains the interruption and preserves enough state to resume.

The right answer depends on the consequence of delay and the consequence of error. A marketing-copy helper can disappear for an hour. A model involved in fraud review, healthcare administration, or industrial operations needs stricter boundaries.

A degraded mode is a product decision expressed through architecture. It cannot be invented by an SDK retry loop during an incident.

Put the Model Behind a Real Boundary

Provider calls should not be scattered through controllers, page components, workflow scripts, and background jobs. Place them behind an application-owned boundary.

That boundary should define the capability the product needs, not merely mirror one provider’s API. For example:

• classifySupportRequest with a versioned result schema;
• extractInvoiceFields with confidence and provenance;
• draftReply with explicit policy and language requirements;
• proposeCodeChange with repository and tool constraints.

The application can then enforce timeouts, output validation, logging, cost limits, and routing in one place. Provider-specific model names, tool formats, authentication, and error types remain inside adapters.

This does not make models interchangeable. It makes their differences visible and testable. Without that boundary, emergency failover becomes a search-and-replace exercise across production code.

Protect the Rest of the Product

A slow dependency can be more damaging than a failed one. Requests accumulate, worker pools fill, database connections remain open, users retry, and one provider incident becomes an application incident.

Every AI call needs a time budget derived from the user journey or job deadline. It should not inherit an arbitrary SDK default.

When error rates or latency cross a threshold, a circuit breaker should stop sending work temporarily. This gives the provider time to recover and prevents the application from wasting resources on requests unlikely to succeed.

Retries must be bounded and selective. Retrying an overload or transient network error can help. Retrying an invalid request, policy rejection, or incompatible tool call usually cannot. Uncontrolled retries increase cost and add load at precisely the wrong moment.

The failure should then move into the defined degraded mode: queue, human review, reduced functionality, or safe stop.

Queue Only Work That Can Be Replayed Safely

Queues are useful for document processing, enrichment, report generation, and other asynchronous tasks. They are not magic.

If a job can run later, the system needs:

• durable storage for the original request;
• an idempotency key so replay does not create duplicate actions;
• a maximum age after which the result is no longer useful;
• clear retry and dead-letter rules;
• versioned prompts and schemas so delayed work is reproducible;
• status visible to users and operators.

Do not queue an ambiguous half-transaction and hope the model reconstructs intent later. Preserve the business state first. Treat model output as a proposed result that must still satisfy current validation before anything irreversible happens.

A Second Provider Helps, but It Is Not Enough

Multi-provider routing can reduce downtime. It is especially useful when providers have independent infrastructure and the task can tolerate differences in output.

But a model fallback is safe only when it has been tested against the same capability contract. Teams need to verify:

• structured-output and tool-call behaviour;
• context and file limits;
• latency under realistic load;
• safety refusals and failure semantics;
• language and domain quality;
• regional processing, retention, and contractual terms;
• cost during retries and peak traffic;
• whether the fallback depends on the same cloud, gateway, identity provider, or network route.

The last point matters after an outage cluster. Provider diversity on a diagram can hide shared infrastructure underneath. Even fully independent providers can experience separate incidents within the same period.

Fallback should therefore be capability-aware. A smaller model may safely classify routine requests but not execute a complex tool workflow. A local model may preserve a basic extraction function but not match the quality of the primary provider. The router should know those limits and select the corresponding degraded behaviour.

Seamless failover is not the goal. Predictable behaviour is.

Keep a Deterministic or Human Escape Hatch

The strongest fallback often uses no generative model at all.

A rules-based classifier can handle the most common support categories. A conventional search index can remain available when generative answers stop. Required invoice fields can be entered manually. A deployment pipeline can require human review instead of waiting for an agent.

These paths may be slower or less convenient. That is acceptable. A degraded mode should preserve the essential business outcome, not imitate every feature of the normal mode.

The escape hatch also needs regular use. A manual process documented in a forgotten wiki page will fail when people discover that permissions changed, the form no longer exists, or nobody owns the decision.

Observe the Capability, Not Just the HTTP Status

An AI endpoint can return HTTP 200 and still fail the product.

Useful observability includes:

• availability and latency by provider, model, region, and capability;
• timeout, refusal, validation, and tool-execution rates;
• queue depth and age of the oldest job;
• fallback activation and success rates;
• output-quality checks tied to business rules;
• cost per completed and accepted outcome;
• correlation IDs across the user request, model call, tool call, and downstream action.

Alerting should focus on user impact. A provider status page may report healthy aggregate service while one model, region, account tier, or tool remains impaired. Your own service-level indicators are the evidence that matters to your product.

Logs also need discipline. Do not solve observability by copying prompts, source code, personal data, and model output into an unrestricted logging platform. Record the metadata needed for diagnosis, apply redaction, and keep sensitive payload access narrow and auditable.

Run an AI Outage Drill

The architecture is not complete until the failure path has been exercised.

A practical game day can be small:

1. Block the primary provider endpoint in a test or staging environment.
2. Add latency and intermittent errors, not only a clean connection failure.
3. Return malformed structured output and incomplete tool calls.
4. Exhaust the fallback provider’s quota or make it unavailable too.
5. Confirm that circuit breakers open and recover correctly.
6. Replay queued work and check for duplicate side effects.
7. Verify the manual path with the people expected to use it.
8. Measure user-visible behaviour, recovery time, data integrity, and operational effort.

The drill should answer concrete questions. Can support see which cases need attention? Can a customer resume without re-entering data? Can operators disable one capability without taking down the entire product? Can the team explain which actions did and did not happen?

If those answers are unclear, another provider API key will not fix the architecture.

Match the Investment to the Business Risk

Not every AI feature needs active-active routing across three vendors. Resilience has a cost: additional integrations, tests, contracts, monitoring, and operational complexity.

A simple classification helps:

• Optional enhancement: show a clear unavailable state and preserve the core workflow.
• Important but delay-tolerant: queue durably, communicate status, and process later.
• Time-sensitive business function: use tested capability-aware fallback plus a human path.
• Safety- or compliance-critical action: default to a safe stop, explicit approval, and complete auditability.

This is architecture-first delivery: spend where failure changes revenue, safety, contractual obligations, or customer trust.

Availability Is a Product Behaviour

The September outage cluster will pass. Providers will recover, publish updates, and improve their systems. Another incident will eventually happen.

The lasting question is whether your software treats an AI model as a convenient tool or an invisible single point of failure.

Production-ready AI does not mean the model never goes down. It means the product has a tested answer when it does: bounded waiting, controlled retries, validated fallbacks, durable state, observable impact, and a safe path that preserves the essential business process.

That answer is built before the outage.