3:20 PM - 4:40 PM UTC (1h 20min duration)
Pomerium Zero Outage: no cluster ID is available
The Pomerium Zero API outage on November 3, 2026 was resolved at 16:40 UTC after the affected deployment was rolled back. The incident began at 15:20 UTC and lasted 1 hour and 20 minutes. This timeline was added retrospectively.
We are sorry for the disruption and for the delay in communicating about it. This report explains the impact, cause, resolution, and areas where we need to improve.
From 15:20 to 16:40 UTC on November 3, Pomerium Zero API requests from clusters failed with HTTP 400 errors containing "no cluster ID available." Pomerium Core treated these bootstrap errors as terminal and exited, interrupting access to applications served by affected Pomerium Zero deployments.
A production deployment at 15:20 UTC introduced the failure. A customer support report at approximately 16:20 UTC alerted us to the incident. Additional customer reports and our own observations confirmed the impact.
We rolled back the deployment and restored service at 16:40 UTC. We documented the incident and its timeline retrospectively.
An automated dependency update changed the behavior of oapi-codegen, the library used to generate API server code. The generated code no longer set security-scope context values by default.
Pomerium Zero's authentication middleware relied on those values to decide whether an endpoint required cluster identity credentials. When the values were absent, the middleware skipped loading those credentials. The cluster API then rejected requests because it had no cluster identity.
We applied the dependency's recommended compatibility change, but it did not cover the cluster endpoints generated in Pomerium Core. Those endpoints continued to omit the context values that Zero expected.
Requests failed closed with HTTP 400 errors. Access was denied, so this was an availability incident.
Existing unit tests did not detect the problem because they bypassed the authentication checks involved in the failure. Our staging usage also did not exercise the affected cluster API path.
Rolling back the production deployment restored service.
We also changed the middleware to use an explicit list of public endpoints. Cluster credential handling no longer depends on the generated security-scope context values.
Our testing needs to cover authenticated cluster API requests from end to end, including bootstrap behavior. Staging validation must exercise these same paths before deployment.
Our production alerts did not identify this outage before customer reports. We need earlier detection of cluster API failures and their effect on customer deployments.
Our incident response and customer communication were too slow. We should have declared an outage and updated the status page as soon as we confirmed the impact.
Pomerium Zero is designed to continue using existing configuration when its API is unavailable. In this incident, HTTP 400 responses caused terminal bootstrap failures instead. The review identified a need to improve that error handling and fallback behavior so deployments are more resilient to API failures.
No incidents reported
No incidents reported