PyPI incident: Redirect Loops on JSON API endpoints.

Affected components

• files.pythonhosted.org - Redirects
• files.pythonhosted.org - Redirects Backends
• pypi.org - Backends
• pypi.org - CDN

1. postmortemJun 10, 2022, 16:32 UTC

   ## Summary PyPI's JSON API experienced an incident that caused redirect loops to occur for all clients when requesting many of the endpoints available on the service. These endpoints provide JSON documents describing projects and specific releases of projects hosted on [pypi.org](http://pypi.org). This incident was initiated by changes intended to make the JSON API more performant and less of a burden on the PyPI backends. Specifically, the combination of [this change to the warehouse codebase](https://github.com/pypa/warehouse/pull/11546) and [this change to the CDN configuration for PyPI](https://github.com/python/pypi-infra/pull/87) update the canonical locaiton for accessing the JSON documents deterministic. This allows for deeper cache efficiency and the ability to move redirects to the canonical URL out to the CDN edge. The result is faster response times for clients and reduced load on the PyPI backends. The redirect updates specifically had an unintended consequence that le

2. resolvedJun 10, 2022, 16:32 UTC

   This incident has been resolved.

3. monitoringJun 10, 2022, 16:01 UTC

   All purges of JSON API documents have completed. Our backends are recovering from the added load of repopulating the entire cache. Any failed purges may result in latent redirect loops for specific projects or releases in the JSON API, these will self-resolve within 24 hours as caches expire.

4. identifiedJun 10, 2022, 13:54 UTC

   The cache purge has cleared all but projects starting with the letter `p`. Our estimates failed to take into consideration the popularity of project names on PyPI starting with p 🙃

5. identifiedJun 10, 2022, 12:38 UTC

   Our mass purge operation is continuing. Based on the current rate that we're able to process all purges the processes should be complete in 45-60 minutes.

6. identifiedJun 10, 2022, 11:34 UTC

   We have started a task which will iterate over all projects and purge the cache for each individually. This will keep the PyPI backends from being overloaded by a completely bare cache. This process will take some time to complete, current estimate is 1-2 hours.

7. identifiedJun 10, 2022, 11:08 UTC

   Some cached responses are causing redirect loops for endpoints on the JSON API. We are working to determine how to clear these cached values without impacting the overall health of PyPI.

What APIStatus.watch saw

Our 5-minute checks didn't record a change in PyPI's overall status around this incident. Smaller or regional incidents often leave a provider's overall status green.

How often does PyPI have incidents?

PyPI reported 3 incidents in the last 90 days, 1 of them major or critical. A typical incident lasted 6h 3m. See PyPI's uptime and incident history