PyPI incident: Queue delay affecting CDN purges and emails

Affected components

• pypi.org - CDN
• pypi.org - Email

1. postmortemJun 28, 2022, 14:21 UTC

   ### Summary PyPI initially experienced delays in features which used a centralized task queue \(such as purging the CDN cache and sending emails\) due to a poorly performing task type. This was followed by an outage due to the combination of the steps taken to mitigate the queue issue and unrelated changes that affected PyPI’s performance, resulting in PyPI becoming unavailable via the web or for installations. ### Details On Thu, 23 Jun 2022 at 04:55 UTC, PyPI admins deployed a change which introduced a new periodic task designed to aggregate statistics about 2FA usage on PyPI. This task was designed to run every 5 minutes and sampled multiple counts of different characteristics of 2FA usage. This task used a database query that attempted to count a large number of rows, and had poor performance, causing high database utilization and a long processing time for the task. However, while long-running, the task was finishing and publishing metrics, and so PyPI admins were unaware of i

2. resolvedJun 27, 2022, 18:38 UTC

   This incident has been resolved.

3. monitoringJun 27, 2022, 16:56 UTC

   Backends are recovering

4. monitoringJun 27, 2022, 16:52 UTC

   A long running migration has caused our backends to become unhealthy, affecting all web traffic.

5. monitoringJun 27, 2022, 16:23 UTC

   We've disabled the specific tasks causing delays in the queue and are continuing to monitor the queue as it drains.

6. identifiedJun 27, 2022, 15:05 UTC

   We've identified specific jobs in our queue that are causing processing delays and are working to remove them.

7. investigatingJun 27, 2022, 14:20 UTC

   We are continuing to investigate this issue.

8. investigatingJun 27, 2022, 14:20 UTC

   A delay in our job queue is affecting CDN purges causing public pages to be stale and delays in email sending for notifications, email validation, password resets, etc.

What APIStatus.watch saw

Our 5-minute checks didn't record a change in PyPI's overall status around this incident. Smaller or regional incidents often leave a provider's overall status green.

How often does PyPI have incidents?

PyPI reported 3 incidents in the last 90 days, 1 of them major or critical. A typical incident lasted 6h 3m. See PyPI's uptime and incident history