PyPI reported this critical-impact incident on its official status page on Mar 17, 2023, 21:04 UTC. It was resolved after 3h 56m.
Right now PyPI is operational. Live PyPI status →
## Summary At 20:36 UTC on 2023-03-17, PyPI’s email provider enforced a suspension against our account. After our remediation efforts and response, our account was restored at 00:36 UTC 2023-03-18. In response, the PyPI Administrators did the following: * Audited our email logs, including delivery and complaint status to determine the underlying factor that had led to our suspension. * Rolled out updates to our signup form to mitigate the behavior that was leading to complaint emails. * Notified our provider of our remediation efforts * Rolled out updates to our retry logic to report exceptions in tasks that were being swallowed by retry logic. The following was determined in the course of our response: * Our monitoring was blind to email failures due to retry logic swallowing the apparently transient errors, so response was delayed by nearly 15 minutes. * An account review was initiated 2021-09-04 by our provider due to a marginal complaint rate. This had been considered at the t
Our vendor has reinstated our sending ability after reviewing our remediation. Emails that were enqueued during the outage will send over the next 1-2 hours.
We have validated that failed email sends are queuing for delivery when our account suspension is lifted. For the moment we are holding off on mitigating by failing over to another provider. If we do not see resolution in 12 hours we will begin work to add another provider as a failover. Regardless we plan to reassess our resilience against email provider outages in the next work week.
We have submitted our response with remediation efforts to the provider and are awaiting their response/resolution. Detail: PyPI replaced reCaptcha on our signup form with a less robust form of automation defense in 2018 due to reports from users in China not being able to sign up. We have temporarily re-enabled reCaptcha in order to reduce inbox-bombing signups against PyPI. We will readdress ways to improve our defenses in the future, without needlessly denying access to users in China to PyPI.
We have identified the contributing factor that led to our complaint rate going above the acceptable threshold of our provider and are working to implement a fix that will satisfy their requirements.
We are working through the remediation process requested by our provider.
All emails sent by PyPI are currently not being sent. Our email provider has suspended our account. We are currently determining how to proceed.
Our 5-minute checks didn't record a change in PyPI's overall status around this incident. Smaller or regional incidents often leave a provider's overall status green.
PyPI reported 3 incidents in the last 90 days, 1 of them major or critical. A typical incident lasted 6h 3m. See PyPI's uptime and incident history