Minor impactResolvedDevOps

PyPI incident: Newly uploaded files are returning 403 from files.pythonhosted.org

PyPI reported this minor-impact incident on its official status page on Jun 2, 2022, 22:20 UTC. It was resolved after 13h 47m.

Right now PyPI is operational. Live PyPI status →

🔔 Get alerted when PyPI has issuesFree · email alerts · no credit card
Jun 2, 2022, 22:20 UTC
started
Jun 3, 2022, 12:07 UTC
resolved
13h 47m
duration
Minor
impact

Affected components

  • files.pythonhosted.org - Files
  1. postmortemJun 3, 2022, 12:07 UTC

    ## Summary The cloud infrastructure that runs PyPI is managed via an automation tool which allows for configuration changes to be version controlled, reviewed, and deployed automatically. This tool includes multiple backends for state sharing and locking to facilitate collaboration between multiple contributors. While migrating the remote state backend and locking features to a more robust hosted platform an error was made when setting the credentials that grant our CDN access to the object storage where uploaded files to PyPI are hosted. The result was that newly uploaded files as well as files that had expired from our CDNs cache were unavailable. The outage began 2022-06-022 20:28 UTC and was initially mitigated 2022-06-02 22:38 UTC by rolling back the configuration to a previous version. The configuration issue in the hosted platform was confirmed and corrected 2022-06-02 12:00 UTC and a corrected new version of the CDN configuration was deployed. Future deploys will include this

  2. resolvedJun 3, 2022, 12:07 UTC

    We have resolved this incident, identified the cause of the breakage, and completed ensuring that it will not regress. An incident report will be provided shortly.

  3. monitoringJun 2, 2022, 22:40 UTC

    A fix has been implemented and we are monitoring the results.

  4. identifiedJun 2, 2022, 22:40 UTC

    The issue has been identified and a fix is being implemented.

  5. investigatingJun 2, 2022, 22:20 UTC

    We are currently investigating this issue.

What APIStatus.watch saw

Our 5-minute checks didn't record a change in PyPI's overall status around this incident. Smaller or regional incidents often leave a provider's overall status green.

How often does PyPI have incidents?

PyPI reported 3 incidents in the last 90 days, 1 of them major or critical. A typical incident lasted 6h 3m. See PyPI's uptime and incident history

PyPI incident on Jun 2, 2022: Newly uploaded files are returning 403 from files.pythonhosted.org · APIStatus.watch