Critical impactResolvedMedia

Imgix outage: Elevated rendering errors

Imgix reported this critical-impact incident on its official status page on Jun 17, 2024, 00:21 UTC. It was resolved after 58 min.

Right now Imgix is operational. Live Imgix status →

🔔 Get alerted when Imgix has issuesFree · email alerts · no credit card
Jun 17, 2024, 00:21 UTC
started
Jun 17, 2024, 01:19 UTC
resolved
58 min
duration
Critical
impact

Affected components

  • Rendering Infrastructure
  1. postmortemJun 21, 2024, 23:54 UTC

    ## What happened? On June 17, 2024, at 00:00 UTC, imgix experienced an extreme spike in requests to our render stack. This unexpected surge caused a failure in our auto-scaling infrastructure, leading to an inability to manage all incoming traffic effectively. A fix was implemented at 00:38 UTC, and the issue was resolved by 01:06 UTC. ## How were customers impacted? Between 00:00 and 01:06 UTC, customers may have experienced failures when requesting new renders. However, previously cached assets served successfully during this time. ## What went wrong during the incident? The incident was triggered by a significant increase in requests, which our automated systems did not properly handle. Although the system started to auto-scale as expected, the unexpected surge caused issues with the health checks used for auto-scaling. The combination of extra traffic and health check failure led to an inability to render new images that required manual intervention to resolve. ## What will i

  2. resolvedJun 17, 2024, 01:19 UTC

    This incident has been resolved.

  3. monitoringJun 17, 2024, 01:06 UTC

    A fix has been implemented and error rates are returning to normal. We are continuing to monitor the service.

  4. identifiedJun 17, 2024, 00:51 UTC

    We are continuing to work on a fix for this issue.

  5. identifiedJun 17, 2024, 00:50 UTC

    The issue has been identified and our engineering team is developing a fix.

  6. investigatingJun 17, 2024, 00:21 UTC

    We are currently investigating elevated render error rates for uncached derivative images. We will update once when we obtain more information. Previously cached derivatives are not impacted.

What APIStatus.watch saw

Our 5-minute checks didn't record a change in Imgix's overall status around this incident. Smaller or regional incidents often leave a provider's overall status green.

How often does Imgix have incidents?

Imgix reported 3 incidents in the last 90 days, 2 of them major or critical. A typical incident lasted 51 min. See Imgix's uptime and incident history

Imgix outage on Jun 17, 2024: Elevated rendering errors · APIStatus.watch