Minor impactResolvedDatabases

Upstash incident: Regional AWS eu-west-1 Cluster Performance Degradation Issue

Upstash reported this minor-impact incident on its official status page on Feb 25, 2025, 14:03 UTC. It was resolved after 7h 20m.

Right now Upstash is operational. Live Upstash status →

🔔 Get alerted when Upstash has issuesFree · email alerts · no credit card
Feb 25, 2025, 14:03 UTC
started
Feb 25, 2025, 21:23 UTC
resolved
7h 20m
duration
Minor
impact

Affected components

  • AWS EU-WEST-1
  1. postmortemFeb 26, 2025, 12:29 UTC

    ### Incident Summary During a maintenance update to the regional Upstash Redis databases in AWS eu-west-1, several databases hosted in that region has unnecessarily triggered a full synchronisation between their primary and backup replicas. ### Root Cause A full synchronisation is the invalidation of the whole data in the target replica and starting a fresh re-population from the source replica. Under normal circumstances, full synchronisation is required only in cases where the data integrity is lost in one of the replicas, which was not the case here. ### Impact This incident impacted the performance of regional databases on AWS eu-west-1 only. Full synchronisation has caused a very high CPU load and caused a performance degradation on some of the databases that has a replica in this region. Moreover, our system throttles the databases that are going through this operation to allocate more CPU to the synchronisation to finish it sooner. No data or consistency has been lost.

  2. resolvedFeb 25, 2025, 21:23 UTC

    This incident has been resolved.

  3. monitoringFeb 25, 2025, 20:54 UTC

    A fix has been implemented and we are monitoring the results.

  4. identifiedFeb 25, 2025, 14:03 UTC

    Regional AWS eu-west-1 cluster is experiencing performance degradation, and we are adding more resources to the cluster.

What APIStatus.watch saw

Our 5-minute checks didn't record a change in Upstash's overall status around this incident. Smaller or regional incidents often leave a provider's overall status green.

How often does Upstash have incidents?

Upstash reported 3 incidents in the last 90 days, 1 of them major or critical. See Upstash's uptime and incident history

Upstash incident on Feb 25, 2025: Regional AWS eu-west-1 Cluster Performance Degradation Issue · APIStatus.watch