Upstash outage: Performance degradation on QStash

Affected components

• EU-CENTRAL-1

1. postmortemNov 13, 2024, 20:38 UTC

   **Product:** QStash **Impact:** Degraded performance, delayed processing of events, and duplicate event deliveries for some customers ## Incident Summary QStash experienced an incident marked by a sudden and extreme load on our servers. This caused a degradation in performance, with extremely high latency for event processing for all users. We also noticed some of the events being delivered multiple times to some of the users. To mitigate the high load, we have increased the capacity as our initial response while investigation proceeds. Eventually, fixes for the issues are confirmed with an issue reproducer and deployed to production. ## Root Cause Analysis In a certain type of usage, failure handling of [failureFunction](https://upstash.com/docs/workflow/basics/serve#failurefunction) can cause recursive calls which causes a leak in the queue of the tasks, causing a severe load on the QStash servers. This also triggered an edge case which caused some of the events to be delivered

2. resolvedNov 13, 2024, 17:27 UTC

   We will be sharing a postmortem about the incident soon.

3. identifiedNov 13, 2024, 15:38 UTC

   The issue has been identified and a fix is being implemented.

4. investigatingNov 13, 2024, 14:14 UTC

   We are currently investigating this issue.

What APIStatus.watch saw

Our 5-minute checks didn't record a change in Upstash's overall status around this incident. Smaller or regional incidents often leave a provider's overall status green.

How often does Upstash have incidents?

Upstash reported 3 incidents in the last 90 days, 1 of them major or critical. See Upstash's uptime and incident history