Minor impactResolvedDatabases

Upstash incident: QStash: Degraded performance in request processing and event logs

Upstash reported this minor-impact incident on its official status page on Apr 2, 2025, 07:15 UTC. It was resolved after 13h 57m.

Right now Upstash is operational. Live Upstash status →

🔔 Get alerted when Upstash has issuesFree · email alerts · no credit card
Apr 2, 2025, 07:15 UTC
started
Apr 2, 2025, 21:12 UTC
resolved
13h 57m
duration
Minor
impact

Affected components

  • EU-CENTRAL-1
  1. postmortemApr 3, 2025, 15:49 UTC

    ### Product: QStash ### Incident Summary Due to high load, the volume of QStash event logs reached to a point which caused latency in the underlying data store operations. Event log creation was slowed down and lead to performance degradation in QStash request processing. In order to resolve the performance degradation in QStash requests, event logging module was turned off temporarily. After deploying a hot fix and configuration changes, we eventually turned on event logging and system went back to stable state again. ### Root Cause At 07:15 UTC we received alerts on the performance degradation and started the investigation. We discovered long running queries for synching event logs from main QStash servers to QStash event server. In order to resolve performance degradation, we turned off event logging functionality as an immediate action. This action turned the performance back to normal levels for QStash requests but left event log processing disabled. We deploy

  2. resolvedApr 2, 2025, 21:12 UTC

    QStash service and Event logs are fully functional without any remaning issues.

  3. monitoringApr 2, 2025, 20:20 UTC

    Monitoring: Main QStash service is back to normal. Event logs service is back online but events will be lagging a few mins.

  4. monitoringApr 2, 2025, 17:33 UTC

    Main QStash service is back to normal. Event logs are still temporarily unavailable.

  5. investigatingApr 2, 2025, 16:33 UTC

    We are continuing to investigate this issue.

  6. investigatingApr 2, 2025, 16:29 UTC

    We are continuing to investigate this issue.

  7. investigatingApr 2, 2025, 16:21 UTC

    We are currently investigating this issue.

  8. monitoringApr 2, 2025, 08:15 UTC

    A fix has been implemented and we are monitoring the results.

  9. investigatingApr 2, 2025, 07:15 UTC

    We are currently investigating this issue.

What APIStatus.watch saw

Our 5-minute checks didn't record a change in Upstash's overall status around this incident. Smaller or regional incidents often leave a provider's overall status green.

How often does Upstash have incidents?

Upstash reported 3 incidents in the last 90 days, 1 of them major or critical. See Upstash's uptime and incident history

Upstash incident on Apr 2, 2025: QStash: Degraded performance in request processing and event logs · APIStatus.watch