On October 31, 2024, our API service experienced a disruption that resulted in a significant number of dropped requests and errors. The incident lasted for approximately 1.5 hours and impacted all users of our public server API.

Customer Impact

Most public server API requests sent during the incident time frame (18:37:29 UTC to 20:12:39 UTC) were unsuccessful and should be re-submitted. We acknowledge the inconvenience and disruption this caused our customers and sincerely apologize for any impact on your operations.

Root Cause

The root cause of the incident was a decrease in the capacity of our API service due to an unsuccessful deployment. This decrease in capacity, combined with higher than usual request volume, led to the service being unable to handle the incoming traffic.

Resolution

Upon discovery of the issue, our on-call operations team was immediately notified and responded swiftly. The team's actions included re-executing the failed deployment to restore normal API service functionality. 

Process Changes and Prevention

Actions Taken:

• We re-executed the unsuccessful deployment.
• We permanently scaled up the capacity of our API service to handle the increased traffic.
• We are implementing additional monitoring and alerting to detect and prevent similar issues in the future.

Ongoing Improvements:

• We are reviewing our deployment processes to minimize the risk of similar incidents.
• We are investigating ways to improve the resilience of our API service to handle unexpected spikes in traffic.

‌

We deeply regret this incident and invite any Fullstory customer who was materially affected to contact support@fullstory.com. We stand by ready to fully address all of your concerns.