Beginning on December 13, 2017 at 6:38 PM EST (23:38 UTC), a bad push to our recorder infrastructure ultimately led to the majority of recordings to FullStory to fail. The outage lasted for 57 minutes, until service was restored at 7:35 PM EST (December 14, 0:35 UTC). This postmortem details the customer impact, the root cause of what happened, how we addressed the problem, and how we will prevent it from happening in the future.
Data collection was only partially available for the duration of the incident, meaning that most customer sessions will be affected. We estimate that approximately 75% of recording bundles were dropped during this time. This means that sessions will be completely unavailable or that some sessions may have noticeable “holes” in the playback experience for any data that would have been dropped during the outage.
Although developer console messages related to failed XHRs were present during the incident, we are unaware of any end-user visible impact. In rare situations in which a web application framework makes such network failures visible to the user, end users may have seen a related notification.
The root cause was a push of bad code to our recorder infrastructure. The code was thought to be safe because it had been deployed in our staging environment for over 9 hours. But the shape and volume of our staging traffic—being sufficiently different from production—meant that the problem did not manifest in the same way.
We were alerted of the problems quickly and reverted to a known good version of the code. However, our auto-scaling infrastructure had shrunk the size of our recording cluster because one symptom of the bad code was deadlock, causing recorder processes to use much less CPU. A bad interaction between the way the cluster management software determines health of a process and our autoscaler caused the incident to persist even after we pushed a known good version of the code.
Once we identified the root cause—recorder replicas weren’t being generated by our autoscaling infrastructure—we manually set the minimum number of replicas to a level that would support all FullStory traffic. The incident was fully resolved within 3 minutes of making this change.
To prevent a recurrence of this incident in the future, we’re looking to add an intermediate step to our deployment to catch failures that may manifest in production but not in staging.
We deeply regret this incident and invite any FullStory customer who was materially affected to contact support@fullstory.com if you feel unsatisfied after this explanation so that we can fully address your concerns.