Beginning on September 27, 2017 at 3:58 PM EDT (August 27, 15:38 AM UTC), a data storage system used to buffer recording data became unreachable causing most recordings to FullStory to fail. The outage lasted for 2 hours, 28 minutes, until service was restored on September 27 at 6:26 PM EDT (18:26 UTC). This postmortem details the customer impact, the root cause of what happened, and how we will prevent it from happening in the future.
Data collection was largely unavailable for the duration of the incident, meaning that no sessions will be available over the affected time period. There was no end-user visible impact to this incident.
A secondary effect of this incident was intermittent playback errors for Go Live sessions, as playback for live sessions relies on the buffered recording data.
A network switch failure in our data storage provider caused the storage system, which is used to buffer session recording data for processing, caused writes to our storage layer to fail. When those writes failed the recording servers informed the Fullstory client code to retry the request, but due to the duration of the data storage unavailability, clients exceeded the point where they will retry and recording data was lost.
The key action to prevent recurrence of this incident will be to implement an additional buffer storage system to retain recording data even when the primary storage system is unavailable.
We deeply regret this incident and invite any FullStory customer who was materially affected to contact support@fullstory.com if you feel unsatisfied after this explanation so that we can fully address your concerns.