Beginning on July 29, 2022 at 9:09 AM EDT (13:09 UTC), a defect released with an update to our fs.js web capture script caused all properties associated with any logged Custom Events to be dropped from the event payload. The outage lasted for 3 days and 13 minutes, until a fix was deployed on August 1, 2022 at 9:22 AM EDT (13:22 UTC). Note that sessions captured using FullStory for Mobile Apps were not impacted by this defect. Additionally, events captured using our Server Events API were not impacted by this defect
This postmortem details the customer impact, the root cause of what happened, how we addressed the problem, and how we will prevent it from happening in the future.
Data capture and collection was fully available for the duration of incident, meaning data capture supporting session replay, server events, other API endpoints, and standard FullStory events were unaffected. However, the Custom Events ingested during this period only contained the name of the Custom Event without any of the event properties associated with it. This means that sessions will still show the Custom Event was called, but the properties of those events will not be available or indexed. Capabilities impacted by this unavailability include: Segments, Metrics, Funnels and Conversions relying on this Custom Event data, during the outage window. Data Export files would also be affected for Custom Events and missing their event properties.
It’s possible that the loss of custom event properties during this time window will appear as an underrepresentation of conversion rates or revenue attribution for that time period. We would like to work with customers to uncover the extent of this impact and explore further actions we can take to help in these areas.
The root cause was the introduction of a defect in the code to our FullStory script (fs.js) that was incompatible with the current version of the FullStory Data Capture Snippet used by our web capture clients. Our systems include a rigorous set of automated integration testing of the FullStory script in place for all our FullStory Client APIs during deployments; however, in a parallel effort to upgrade our FullStory snippet we updated our automation suite to use the upcoming snippet and lost coverage for the previous snippet version. The code was thought to be safe because it had passed the checks and tests within our Continuous Integration (CI) pipeline and had been successfully deployed in our staging environment for 14 days without a reported issue.
Once we identified the root cause — incompatibility with the current FullStory snippet and the FullStory script (fs.js) — we rolled out a fix for the FullStory script to begin data collection for Custom Events. Sessions started an hour after the 9:22 AM EDT (13:22 UTC) deployment should be working as expected.
To prevent a recurrence of this incident in the future, we’ve completed the following action items:
Rolled back FullStory script and snippet release to the prior working version.
Added manual QA verification steps of our FullStory Client APIs before each release.
We also have work slated on our immediate roadmap to:
We will report on the progress of the above action items on a regular basis.
We deeply regret this incident and invite any FullStory customer who was materially affected to contact email@example.com if you feel unsatisfied after this explanation or if we can provide additional processing to your data to help with the metrics reporting over the impact period. We stand by ready to fully address all of your concerns.