FS.event properties dropped
Incident Report for FullStory
Postmortem

2022/07/29 Capture FS.event Outage

Beginning on July 29, 2022 at 9:09 AM EDT (13:09 UTC), a defect released with an update to our fs.js web capture script caused all properties associated with any logged Custom Events to be dropped from the event payload. The outage lasted for 3 days and 13 minutes, until a fix was deployed on August 1, 2022 at 9:22 AM EDT (13:22 UTC). Note that sessions captured using FullStory for Mobile Apps were not impacted by this defect. Additionally, events captured using our Server Events API were not impacted by this defect

This postmortem details the customer impact, the root cause of what happened, how we addressed the problem, and how we will prevent it from happening in the future.

Customer Impact

Data capture and collection was fully available for the duration of incident, meaning data capture supporting session replay, server events, other API endpoints, and standard FullStory events were unaffected. However, the Custom Events ingested during this period only contained the name of the Custom Event without any of the event properties associated with it. This means that sessions will still show the Custom Event was called, but the properties of those events will not be available or indexed. Capabilities impacted by this unavailability include: Segments, Metrics, Funnels and Conversions relying on this Custom Event data, during the outage window. Data Export files would also be affected for Custom Events and missing their event properties.

It’s possible that the loss of custom event properties during this time window will appear as an underrepresentation of conversion rates or revenue attribution for that time period. We would like to work with customers to uncover the extent of this impact and explore further actions we can take to help in these areas.

Root Cause

The root cause was the introduction of a defect in the code to our FullStory script (fs.js) that was incompatible with the current version of the FullStory Data Capture Snippet used by our web capture clients. Our systems include a rigorous set of automated integration testing of the FullStory script in place for all our FullStory Client APIs during deployments; however, in a parallel effort to upgrade our FullStory snippet we updated our automation suite to use the upcoming snippet and lost coverage for the previous snippet version. The code was thought to be safe because it had passed the checks and tests within our Continuous Integration (CI) pipeline and had been successfully deployed in our staging environment for 14 days without a reported issue.

Resolution

Once we identified the root cause — incompatibility with the current FullStory snippet and the FullStory script (fs.js) — we rolled out a fix for the FullStory script to begin data collection for Custom Events. Sessions started an hour after the 9:22 AM EDT (13:22 UTC) deployment should be working as expected.

Process Changes and Prevention

To prevent a recurrence of this incident in the future, we’ve completed the following action items:

  • Rolled back FullStory script and snippet release to the prior working version.

  • Added manual QA verification steps of our FullStory Client APIs before each release.

We also have work slated on our immediate roadmap to:

  • Expand our Continuous Integration pipeline with more automated testing against all major versions of the FullStory snippet and FullStory script releases (and release candidates) to catch incompatibilities in development
  • Build monitoring dashboards to better and more quickly observe issues in Client APIs as part of Release Playbook
  • Build internal alerting against our monitoring to more quickly take remediation action 
  • Explore possibility for enterprise customers to subscribe to long-term version options of the FullStory script and snippet

We will report on the progress of the above action items on a regular basis.

We deeply regret this incident and invite any FullStory customer who was materially affected to contact support@fullstory.com if you feel unsatisfied after this explanation or if we can provide additional processing to your data to help with the metrics reporting over the impact period. We stand by ready to fully address all of your concerns.

Posted Aug 02, 2022 - 17:15 EDT

Resolved
On Friday, July 29th, at 9:09am EDT, a bug was introduced via a release to fs.js that caused FS.event options to be dropped. This issue was patched on August 1st, at 9:22am EDT, and the issue was resolved. Any FS.event call with eventProperties (as defined by our API documentation, https://developer.fullstory.com/custom-events) were dropped. Events generated in affected time period will not have the relevant data attached to the event.
Posted Jul 29, 2022 - 09:00 EDT