Between 2023-11-07 5:49 PM UTC and 2023-11-08 11:39 AM UTC an update to our capture service caused some web sessions to be initialized in a corrupted state that prevented capture data from being processed successfully. Replay and analytics features that rely on this session and event data were impacted; missing activity during this time period may impact Metrics, Funnels, Dashboards, and Conversions. Additionally, the impacted sessions are not available for session replay.
This postmortem details the customer impact, the root cause of what happened, how we addressed the problem, and how we will prevent similar incidents from happening in the future.
During the incident, customers using FullStory Relay or those with CSP policies that disallow access to the edge.fullstory.com CDN might have failed to capture entire sessions or have captured sessions that are missing a subset of their pages.
An update to our capture settings service caused some sessions to be initialized with inconsistent state and for which our backend data capture service was not able to process some pages for these sessions. In cases where the primary CDN-backed settings could not be accessed (see above notes on Relay and CSP), the client hits a fallback endpoint. This fallback endpoint did not contain accurate capture instructions for the client, leading to corruption of the local capture state.
Our internal monitoring alerted Engineering of corrupted sessions being processed, which resulted in the deployment being rolled back at 8:52 PM UTC. After observing lingering data capture errors after the rollback and testing possible impacts of the defect, a remediation process was developed to recover corrupted device identifiers for impacted users such that new capture sessions could be initiated.
We are committed to preventing this type of incident in the future. We’ve completed the following action items:
Here are additional steps we’re taking:
We deeply regret this incident and invite any FullStory customer who was materially affected to contact email@example.com. We stand by ready to fully address all of your concerns.