2026/02/24 Data transport to data warehouse significantly delayed for some customers
This postmortem details the customer impact, the root cause of what happened, how we addressed the problem, and how we will prevent it from happening in the future.
Some customers using Fullstory Anywhere: Warehouse may have observed a significant decrease in the number of events being transported to their data warehouse between Feb 24, 2026 to Mar 13, 2026. The impact here is limited to object storage destinations such as Amazon S3, Azure Blob Storage, and Google Cloud Storage.
The Fullstory platform has two internal pipelines that generate event data and notify downstream systems. There is a streaming pipeline for real time event processing, and a batch pipeline for everything else. Object storage data destinations receive notifications from the latter whenever it processes session and page entities that are deemed inactive i.e. no longer receiving event data.
On Feb 24, 2026 the team made a change to the streaming pipeline to track additional metadata about page entities that were being processed. Unfortunately this change had an unintended effect on the batch pipeline which caused notifications to not be fired upon successful processing for a subset of events. Thus event data that should have been transported to object storage to instead remain buffered indefinitely.
On Mar 11, 2026 the team investigated the issue and pinpointed it to a code change that was deployed on Feb 24, 2026. A fix was deployed on the same day as the issue was found. From there events from newly processed page entities began flowing to affected customer’s data warehouses again.
Once the fix was in place the team focused on impact analysis and remediation. A backfill process was kicked off on Mar 12, 2026 to transport event data that did not originally make it to affected customer’s data warehouses. As of Mar 13, 2026 the backfill is complete and any gaps in event data have been filled.
No further action is required from those impacted.
Our team has completed a thorough review of the incident, fixed the relevant bug, and implemented a backfill process for remediation here. We have also identified several areas of improvement with our metrics and monitoring, as well as system changes to prevent this kind of unintended cross-pipeline impact.
We deeply regret this incident and invite any Fullstory customer who was materially affected to contact support@fullstory.com. We stand by ready to fully address all of your concerns.