This incident has been completely resolved. The backfill to restore data for the last remaining customers has completed, so all missing sessions due to this incident should now appear in FullStory.
If you believe this incident affected you and you believe there are still sessions missing, please reach out to us at support@fullstory.com and reference this incident.
Posted Dec 21, 2017 - 11:21 EST
Update
Regular visitors to our status blog (wait—we have those?) may have noticed that this incident is still outstanding. We want to shed additional light on what happened and what we’re doing to bring this to resolution.
Last week on December 5, two of the machines in our search infrastructure failed, causing those machines to stop returning search results for the subset of customers who had data on those minds. Keep in mind that this only affected a subset of our customers—most customers would not have noticed anything awry.
The very first thing we did was identify active, smaller customers for whom all of their search data was on one of the two machines that failed. For those customers, we immediately moved their search data onto new machines and re-indexed their sessions. All of those customers’s sessions are now (and have been since December 5) fully indexed and should be experiencing no problems whatsoever.
The remaining challenge is for customers who had some search data on the failed machines. For these customers, some sessions will be missing from search results. We’re actively working on restoring this data without interrupting the ability to search for non-impacted sessions. We expect to have this fixed in the coming days.
In the meantime, if you believe you might be affected by this incident and are having trouble finding sessions in your FullStory account, please reach out to us at support@fullstory.com and reference this incident.
Posted Dec 12, 2017 - 15:57 EST
Identified
At this time, new and recent sessions should be appearing normally. Some older sessions are still missing from the search index which we are working to restore.
Posted Dec 05, 2017 - 10:36 EST
Investigating
A small percentage of search requests are failing, and successful searches may be missing some sessions from search results. We are currently investigating causes and remediation options.