After monitoring performance throughout the day, we feel confident that the incident from earlier today as well as the ongoing incident throughout last week have been resolved.
If you continue to notice issues with search results not showing up, user details not loading, or anything else that seems out of the ordinary, please reach out to us from within the app or via our contact page: https://fullstory.com/contact/.
Posted Jan 16, 2017 - 16:48 EST
Monitoring
We've restored the unresponsive portion of our search infrastructure and are continuing to monitor performance. Initial indicators suggest that full search performance has been restored for all customers.
Posted Jan 16, 2017 - 11:41 EST
Investigating
In regards to the issues from last week, we were able to restore full search functionality for all customers last Friday and performance has been stable throughout the weekend.
However, at around 9:50 AM EST, a portion of our search infrastructure became unresponsive, affecting a subset of our customers. Affected customers may notice errors in search and playback for sessions hosted in that area of infrastructure may fail. Our engineers are currently focused on restoring this bit of infrastructure. We'll keep the status blog updated with details as we have them.
Posted Jan 16, 2017 - 10:46 EST
Monitoring
We've updated the status to "Monitoring" to reflect that performance continues to be stable for all but a few of our customers. We're continuing to engineer, test, and deploy changes to our search infrastructure to make it more resilient. For the customers that continue to be affected, we'll reach out with status updates specific to your account.
Posted Jan 13, 2017 - 10:48 EST
Update
For most customers, FullStory was stable throughout the today. We're still seeing performance degradation for a small set of customers, and addressing these issues remains our top priority.
We have made changes in our production system to eliminate unnecessary query load to the search cluster and other configuration changes that should help with its performance. We've outlined multiple paths forward to continue to reduce load and are tackling them one at a time.
Posted Jan 12, 2017 - 17:44 EST
Update
Yesterday evening, we were able to isolate a particular area of our infrastructure which we believe is causing the bulk of the performance problems for the FullStory app. Determining the root cause as to why this area of the infrastructure is a performance bottleneck is our top priority today.
As of now, most customers should notice much better performance within the FullStory app. You may notice periodic performance degradation as we continue to test our infrastructure to isolate the root cause.
For the customers that continue to be affected by this incident, our huggers will be in touch with additional details and what to expect.
Posted Jan 12, 2017 - 09:59 EST
Update
We have made significant progress in identifying key cause and effect scenarios. However, we have yet to identify the root or combination of causes that can be cured in order to return search speeds to normal. As such, continuing to work on this open issue remains our engineer's top priority until service is returned to normal. We will also continue to post regular status updates as we have more information and until the issue is resolved
Posted Jan 11, 2017 - 17:52 EST
Update
We understand this incident is running quite longer than expected and wanted to provide an update. To reiterate the symptoms customers may be experiencing, users may receive errors while attempting to load session lists (e.g. "Failed to load user list.") and new sessions may take longer than normal to appear. If you're noticing problems with something loading in FullStory, it's likely due to the current incident.
Here's what's happening: starting Monday, we started noticing that our search infrastructure was responding much slower than normal to search queries. The symptoms presented in this outage have been different than what we've experienced in previous search outages, causing our normal remedies to not work as expected.
We're going to continue troubleshooting by diving deep into individual search queries to identify what may be causing the overall slowness. Once we identify the proper root cause, we'll be much closer to coming up with an appropriate solution.
To be sure, fixing this issue is the top priority for our engineering team.
Posted Jan 11, 2017 - 09:59 EST
Investigating
We are experiencing an elevated error rate for searches and session indexing. Users may receive errors while attempting to load session lists (e.g. "Failed to load user list.") and new sessions may take longer than normal to appear.