All search servers are up and operating normally. We are continuing to monitor the situation as the backlog of sessions is processed. There will be some delay before recent sessions show up in search results.
Posted Jun 28, 2017 - 13:56 EDT
Update
We're continuing to work to bring up our search infrastructure. Some search nodes have been restored, but not all. As such, some customers may be able to log in and search for sessions within FullStory, but for others FullStory will continue to be down. We'll continue to post updates and let you know when service has been fully restored for all customers.
Posted Jun 28, 2017 - 12:24 EDT
Update
Quick update. We’ve finished the manual processing of events and we’re currently working to bring up the part of our search infrastructure that processes events automatically so we don’t have to do it manually. This part of the infrastructure is required in order to restore functionality to the FullStory app.
We’ll continue to post updates, as available, and confirm when processing has returned to expected performance.
Posted Jun 28, 2017 - 11:36 EDT
Identified
While our engineers our working to bring up the FullStory app, we wanted to provide some additional insight about what's going on. Our search infrastructure experienced a sudden increase in load, which caused our infrastructure to get behind in processing new events coming in. The solution is to manually process the events to give it a chance to catch up. We’re currently doing that and are about 60% done. We'll update again here if/when anything changes.
Posted Jun 28, 2017 - 11:01 EDT
Investigating
We are experiencing an elevated error rate for searches and session indexing. Users may receive errors while attempting to load session lists (e.g. "Failed to load user list.") and new sessions may take longer than normal to appear.