Summary:

Some sites using FS.identify experienced exceptions when calling the API due to a namespacing problem in the JS API. Whether or not the exception was thrown depended on a timing issue related to the loading of the FullStory script. The problem was introduced at 12:00 pm EDT on June 1, 2016. We had a fix deployed and in production at approximately 12:54 pm. It took until about 1:15pm for all the content distribution caches to clear. Once all caches were cleared, this issue was resolved for all new page visits.

Post Mortem:

At approximately 12:00 pm EDT on June 1, 2016, we rolled out an update to our recording script bootstrap that enabled customer configuration of the namespace that FullStory uses for its JavaScript API. This feature allows sites whose script uses the identifier window.FS – which can happen even inadvertently due to JS obfuscation – to still be able to use FullStory and FullStory's JavaScript API on a (non-colliding) namespace of their choosing.

A bug in the initialization logic of this change caused some of the FullStory JavaScript API (FS.identify, FS.setUserVars and FS.clearUserCookie) to be absent from the FullStory namespace after the main recording script loaded. Calls to these APIs that happened prior to the loading of the main FullStory recording script were safe and would not throw exceptions and thus did not cause any issues. But calls that happened after the main FullStory recording script loaded, threw exceptions due to the missing APIs.

FullStory already has safeguards in place to make it impossible for the JS APIs installed by our main recording script to break our customer's code. All API entry points into this code are guarded by try/catch blocks internally. However, the problem in this instance lived within our script bootstrap, which rarely changes, and controls the loading of our script and the installation of the APIs themselves, which was outside these safeguards.

We became aware of this issue at approximately 12:20 pm EDT and had a fix live in production at 12:55 pm EDT. After content distribution caches cleared, this issue was resolved for all new page visits as of around 1:15 pm EDT.

Introducing a bug that affects our customer's site is our worst case operational failure and we consider it completely inexcusable. We will continue to refine our testing and deployment practices to ensure this failure mode can never happen again. Thank you for your patience and know that we are making every effort to eliminate all such problems in the future.

-FullStory Engineering