Beginning on August 12, 2017 at 8:00 PM EDT (August 13, 12:00 AM UTC), the SSL certificate for r.fullstory.com expired, which caused all recordings to FullStory to fail. The outage lasted for 14 hours, 56 minutes, until service was restored on August 13 at 10:56 AM EDT (14:56 UTC). This postmortem details the customer impact, the root cause of what happened, how we addressed the problem, and how we will prevent it from happening in the future.
Data collection was unavailable for the duration of the incident, meaning that no sessions will be available over the affected time period. Although developer console messages related to failed XHRs were present during the incident, we are unaware of any end-user visible impact. In rare situations in which a web application’s framework makes such network failures visible to the user, end users may have seen a related notification.
The SSL certificate for the domain r.fullstory.com expired, which led to 100% of recording requests to fail during the affected time. We knowingly allowed the certificate to expire because we had deprecated it, intending to supersede it with a newer wildcard certificate for *.fullstory.com. We incorrectly thought that the primary recording load balancer was using the newer *.fullstory.com certificate, as we had made the corresponding change correctly on our internal staging environment. In fact, the production environment continued to be configured to use the older, expiring certificate for r.fullstory.com. The root cause of this outage is that we lacked a process for deprecating certificates that ensured older certificates are no longer inadvertently in use.
In addition to the root cause, the incident was prolonged by an unexpected behavior in one of our traffic monitoring and alerting systems, preventing our engineers from becoming immediately aware of the customer impact of the expired certificate. Although we had longstanding probes to continuously check availability of the relevant HTTPS endpoint, the monitoring system did not enforce strict SSL checking and therefore reported 100% uptime despite the certificate having expired. Strict SSL checking behavior in the monitoring system would have caused these probes to fail, making immediately apparent that the deprecated r.fullstory.com certificate was still inadvertently in use, as described in the root cause analysis.
Once we became aware of the incident internally, we updated the configuration of the affected recording endpoints to use the newer wildcard certificate for *.fullstory.com, fully resolving the issue.
To prevent a recurrence of this incident in the future, we have identified two changes that we will make going forward. First, we will develop and adopt a more systematic approach to SSL certificate evolution to prevent inadvertent use of deprecated certificates. Second, we will initiate a recurring program to intentionally simulate this failure mode to ensure that the relevant monitoring alerts do indeed trigger as expected.
We deeply regret this incident and invite any FullStory customer who was materially affected to contact support@fullstory.com if you feel unsatisfied after this explanation so that we can fully address your concerns.