Incident Report - October 15, 2015

[ray_rauch]

Tealium iQ currently has two Tier 1 CDN providers and two Tier 2 CDN providers in rotation as part of our multi-CDN infrastructure.

Beginning on September 24, one of Tealium’s Tier 1 CDN providers deployed a new version of their compression software, which caused intermittent failures for Tealium customers. The nature of the failure resulted in an HTTP header and body mismatch. The root cause of the compression software failure was due to erroneous HTTP header information; while the Content-Encoding headers identified the files as compressed, the files themselves were not. This intermittently prevented proper file handling. We first detected this error in the European region and on September 24th we disabled that CDN provider in Europe.

Beginning on October 2nd, Tealium enacted service contingencies to rebalance the additional Tier 1 traffic to one of our Tier 2 CDN providers.

On October 7th we began detecting the error previously confined only to the European region in the US and Australia.  At that point, Tealium completely removed the offending Tier 1 CDN provider from our CDN fabric rotation.

Following the increase in volume on October 9th, our Tier 2 CDN provider experienced its own hardware failure which impacted a limited number of customers in the United States, particularly in the midwest. In response, Tealium rebalanced exclusively to a Tier 1 and Tier 2 provider.  Since October 7th, all global traffic has been running on two of Tealium’s most trusted CDN providers

We have started automated testing on our out-of-rotation Tier 1 CDN to get them back into the rotation. Once that is complete we will start testing of our second Tier 2 CDN to bring them back into the rotation.

Once we are satisfied with the action plan for restoring multiple Tier 1 CDN providers to service, we will complete our root cause analysis and provide a report on our action plan to improve monitoring, visibility and transparency for issues that could potentially affect customers in the future.