Highlighted

Incident Report (February 28, 2017) - AWS S3 Outage

Employee Emeritus

This morning Tealium's services were affected by the AWS S3 Outage.

Summary: AWS S3 Outage - TMS and UDH Service Disruption, February 28th, 2017
Description: Like many Internet based service providers, the ongoing AWS S3 outage is causing reverberating effects across the internet. Because several Tealium products and services are dependent on AWS S3, our customers may experience intermittent issues while using some Tealium services.  Additionally, connectors targeting vendors who are dependent on S3 may also be impacted.
Status: Closed 
Priority: P1
Region: US East
Affected Services: Data Layer Enrichment Upload API, EventStore, EventDB, AudienceDB, AudienceStore
Time Reported: 9:45am PST, February 28th, 2017
Time Resolved: 3:10pm PST, February 28th, 2017
Updates:

10:22am PST: AWS Reports they’ve identified an issue with it’s Simple Storage Service and are working to restore service

10:33am PST: AWS Reports troubleshooting efforts are ongoing and to continue to expect high error rates

Screen Shot 2017-02-28 at 11.23.16 AM.png

11:35am PST: AWS Reports they have discovered root cause and working on remediating the issue to restore service

Screen Shot 2017-02-28 at 12.02.43 PM.png

2:11pm PST: The latest update from AWS on the S3 outage is as follows:

Screen Shot 2017-02-28 at 2.04.27 PM.png

Tealium Engineers continue to monitor AWS S3 status for full service restoration. Once S3 services are fully restored, Engineers will start verify proper functionality of Tealium services.


3:10pm PST:
 
The latest and final update from AWS is they have fully restored service to S3 (and other affected components).

aws-s3-outage-resolved.png

Tealium Engineers have verified all Tealium services are fully operational.
 
There will be no further updates on this incident.
Note: As a result of AWS' S3 outage today, EventStore, EventDB, AudienceStore and AudienceDB suffered data loss during the impact window, between ~10am-2:30pm PST.

 

Remember to give me a kudo if you like my post! Accepting my post as a solution is even better! Also remember that search is your friend.