by Marissa Mayer, Yahoo CEO
This has been a very frustrating week for our users and we are very sorry.
For many of us, Yahoo Mail is a lifeline to our friends, family members and customers. This week, we experienced a major outage that not only interrupted that connection, but caused many of you a massive inconvenience — that’s unacceptable and it’s something we’re taking very seriously. Unfortunately, the outage was much more complex than it seemed at first, which is why it’s taking us several days to resolve the compounding issues.
So, what happened?
On Monday, December 9th at 10:27 p.m. PT, our network operating center alerted the Mail engineering team to a specific hardware outage in one of our storage systems serving 1% of our users. The Mail team immediately started working with the storage engineers to restore access and move to our back-up systems, estimating that full recovery would be complete by 1:30 p.m. PT on Tuesday.
However, the problem was a particularly rare one, and the resolution for the affected accounts was nuanced since different users were impacted in different ways. Some of the affected users were unable to access their accounts, instead seeing an outdated “scheduled maintenance” page which was a confusing and incorrect message (this has since been corrected and updated). Further, messages sent to those accounts during this time were not delivered, but held in a queue.
Over the remainder of the week, we worked around the clock to restore access and all messages to inboxes. This has included restoring IMAP access for people using other email programs like Outlook or Apple Mail to access their Yahoo Mail.
As of this afternoon, we’ve restored access to almost everyone and delivered the backlog of messages. We will continue to work on rolling out IMAP access and to fully restore inbox state (for example, which folders messages were placed in, which messages were starred, etc). This process differs for each user and as restoration continues, we’re committing to communicating directly with you on progress on an individual basis.
Above all else, we’re going to be working hard on improvements to prevent issues like this in the future. While our overall uptime is well above 99.9%, even accounting for this incident, we really let you down this week.
We can, and we will, do better in the future.