Below is a copy of the letter we emailed to the BigDoor Publisher Network this morning detailing the recent impact from the Amazon Web Services outage late last week. As a company transparency and feedback are crucial to the work we do here and as such we thought it was important to share this letter more broadly on our blog.

As a valued BigDoor publisher we wanted to take this opportunity to thank you for being patient with us through the latest issues stemming from the Amazon Web Services outage. We’re happy to report that as of Monday morning, all customer facing systems are back up and live and have been since Friday at roughly 1:30 p.m. PST – with a few exceptions that caused our API to intermittently go down.

To add a little more clarity as to what happened and how it may have impacted you, on Thursday, April 21st at 1:30 a.m. PST Amazon Web Services EC2 service failed. The BigDoor team immediately jumped in to action but over the course of the next twelve hours our more than 250 live publishers were affected by the AWS outage. We spent a better part of the day in constant contact with our publishers (via Twitter, emails and phone calls) trying to keep our level of transparency about this situation. Our team definitely felt great frustration with the lack of communication coming from Amazon Web Services – so much so that Keith Smith, our CEO, penned a guest post for the Seattle news site Geekwire. We’re not ones to point fingers and lay blame, in fact we often talk about how much we love AWS; we made the decision early on to host everything we do with Amazon Web Services. In the ten months since we launched the public beta of our free, self-serve gamification platform we have handled over one billion API calls and that simply would not have been possible with our small team and limited budget without AWS.

When it became clear that AWS was not communicating with the urgency we felt was necessary, our team quickly stepped into action so BigDoor’s API and publisher accounts were up faster than most systems that were affected by this nationwide outage nationwide outage. While it’s true we were able to fail-over to backup systems to a degree, over the last year there were additional fail-over capabilities we chose not to implement due to time and cost – this is the balance we’re always trying to make as a startup.

As of today, AWS has yet to recover our last critical system, a service that is key to managing JavaScript components. Since that service has yet to recover, dependent services on publisher.bigdoor.com are not functioning properly. AWS continues to communicate that we’re hours away from complete recovery, and at some point we may need to bite the bullet and set a new course for ourselves with regard to this server instance (at relatively high cost for us). For now, we’re still in a wait state. We’ve decided to keep allowing logins on the site because other critical functions are working as expected (changes to your underlying economies are being saved properly to the backend systems, for example). Rest assured all of your data is backed up in addition to the redundancies that AWS has provided, in multiple locations.

We’re hungry for feedback on this outage – it caused a massive shockwave for all customers hosted in this AWS Availability Zone, and certainly cost BigDoor a great deal of time and money, but it’s hard to gauge its effects across the board for our publishers without your candid feedback. If you haven’t already, please do let us know how severely this outage affected you, anything you think we should have been doing differently, and what you did to mitigate. We try hard to learn as much as we can from these things, and your feedback is essential in that process.

On a positive note, some of you may have heard our news a few weeks ago when we announced the launch of the BigDoor Engagement Economy, a way for sites to engage their users while monetizing their content. The economy is comprised of rewarding user actions with both status and a branded virtual currency as well as an innovative new engagement mechanic called Quests. BigDoor Quests allow a directed engagement experience for a publisher’s users that reward them with additional status and virtual currency. Quests are a new performance based ad format that is being sold on a Cost Per Quest (CPQ) basis. This is one of the BigDoor monetization moments that will make gamification a profit center and not a cost center for you the publisher. While this program is currently in private Beta we encourage publishers to contact us if you are interested in learning more.

Thank you again for your continued patience and support through all this.

Sincerely,

Jeff Malek
Co-Founder and CTO