Memo: ‘Gawker tech team didn’t adequately secure our platform’

Romenesko Memos
“On several fronts — technically, as well as customer support and communication — we found ourselves unprepared to handle this eventuality,” says Gawker chief technology officer Thomas Plunkett’s memo to staff. “The tech team should have been better prepared, committed more time to perform thorough audits, and grown our team’s technical expertise to meet our specific business needs. As a result of not having done these things, we have not adhered to standards expected of us, and our response was inadequate.”

From: Thomas Plunkett
Subject: The Gawker Media security breach — status and moving forward
To: [Gawker staff]
Date: Friday, December 17, 2010, 4:43 PM

Everyone -

As you know, this has been the Gawker tech team’s most difficult week ever. This note has been too long coming, but the following is meant to communicate several things: what happened, our current activities, and our plans for moving forward. I suggest you read all of this as I am making several recommendations below, and we are implementing some changes that will affect all of you.

What Happened
Gawker Media servers and some company email accounts were compromised by hackers at some time during the last few weeks; the compromise was made public to us (and everyone else) this past weekend. In recent weeks, intruders were able to gain access to our web servers by exploiting a vulnerability in our source code, allowing them to gain access to user data and passwords. With this information, they were able to gain access to the editor wiki, some Gawker Media email accounts, and other external resources.

It is clear that the Gawker tech team did not adequately secure our platform from an attack of this nature. We were also not prepared to respond when it was necessary. These things can be attributed to several factors.

First, we never planned for such an event, and therefore had no systems, or processes in place to adequately respond. Our focus as a team (and company) has been on moving forward. This put up blinders on several fronts. As a result, numerous wrong decisions were made by me this past weekend in responding to the security breach.

Further, attention to completed work is every bit as important as attention to upcoming work. Our development efforts have been focused on new product while committing relatively little time to reviewing past work. This is often a fatal mistake in software development and was central to this vulnerability.

Finally, we have not only seen tremendous growth as a company, we have never been afraid to take an unpopular or controversial stance with regard to individuals or organizations. Let’s face it: we draw the ire of many. This creates a unique set of demands to meet rapid growth as well as threats that often specifically target us. We did not establish standards and practices to handle growth and the fact that we have a target on our back.

On several fronts — technically, as well as customer support and communication — we found ourselves unprepared to handle this eventuality. The tech team should have been better prepared, committed more time to perform thorough audits, and grown our team’s technical expertise to meet our specific business needs. As a result of not having done these things, we have not adhered to standards expected of us, and our response was inadequate. The remedy to this situation will not be immediate, but it will be swift as possible.

Current Activity: Regaining Control
The tech team have moved our operation to the third floor of the Gawker Media office in order to focus on the work that needs to be done. We are currently in the process of performing a complete review of what happened with an independent security firm.

Here’s what we’ve done so far to regain control:

We have been able to establish a fairly complete timeline of intrusion activity, and have identified compromised assets within Gawker. We have re-established control of compromised systems including our Google Apps accounts. As a result, you will have to reconfigure your Google Apps access (more on this below).

In addition, we have addressed all known vulnerabilities and will continue auditing our system for security flaws, and we have made appropriate changes to administrative accounts to our web and application infrastructure. There are many people reviewing our code base, and because of this, we will also reach out to members of the technical community to harness their expertise. This process will continue as we move to an entirely new, hardened web infrastructure.

We have introduced a help desk to address commenter concerns related to the breach. This will continue to exist as long as it is needed. Scott, Greg, Jeremy, Nick and a host of interns, and many of you, have been active in the threads, and communicating as much as possible as we work through this event.

Moving Forward
We’ve learned many lessons from this experience, both as a tech team, as a company, and as individuals. If there’s one lesson nearly all of us learned, it’s that we can and must be smarter with passwords. Lifehacker is a great resource for password advice (and there are many others). I suggest you start here: http://lifehacker.com/184773/geek-to-live–choose-and-remember-great-passwords.

Efffective immediately, we have enabled SSL, a more secure method of communicating over the internet, for all users with Gawker Media accounts on Google Apps (this does not affect your personal Gmail). Those of you not using web-based Gmail will have to reconfigure your clients (this includes any desktop mail client as well as other devices). The attached document provides instructions to make this easier, and includes information to configure different devices including iPhone, Android and Blackberry phones.

Also effective immediately: If you require access to sensitive materials (legal, financial, or accounting documents) on Google Docs, you must have two-factor authentication setup on your account. No documents will be shared with personal Gmail accounts. We are also strongly encouraging all staff to setup two-factor authorization even if you do not require access to sensitive material.

We will enforce a policy that sensitive information not be posted to the editor wiki. This policy will also apply to chat communications (e.g., Campfire, AIM).

On all of our sites, we will be introducing several new features to our commenting system to acknowledge the reality that we have lost the commenters’ trust and don’t deserve it back. We should not be in the business of collecting and storing personal information, and our objective is to migrate our platform away from any personal data dependencies (like email & password). We will push further integration of external account verification sources using OAuth (like Facebook, Twitter, and Google) for those that want to use them, and we’ll also be introducing disposable accounts. Disposable accounts are similar to the service a pre-paid phone offers to drug dealers (a disposable, untraceable communication device). Commenters seeking anonymity will be able to do so confident that when necessary they can simply toss out the account and there will be no connection to the individual. They will work like this:
- no password will be stored
- no email will be stored
- account can be used as long as you have the key code; lose or delete it, the account is abandoned.

In addition, we are establishing a public Gawker Tech & Product blog (a long time coming) from which we will communicate product information as well as product plans to our readers. You can expect to see it by early next week.

This has been a very unfortunate event in Gawker Media history, and we have learned much from it. Above all, this has been an enormous inconvenience for everyone affected, and for this I apologize. You can expect a much more responsive and proactive technology and product team for 2011. You can also expect a much more public me — if there is one critical thing that has been missing, it is a lack of consistent communication from me. That will change.

Regards,

Tom Plunkett

We have made it easy to comment on posts, however we require civility and encourage full names to that end (first initial, last name is OK). Please read our guidelines here before commenting.

  • http://www.facebook.com/profile.php?id=213359 Matt Jones

    What makes you think Gawker stored plain text passwords? The dump that was leaked was hashed and salted (albeit insecurely).

    And old-school passwd files on some unices did have plain text passwords. See http://serverfault.com/questions/116281/in-linux-debian-did-the-passwords-etc-passwd-ever-been-stored-as-plain-text.

  • http://fak3r.com fak3r

    Oh, and it took me a second to realize it, but these comments are run by Disqus!

  • http://fak3r.com fak3r

    I got that from it as well, but still, it’s a strange reference to make.

  • http://fak3r.com fak3r

    Great points, Disqus would go with their ‘never store user info’ mantra – why not? It’s obvious they’re not experts in user accounts, so let a company that focuses on only that deal with it. Your WordPress point follows this as well, what is a more secure platform, a closed source system that has a few with access to it for auditing, or an open source model where millions comb through the code?

  • http://www.facebook.com/brentlaminack Ivey Brent Laminack

    Let’s see… How long have serious computer science practitioners known not to store clear-text passwords? Unix is right about 40 years old and they’ve never done that since day one. In fact, every computer security guideline I’ve ever seen, such as NIST SP 800-23, etc. all say NEVER STORE CLEAR-TEXT PASSWORDS. I’m sorry, but this is computer security 101. Clearly neither Mr. Plunkett nor any member of his team has ever read a security standard. I may sound a bit harsh, but I’m a CISSP and I’d challenge anybody to show me any serious standard that says storing unencrypted passwords is acceptable.

  • http://twitter.com/danditomaso Dan Ditomaso

    ………

  • http://www.occidentaldissent.com nuke904

    tl;dr

  • http://www.facebook.com/sdguero Ryan Bray

    Looks like a certain CTO is trying not to get fired….

  • Anonymous

    Wait, Gawker’s gonna have a normal commenting system, instead of the secretive “you must audition first and follow the party line when making comments or they’re gonna get deleted” system?

  • Anonymous

    They should stop reinventing the wheel. Switch to one of the many blogging platforms out there like WordPress and use Disqus for commenting.

  • Anonymous

    And thanks for this post, I forgot about my disqus account :)

  • Anonymous

    I had a few accouts hacked. It just forced me to clean up my web habbits. I can’t blame Gawker for being stupid when I was just as stupid, and I will continue on Jalopnik. Kudoes to Linked-In for scrubbing their system and notifying me!

  • Anonymous

    I had a few accouts hacked. It just forced me to clean up my web habbits. I can’t blame Gawker for being stupid when I was just as stupid, and I will continue on Jalopnik. Kudoes to Linked-In for scrubbing their system and notifying me!

  • http://ditherati.com/ Owen Thomas

    That’s a singularly noisome notion. I suppose you also believe journalists in Mexico should stop reporting on the activities of drug gangsters — in fact, some have, because of threats to their lives. Is that really the world you want, where the brutish and strong can suppress reporting about their actions?

  • http://www.mac-adam.com/ Adam Turetzky

    Again, you don’t taunt the internet hacker community by calling them a bunch of useless script kiddies when your password for everything which keeps your 1.3 million users personal data safe is: 24862486

  • http://twitter.com/mkelley mike k

    Most skilled? Using brute force scripts isn’t being skilled. It’s using a script.

  • http://rendion.myopenid.com/ render

    Be a man and reveal the source code vulnerability. Post the code. It could help alot of people, or it could just embarass you.

    A man would post the code that was exploited. Be a man.

  • http://pulse.yahoo.com/_LGKTCBBVQJBMHRK3YSWTRGEDAE A

    What a lot of corporate speak without much content.

    – A Thompson

  • http://twitter.com/GlennF GlennF

    In fairness, he said, “are similar to the service a pre-paid phone offers to drug dealers,” using service to mean the tool by which drug dealers use it, not that pre-paid phones are only used by drug dealers.

  • http://www.mac-adam.com/ Adam Turetzky

    Hey Tom Plunkett, here’s a tip, tell your editorial staff not to sling insults and literally dare the most skilled hackers on the internet to break into your site and dangle your readers personal information as the prize.

  • http://robvincent.net Rob T Firefly

    This guy seems to be doing the best he can under the circumstances, taking lumps and moving forward. Good for him.

    I did not, however, realize that using a prepaid phone made me a drug dealer. That’s going to be a real heavy adjustment to make to my lifestyle; I don’t like gangsta rap and I don’t even have an aspirin in the house.

  • http://pithagora.com fjpoblam

    I approve and applaud your push toward OAuth verification. Were more site and cross-site (such as yours) account handlers to follow your lead, web use would surely be easier for many folks.

    I can add this suggestion, though, which I’ve never seen implemented. For a user who already has a “conventional” account and wishes to discontinue it in favor of OAuth verification, there should be a path for transition that does not require closing the conventional account and opening a new OAuth account.