27
Jan

Long Day at the Salt Mines

In case anyone is curious about what it’s like to run a website, I submit today’s adventures as evidence…

I awoke this morning to an inbox full of 500 errors. Apparently, sometime during the night something went bonkers on the server and now people were seeing errors whenever they tried to view their profile.

Some quick digging showed that it was related to an SQL query that pieces together the recent updates for your friends. I managed to fix the query, but the underlying problem was…nobody had any friends! Either OP had gone through a massive hate-fest where everyone unfriended each other in a mass exodus, or something was wrong. I knew it was going to be a rough morning. I called Ryan, and luckily he was available to help. So, we both got to work.

We quickly narrowed it down to two possibilities: a bug, or an intruder. The worst case scenario was that we had an intruder on the server who thought it was funny to unfriend everyone. This isn’t standard operating procedure for hackers, but who knows what goes through the minds of those jerkos. But, we take security seriously and decided we needed to be absolutely sure. So, while Ryan combed the code looking for a bug (the most likely scenario), I started looking through server access logs to see if our server had been compromised.

Surprisingly enough, someone had been trying to break in. More accurately, an automated brute force login script was running against one of our servers. We have a program to defend against this, but it had apparently crashed silently the day before. Someone had spent 24 hours trying to break in. I was 99% sure they didn’t succeed, but coupled with the data loss it was extra suspicious.

Luckily, while I was trying to gather more evidence, Ryan found the actual bug that caused the problem. We could breathe easy knowing no one had made it into the system. All our problems were due to my own incompetence. Still, while we were down in the guts, it made sense to tighten the security a little. Web security is a long scale with your grandma’s unprotected AOL account on one end, and your bank’s ultra-secure website at the other. We’re proud to place ourselves much closer to the bank than grandma. We use SSL to encrypt your login credentials, and we also use salted hashing to disguise your password in our database. So, even if someone did manage to break in and steal our database, they would never be able to get your password. (Note: This is why we can never send you your password: We don’t actually know it!) Plus, we try to keep our server very secure so a break-in is practically impossible.

We updated some of our security procedures and put in place some practices we’d been meaning to do for a while. The end result is that we spent about 6 hours each making tweaks and adjustments and you, the user, will probably never see any difference. That thought always depresses me a little, but I’d rather it be like this than have to send out a massive email explaining that we were sloppy and someone stole or deleted all your data.

Finally, as I’m writing all this (11:30 PM, Eastern), Ryan is restoring all the deleted friend data. We keep offsite backups of all your data, just for cases like this. Being prepared for a worst case scenario is part of our strategy.

Not exactly an average day: Responding to lots of “It’s broken!” emails (which are very helpful, by the way, so keep sending them), sifting through server logs looking for the boogey man, fixing bugs, tightening security, and finally restoring data. I’ll probably get to bed around 12:45 tonight after verifying that everything is running smoothly. Things are usually quieter, but we’ve had days that were crazier.

I hope you enjoyed this glance behind the curtain. We love working on OP, and that’s why we’re here banging away on it late into the night. I just hope everyone out there loves using it as much as we love working on it.

Award Winning!

Gold ENnie for Best Website 09'-11'


Silver ENnie for Best Website, Best Podcast 2012-2013
Petrified Articles
Categories
© Copyright 2010-2024 Words In The Dark. All rights reserved. Created by Dream-Theme — premium wordpress themes. Proudly powered by WordPress.