My time making MMO games, securing them, and the future.

UO Dragon Addition

People who have read my bio or that have seen my LinkedIn page know I worked at Electronic Arts (EA) for a very long time.  But I kind of don't go into a lot of what I did there, and some of that is for good reason.  But I thought it would be fun to start talking a bit about MMO gaming and security, and in doing so I want to tell some fun war stories from back in the day.  To get to that point, I will start with sharing a story on how I came to EA, the game I worked on (hint, the logo is this blog's picture), and give a little taste of the future MMO/gaming security stories to come.  Who knows, maybe this ends up turning into a tell-all book on the early days of MMORPG.

A long time ago in galaxy far, far away - I have always wanted to do that...

I lived in a land so magical and happy we called it  "Maryland", and I worked in a large company where I built huge networks and designed such fantastic things as the world's first-ever online hotel reservation system (this is the part where you all make gasping noises and start to go on Ebay looking for autographed photos of me.)  And in my spare time, when not developing some of the first ecommerce systems and designing private data centers, I beta tested computer games, for the low-low price of giving me the game for free and letting me test it.  Oh I was the mad man back then, I had two-channel ISDN, yes all 128kb at my disposal for low latency fragging.  (Did I mention I played a LOT of Quake.)  This of course happened when I wasn't playing MtG at the local comic store or at friends' houses.  And before you ask, yes I had a real live girlfriend... who to this day makes fun of me for doing all the things I have just mentioned and is really quite a normal person.  She played MtG too, but only so she could spend time with me.  We call that "love."

Oh yea, so anyway... I started play testing this really cool concept of a computer game.  You see, it was going to be massive, multiplayer, persistent state, and you could only play it while you were online.  MMOPRG or simply MMO is what we call it now, but back then this was CRAZY TALK!  I spent hours, sometimes entire weekends, playing in a persistent world that was constantly reset.  Waiting patiently for servers to come back online, and often loosing connection while I was able to play only to find that the persistent world wasn't so much "persistent" as it was "quasi frequently backed up" and I lost about a good hour or so of play time from the last point the world was saved to when I lost connection.  I suffer a lot like this as I am hardwired to be an early adopter of all technologies, gizmos, and software.  The up-side, I really appreciate things when they do work.

So while playing this game, or really wait for it to be online again, I would sit in the IRC chat channel with the developers.  I found that I was often telling one of them my hypothesis for what was happening with their network and servers.  Finally, he asked if I happened to have a very specific certification because the person they had that did have this certification and background was gone and clearly they needed someone like that.  Luckily for me, when not working myself to death and distracting my mind with fantasy games I actually had taken the time to go to school and get certified so I could have a job where I worked myself to death and needed fantasy games to distract my mind.  Wait... no... well... yea that's right.

I interviewed, flew to Austin, Texas for an in person interview, and got the job.  I was now the head of IT for Origin Systems Inc, (OSI) a studio of Electronic Arts and I was there to make things go! And Go they did!  We got that network revving, the servers humming, and soon we were at thousands and tends of thousands of people playing at one time... eventually we got to hundreds of thousands but that is a longer story that is highlighted by me apparently finding red kryptonite and turning into a total ass for a few years.  It was fixed by Gordon Walton telling me I was "far too young to have this much angst" and he only has the excuse because he was old and mean.  I'm paraphrasing, but that about sums it up really.  I am a huge fan of Gordon, and he just wouldn't be Gordon without all the mean and angst and sarcasm. Although I think I am as old today as he was back then, so perhaps I can... no, doesn't work for me.

Anyway, I liked security even back then and was quite heavily involved in the setup of the rudimentary security we had in Ultima Online (UO).  We had several #FAIL moments as we were kind of blazing the trail in commercial MMOs.  First, we kind of shipped out thousands of copies of the client with no client to server encryption and only later turned on the feature.  This meant everyone and their goat had network captures of plain text network traffic to use to decrypt the conduit once we started using encryption.  (Remember, the decrypt key is actually on the client... so you know, it can read the traffic and encode traffic in response.)  

Commercial MMO was new, and we had to prove to EA that it could be profitable.  That meant not only was UO actually designed to be a pilot, but a lot of our ideas were going into the "when we build this again and do it for keeps..." list.  Later we realized this was for keeps, and we continued to retrofit stuff into it.  Fortunately UO has an amazing level of logging and engineering tools behind it.  This has a lot to do with the amazing network and administration team they built before I was even offered the job.  E.g. the guys who grilled me during the interviews.  This lead to a lot of good CSR work, and eventually concepts like community relations.  

MMO has come a long way since these times.  We now realize that security in the game is just as, if not more, important to the success of the franchise as features and good UI.  From conception, security is looked at in all its forms.  From the architecture of the network and servers, to client-to-server communication, back-end support systems, payment systems and even client security.  Cheating in MMOs is profitable for those who are good at it, may be devastating to it's victims, and damages brand image and market confidence. Attacks on Sony earlier this year have shown the entire world what can happen to your stock price if you are the victim of focused attacks.  And that has caused changes within the rest of the industry.


I will put together a few war stories, showing the different types of attacks and methods that "hackers" use to victimize customers of MMOs, the companies that manage those systems and give insight into what you can do to better protect yourself.  Look for them soon on