Game Update 44 just went out the door two weeks ago.  During the time it was sitting on the Test Servers, developers were turning around fixes for bugs faster than I had ever seen before.  But there is nothing like having thousands of people try out a new zone or a new feature for a couple minutes to find the things we missed or didn’t think of faster than you can say “oops.”

So what I have learned since I’ve been working on EQII is that along with new content and a ton of little fixes and tweaks, one thing each Game Update brings along with it is a few Hotfixes.  Getting those Hotfixes from the hot little hands of the developers to the servers is where I come in, and often what I spend a decent chunk of my day doing.

Deciding what is going into a hotfix is a science in itself and could fill up several pages of a blog (foreshadowing of whats to come perhaps?) J.  But once the developers are done with their changes, I, or they, move it into a special branch just for doing Hotfixes (A branch is a full copy of the game and all the data needed, we usually have 3 branches at any given time).  I digress a little bit here, but if you’ve ever looked at the “SOEBuild” number at the top of your client window and wondered why it has a letter like “T” or “L” at the end of the number, that’s a reminder to me and others about what branch that particular build came from.

After all the data is in the correct branch of code, I push a button and it gets built by a farm of machines (I push a lot of buttons at work, kinda like George Jetson but with longer hours).   In most cases it sends me an email saying everything went well, but sometimes things go badly and I go bug someone to fix their stuff and try building it again.  If it’s a coder we have a really ugly doll named “BoB” that gets to sit on their desk until the next coder gets the “Breaker of the Build” title.

So once we have a complete build, I push another button and it gets put up onto a server.  If I’m lucky I’ll have remembered to email out to people that the server is coming down for an update.  If you’ve ever been on early in the morning and the server came down and we forgot to broadcast, that’s what it’s like in the office if I forget to send the email about the server coming down.  A few times I’ve heard people groaning and making unhappy noises from halfway across office floor just after I push the button, so I try to avoid that as much as possible.

Anywhere from 30 minutes to 3 hours after I pushed the first button we will have a new version of the game all ready for QA to look at.  I usually send over all the changes we made to the QA team via email and wait patiently for the results.  I say usually because sometimes something is so urgent to get out to the live servers we skip the patient email part and jump right to phone calls and nervous waiting. J

If everything goes well in QA, this is almost the end of my time with the hotfix.  I push one more button to move the files to the live environment, compile the update notes for the Launchpad and Gnobrin to post, and then submit a ticket with our GameOps group.  The GameOps group updates the network status page with the downtime and actually gets the client and server files from where I put them onto the Live Servers and Patchers, usually the next day or the next Monday if I’ve submitted the ticket late in the week.

The other thing that can happen quite often is QA sends back their findings on why the build really shouldn’t be sent live and what’s wrong with it.  Many times we’ll have four or five items that the QA group finds are working just fine but one of the things isn’t working.  The worst kind we can get back is when a change we made causes the client to crash all the time, when really the devs were trying to make it crash less often.  When this happens the whole process starts over from the beginning, even though we fixed more stuff than we broke, we still can’t put the Hotfix out to live servers until everything looks good or at least a lot better than what we had before.

Usually in the mornings I’ll start out by labeling the internal updates with the letter A and then B and so on.   A few times I’ve gotten all the way to L, M and N builds, but that is usually the exception rather than the rule on the number of iterations in one day.  By the time we get to N, its usually time to go home and try again the next day.  So far the one thing I have never run out of is fixes and tweaks that developers want to get out based on feedback or their own play sessions the night before!

 

Jennifer “Kirstie” Gerull
Associate Producer – EQII