Interview with Dane Caruthers
- September 17, 2003 15:07 PM PST
- Email this!
Think game testing is easy? The lead tester for Deus Ex: Invisible War thinks otherwise.
You?d love to be a game tester, right? Long hours of just playing video games, sitting in front of a TV and getting massages from sexy supermodels all day sounds like something you?d be interested in? Well, tough. If you ask Dane Caruthers, lead tester on Ion Storm?s Deus Ex: Invisible War, testing is more than just a job?it?s a repetitive, meticulous, and well, highly rewarding job.GamePro: How long have you been testing Invisible War?
Dane Caruthers: We?ve had the game in test from as far back in pre-production. Basically, as long and they?ve had a proof-of-concept tech demo, we?ve been testing it. The testing process actually starts with the design doc, going through ?here?s how the world should work,? and then checking as they go along to make sure that they?re following their own design philosophy and to make sure that the world works they way they said they wanted it to. As far as playable, usually it depends on the game company. A lot of times, the pre-production takes about a year, depending on the scope of the game that you?re trying to deal with, and then you start getting playable versions that you can actually start putting bugs in on and start looking for the product defects, like ?This just isn?t right.?
GP: So were you guys working on another game here during the pre-production?
DC: We?ve been working on Invisible War and Thief simultaneously. So we?ve got some of the QA teams working on Thief, and most of the teams right now are working on Invisible War, because, well, it?s coming out first. [laughs] So, kinda prioritizing there.
GP: How many people are on the team?
DC: Oh wow. Right now we?ve got about a dozen folks on Invisible War QA, and we?re actually hiring some more people this week for the final push. So we?re probably hiring another four to six people to just really hammer on the game. With a game of this ambition, you need that many people so things don?t slip through the cracks.
GP: How do you approach testing a game like this? Do you have to test every individual option and every combination of options?
DC: Oh yeah. With something like this, general testing, you can usually use checklists, or, you know, here?s your test script, and you can run through. That works for things like testing the bio-mod system, or does the icon light up, going down to very low-level stuff. When I hit the button, does this happen? Check. And you create a checklist for that. Working all the way up to the game logic. Like, can I go through the Order path in Seattle? Can I go through the WTO path in Seattle? With a game like this, we end up doing a lot more freeform testing, which more consists of, ?Okay, start a play-through, I?d like you to use this selection of bio-mods and these weapons, and go with this faction. Make sure it?s possible, because we want to make sure that whatever the player wants to do is supported.? I mean, that?s the whole point of this game. Whatever you want to do, we want to make sure you?re able to do it and don?t get stifled by us just having missed something. So a lot of time we just let the testers loose and say, ?You! You never get to pick up a weapon. Go!? or ?You! Set fire to everything! Go!? ?You! Nothing but grenade and a crowbar! Have fun!? Can you get through it like that? Any number of combinations like that. In a game like this, there are nearly infinite combinations of things you can do. The challenge is, we know we can?t catch all of them, but we want to at least get the general stuff along the bell curve?most of the stuff in the middle, and the stuff at the far ends.
GP: Will testing continue on this past launch? After all, on PC you can offer patches.
DC: Yeah, that kinda depends. The goal here is to ship something that we don?t have to patch. So, with any luck, no, testing will not continue after we get it out the door. When you?re dealing with a console title, you don?t get to patch, so we?re being very, very thorough. More than we usually are. I mean, I don?t want to ship something on PC that needs a patch anyway, but when you?re dealing with a console, you don?t have a choice. So any known issues, you?re going to either ship with stuff you can live with, or which are minor enough that it?s really not important, or you fix it. Those are really the only two options. We?re really trying to get away from the mentality of, ?Oh, we can patch it after launch,? which is an industry trend that I hate. And all players hate it, gamers hate it, nobody likes it. We all hate it as well. So we don?t wanna do it, and we?re doing our best so that we don?t have to.
GP: When you get close to shipping time, who decides whether you can go back and fix this bug that might push you past ship date, or whether that bug just goes, no matter what?
DC: Usually if it?s going to take you past the ship date, that bug?s probably going to ship, unless it?s something very major. As you get closer to the end of the project, you end up with something where Warren and Harvey and Bill (our producer) and a couple of the leads get together, and we go through the entire bug database and every open bug. Like, ?Okay, this. Is this something we can live with, does it make the game unplayable, do we hate it?? And that usually takes two or three days, of people just shouting at each other, ?Oh, that?s not that important, nobody will ever see this,? or ?Dude! Everybody?s gonna see this! It?s terrible!? And going back and forth, that kind of give and take, and letting go minor issues, like, ?Oh, well, there?s a typo in this conversation. Sorry. Don?t care. Ship it.? As opposed to, ?Hey, there?s a known crash issue if you do this fairly common occurrence. Okay, fix it, get on it now.?
GP: When you get a new build in, do you have to start the whole testing process over, just in case they introduce a new bug?
DC: Oh, absolutely. We actually go through three stages of acceptance testing. When we first get a build, we just get the editor, which is what the designers are going to be using, and we test that. We smoke test that to make sure they?ll be able to use it for what they need to. We don?t get any content or anything. They have time to hook up whatever data changes they need to make to the new code. We get that the next day, and then we go through another smoke-test process on that to make sure there?s nothing hideously broken. Then we open it up to all of QA, and we get into serious acceptance testing. At the moment it?s usually taking us two or three days before we are willing to accept a build for full testing. It?s at that point that we start regressing outstanding bugs and really just hammering on the game. But part of our acceptance testing is, ?Can I play through the whole game?? which, even on a speed play-through is going to take you a day or two, depending on what you?re trying to do.
GP: How many times have you had to go through all that?
DC: We get a new build every week.
GP: Since??
DC: Let?s see. We?ve been doing a build a week for two or three months. Before that we were generally getting two builds a week, but we weren?t as stringent on the acceptance process. The build process will speed up toward the end of the project, because you?ll still be doing the speed play-through, and you?ll still have everyone touching all of the systems for acceptance, but they won?t be changing as many things, because it starts getting like, ?Okay, we?ll only be touching critical fixes. We?re only gonna fix things that are really important.? And those minor changes we?d like to make, well, we shunt those off to the side and that?s what we look at in the end.
GP: Is there anything you?d like to add about the process of testing?
DC: For anyone who thinks it?s an easy job, look at it this way: You get to play the game over and over and over, until it?s actually fun and it works, at which point they take it away from you [laughs] and give you something broken to start over on. Once it?s really fun, we don?t get to play it anymore! You get to play it. [laughs]
GP: Do you think you and the other testers will be playing the game a lot after it ships?
DC: Probably. We actually did with Deus Ex. Especially the PC multiplayer. We had that going all around the office after it shipped for a couple of months, just merrily blowing the snot out of each other on Friday afternoons. I?ve still got it at home on the PC, on the PS2, and I fire it up now and then, for nostalgia if nothing else. Although I?ve pretty much got the game memorized. I?ve probably played it more than any other living person.
