« Following Up on the WoW Raiding Article | Home | It’s 4AM. Do You Know Where Your Boot.ini Is? »
Can We Put the Testing Back In Beta Testing?
By Cameron Sorden | December 5, 2007
Have you ever been a beta tester? I’d bet that most of the people who read my blog have either been a beta tester at some point or are waiting for a beta invite right now. New question: For those of you who have been a beta tester, how many bugs did you report while testing? For those of you that are waiting for an invite, are you excited to contribute to the bug-hunting community and submit lots of reports so that a better game eventually ships?
Or wait… were you just excited to try the game out before launch?
There’s a problem with the way beta testing is handled these days, and it’s hurting the MMO industry. Richard Garriot mentioned in a Gamasutra article today that he thinks allowing too many beta players into Tabula Rasa before the game was really ready to demo actually damaged the game:
“I actually think the biggest mistake was made not by the marketing department, but by the development team. We invited too many people into the beta when the game was still too broken. We burned out some quantity of our beta-testers when the game wasn’t yet fun,” he said, adding, “As we’ve begun to sell the game, the people who hadn’t participated in the beta became our fast early-adopters.â€
He continued, “And the people who did participate in the beta, we’ve had to go back to and say ‘look, look, we promise: we know it wasn’t fun two months ago, but we fixed all that. Really, come try it again.’ We’ve had to go out and develop free programs to invite those people back for free before they go buy it. So the beta process, which we used to think of as a QA process, is really a marketing process.â€
The perception of what it means to “beta test” a game has undergone a dramatic shift over the last few years. The actual purpose of the beta period is to allow a larger population into your game once you’ve nailed the major bugs so that you can start hunting down the smaller, less obvious issues that players will run into when they play your game. Over time, however, beta players have shifted away from bug hunting to simply playing during the beta period and using it to enjoy the game before it actually releases, no monthly fee attached. I’d be really surprised if even one out of five beta testers was submitting bugs on a regular basis.
This isn’t really the fault of the players, though. Game companies, in collaboration with fan and review sites (especially Fileplanet), have turned beta invites into a prize and a promotional tool. You don’t earn your beta slot through a demonstrated intent to find and report bugs along with prior testing experience. You’re given it in exchange for buying a site subscription or for winning a drawing. Companies just expect the beta period of the game (and especially the open beta) to carry a significant portion of free-riders that are simply there to play the game and have no interest in finding and reporting bugs. Then, when the game launches, there’s no free demo offered until months later, usually.
Marketing your games through the open beta like this is a huge mistake for a number of reasons. When you’re showing a product to your customers, you need to bring your A-game to the table. Even during the late stages of the beta process, you’re performing tweaks, tuning, fixing last-minute bugs, and adding new content (especially high-end content, usually). Letting players see the game before it’s really ready to be shipped, even if it’s almost ready, is sure to cause some discontent with the players who are just there to try the game out and see if they like it. If players are interested enough in the game, they’ll just as happily download it a month or two later when you’ve got a lot of those issues resolved and you want to launch a proper free demo program.
Another problem is that the really hardcore fans of the game and the ones that have been following it for the longest time tend to get into the beta fairly early, especially if you’re inviting large groups of people at middle-early stages of the beta. This means that a sizable chunk of your most vocal core audience, the ones that post on message boards and blogs, will sometimes have been playing the game for months in a semi-complete state before its intended release date. As soon as the NDA is down, they’ll post lengthy analyses about the flaws present in the game and stop playing, never having bought it– even if those issues get resolved in the next few months. This was the problem that Tabula Rasa ran into and that Garriot was talking about.
The whole idea behind using open beta as a marketing tool is a little wacky, honestly. Explain how it makes any sense to me to let everyone try your game for as long as they want, to any level and free of charge, until a specified cut-off point (after which the only way to try the game is to pay $50-70 for it), and then to set your cut-off point at a time before the game is actually finished and presentable. Wouldn’t it make way more sense to have a free demo available throughout the life of the game and available at a player’s discretion, with an established level cap?
Instead of this open beta nonsense, what companies need to start doing is scaling back their beta testing to really focus on actual testing with small groups of dedicated players again, while also preparing a separate demo program to help market their game to the rest of the audience. Ideally, the demo program would come a month or so after the initial push, to help drive interest in all the players who were following the game but weren’t interested enough to buy it without trying it. Also, that would give the company time to smooth out any gameplay issues which came to light in the first few weeks and prevent a flurry of comments from players who had been playing for months coming out the day after the NDA drops.
A free demo of an MMORPG is a necessity in today’s market. People have a lot of choices, and they like to know what they’re paying for. The industry standard right now is to show them an unfinished product and expose them to early informal reviews from players who were playing an incomplete game for a long period of time. That’s a ridiculous way to manage your marketing efforts. You end up hurting your playerbase in the long run.
Instead, companies should be keeping a tight lid on their games until they launch (outside of the usual screenshot/video hype), allow a minimal number of testers in for testing purposes, and then let everyone form their own opinions about a finished product when it launches. When you’re excited about an upcoming game, wouldn’t you want to see a polished, completed, and fun product when you try it instead of getting burned out on a half-baked endeavor before it can fully mature?
Topics: Random |
December 6th, 2007 at 12:35 am
Nice article, especially with reference to Tabula Rasa - as I’ve been in that beta. When I was there, I was trying to test it, not just try it out.
I raised a number of tickets, and posted on the forums, and didn’t get an official response from devs to either of those channels. It was quite a frustrating experience to test that game, because when I was trying to test rather than play for free, those efforts weren’t rewarded at all.
What I was left with was a frustrating experience on both sides - the game wasn’t finished, it was IMO not worth any monthly fee, despite being somewhat fun, and I got a clear indiciation that my $15/mo wasn’t going into support for the game: there was a poor customer experience for technical issues.
Now if we contrast that to something like Hellgate (oh the irony: the game is lots of fun but buggy as all hell even a month after release), the devs and CSRs were active in the forums whilst the betas were occuring. They collected information, passed it back out, and you felt like you were doing something useful AND getting to play the game early.
I’d like to see people stop with open betas too, unless they really want to test the game and provide the sorts of support channels that entails for filing issues and tracking bugs. The idea of a test to play after the game goes live is much more sensible - but you’ve got the problem where the *perception* of games today is that they only do a trial as and when subscriber numbers are dipping.
So it’s almost suicide to do what you should do, because people read it wrong.
I don’t think there’s a win here yet - I bet that both gamers and companies need to do some maturing in the product lifecycle first.
December 6th, 2007 at 9:02 am
I talked about this back when Vanguards beta hit and how they went overboard on their beta sign up process. I argued, in whole, that developers place way too many resources into the “advertising” of their betas.
The argument came up, that the focus of testing and testing processes needs to be set to centralize on the testing and not “who or how many” get invited.
That is why I will always argue that beta testing is best left to companies that do it professionally. The non-gaming software industry has relied on out-sourced testing since it started, but somehow game developers continue to press for internal testing and public betas.
Not only does this solve a lot of problems with NDA leaks, but it also allows developers to really narrow their testing sessions and get feedback from people that KNOW how to test and how to provide proper feedback.
Without knowing how much it costs to run a public beta compared to hiring a testing firm, I have no real evidence to say it is economical. However, it still makes more sense in a time when so much money is being flooded into MMO development.
December 6th, 2007 at 3:54 pm
As I wrote in a blog today, I think it’s more complicated than that.
A gamer can be excited about a game based on all the preview information, but that preview information won’t tell him if he/she would enjoy the game. True, the finished game will be different than the beta, but much about the finished product can be seen in the beta.
Testers often sign up for the beta of a game they’re excited about, then get invited in to realize that the game they’re actually testing isn’t worth their time.
They took the gamble and still owe the developers some testing time, but I don’t blame testers for not committing a great deal of effort to a game they’re not interested in. The sad reality is that interactive entertainment is hard to estimate until you start interacting with it. Trails, FAQs, and all that don’t offer a very clear idea of what they would be testing, should they sign up for beta.
December 7th, 2007 at 7:08 am
Now that these games are becoming such hot little numbers, it’s a given that more and more people will want to be in the “beta” for bragging rights, early free playing, etc.
What needs to happen is really pretty simple though. Shorter testing phases, focused tests, and more discerning applications. Go to Jumpgate Evolution’s page, and you’ll see a simple address entry spot to sign up for beta. Just an e-mail address. No questions, no why do you want to test. Just “give us your e-mail and maybe we’ll let you in to play and test”.
And companies wonder why the attach rate of beta testers is so low. They wonder why for every 10 invited maybe 1 stays and tests the whole time.
I’ll admit to quitting on beta tests before they’re over. I did so with TR after the NDA dropped. But for all the game’s I’ve tested, when I play, I bug report, I give forum and e-mail feedback. Does this make me the world’s best beta tester? Nope.
But it does make me better than the folks who sign up, get in, play the game, dislike it, and then call it shit later on or just plain break the NDA.
If the game makers are more selective, and the applications more in-depth, they can avoid a lot of these problems. Save the thousands of invites for when you want to use the Open Beta marketing tool, but keep the actual testing phase focused and small.
Sorry for the rambling… way too early in the morning for truly coherent thought.
December 7th, 2007 at 10:36 am
I agree with what you say, except that instead of a demo, it should just be a free week or two of the actual MMO after the game officially launches. Keep the anticipation level at a fever pitch.
Keep the general public drooling over what the game might be like. No free previews before it’s ready. And launch the game when it’s really ready to launch. Don’t let any of us who might review the game get to try it before it’s in a finished state.
December 7th, 2007 at 1:50 pm
While I still rant and rave in forums when people get in betas just to try the game for free, I’ll also admit to not fully testing the past couple betas I had been in, Tabula Rasa mainly.
I fell for the TR hype. I really did. I wanted to believe it. I signed up early for beta (though didn’t get my slot until the last couple months prior to launch) and was fully intending to devote my time to testing every whacky thing I could think of and submitting feedback to help make TR the best it could be. By the time I got into beta the impression was that tweaks aside, this was going to be TR and nothing any of us latecomers said or did was going to change that. The gameplay itself at that point in time was such a turnoff as well. I was just shocked. Maybe that’s how the Vanboi’s felt when they were sucked into the McQuaid Hype Machine and were left with a collective “WTF?” when Vanguard launched too early. Needless to say I didn’t feel compelled to even login to TR to play, much less test. Their bizarre server hours conflicted with my work schedule only making matters worse.
Thinking about it though, The Burning Crusade beta may have burned me out on beta testing as much as it was a huge contributing factor in burning me out on WoW, as well as being the final straw that caused me to cancel my sub. In addition to work and real life, I was hardcore-ish in WoW towards the end, raiding nearly nightly with 3 characters. Occasionally doing two raids per night. My days were divided between farming for gold for raid repairs and consumables and very actively beta testing TBC. I was pretty hardcore about submitting every little (and big) issue I could find and going out of my way to find them. It all just got to be too much in too short a time span. I have only myself to blame for that, and it’s not a trap I plan to allow myself to fall for ever again.
Ironically, I’m hearing nice things about TR post-launch, but I still think I will stick by my decision to give it at least 6 months, if not longer, before I decide whether or not I’m interested in giving it a go.
I’ve read a few blog posts over the past year about beta testing, and I agree with most, including Cameron’s. I absolutely believe the devs need to be active in the process. In-game as well as forums. Make it simple to submit bugs, tickets, whatever. And here’s an idea: tell the testers specifically what you want tested in the current build, in addition to whatever they might find along the way. Give us direction and guidance. We already go crazy and “sandbox” it all anyway, finding little bits of minutia the internal testers didn’t catch, but if you include specific code in a build, then put up a huge in-your-face obvious announcement in-game telling us all what specific actions, locations, whatever that you want feedback on. Help us help you.
December 7th, 2007 at 2:02 pm
I also believe beta testing is a privilege not a right, and perhaps there should be a quota to fill to keep the position. If I get a beta slot and for whatever time period submit nothing, take my slot and give it to someone else. Otherwise I’m just a weak link in the chain, leeching a slot that could go to someone better able and willing to do the job they volunteered for. Maybe that’s how both the devs and testers should think of it: a volunteer job.
Along with that, the devs (or better, the publisher) should keep a database of their testers and those who did a great job could get priority invitation to the next beta from that publisher if the game fits within the testers gaming profile.
Finally, perhaps reward the core testers who stuck with the process through thick and thin. Something as small as naming an NPC after them or (if characters get deleted at launch) making those characters themselves profession trainers, or somehow including them in a storyline somewhere.
We have leaderboards for PvP (MMO and otherwise), end-game gear for the raiders to strut around in, The Chronicles of Spellborn is putting up statues of the PvP leader (in whatever gear he was in at the time). Whether we’re actively “competing” or not, we do like to be rewarded somehow. If being a beta tester is like a volunteer job, find some way of acknowledging, rewarding and thanking those testers for the hard work they put in to make your game the best it could be at launch as their “compensation” for that job.
December 7th, 2007 at 6:16 pm
I agree with Richard to a point.
Large numbers shouldn’t be invited into early stages of beta which, from a design view, should be only focused on bug resolution in the games systems.
The large open beta should be reserved for when the game is pretty well finished, has as few bugs as possible and just requires stress testing, final balancing issues due to numbers and should be for a limited time (2 weeks tops).
A lot of “beta testers” aren’t there to do the developer any favours, they are just there to cop a free “feel”, and will leave with a “set in stone” mentality where an issue they had with the game will be what they remember months after (even if that issue was patched the day after the guy left and people will have trouble remembering what the “beta tester” is ranting about in that post 5 months later).
A lot of the “informed opinions” of what a game is like on boards like FoH are obviously from the beta period of games, and when they get called out about “It isn’t like that now”, you’ll more than likely get a “can’t polish a turd” type response.
So, yeah, long term open beta do actually hurt a game, and can for long time to follow…
QA should also have much more power in the process, but that’s another issue entirely…
December 7th, 2007 at 6:29 pm
I’d think Talyn brings up a good point on a thing that even I’ve fallen into (I’ve done a lot of beta, both single player and MMO), where people don’t feel like they get listened to in a large beta.
The more people that get in to the “beta”, the worse that is, and even though most folks see “open beta” more as a “preview”, the fact that it’s still called a “beta” means they have certain expectations (one being that feedback is listened/responded to).
I believe that “expectations” are actually one of the area’s where most MMO’s are being let down these days, there is too much “hype” and not enough player expectation management.
After all the hype you get hit with for most games these days, it’s very unlikely the game will actually meet the players expectations…
September 10th, 2008 at 5:19 pm
[...] we stop pretending that an open beta is anything but a free trial? This complaint ties in with a post I made last year where I complained about the practice of using an open beta as a marketing tool. [...]