Play testing and the process of gathering actionable feedback on game levels can be a tricky thing. Everyone plays games differently, and I’m sure we all know that guy / girl who gets stuck on the most basic puzzle sequence in a level. As level designers we try and limit these progress blockers as much as possible with clear and understandable designs but inevitably some will slip through. But wait, there is hope! A group of individuals exist whose purpose is to test, test, test until these wrinkles are no more. GladOS would be proud of the men and women of the Beta Testers Collective.
Get help squashing bugs with the Beta Testers Collective
Modding & Development
Originally formed under the wing of PlanetPhillip.com and recently breaking off to form its own entity, the Beta Testers Collective (BTC from now on) are a lean, mean, testing machine that aims to raise the quality of levels they get their hands on by rigorously putting them through their paces. I caught up with the lady at the helm of this particular ship to shed some light on the BTC and their methodology. Level designers take note!
Cue suitable intro music!
Welcome to LambdaGeneration! Could you explain what the Beta Testers Collective is for the readers that perhaps haven’t heard of you?
Hello there and thank you for this opportunity. First let me introduce myself, I’m Ade, the now not so new leader of the BTC. The name in question denotes, as you may have guessed, a group of dedicated testers that provide feedback mostly for Half-Life 1 and 2, Episode 1 and 2 custom maps and mods that are in the stage of development. We believe in testing early and testing often, the Beta is just a term most people are familiar with and it’s also the stage most authors start testing at.
You’ve been independent for a while now but originally you were part of PlanetPhillip. How was the BTC created and how has it changed over time?
With independence comes great responsibility but I managed to absorb most of it and hopefully the transition was barely felt by testers and authors alike. Phillip (owner of PlanetPhillip.com) undoubtedly is irreplaceable. Around 2008 he felt that too many mods were suffering from poor testing and his idea for a specialized team to fix this had enough followers to form a project.
Over time, we’ve undergone many changes, from the obvious membership list, to the offered feedback formats and posting means and platforms. Most of us are old timers on PlanetPhillip and have interacted or worked with him for a long time now but that darned thing called Life intervenes once in a while rendering some of our colleagues inactive and so nowadays we’re also searching for new testers outside this circle. I believe the name has always been the same, long but precise, and despite being given the option to change it, I stuck with it for many reasons.
What methodology do you employ at the BTC when looking at maps?
The main things to look for while testing have actually been the same throughout. In general, a map or mod can be viewed and judged from several points of view, besides the usual overall experience and visuals that your average player considers, like: intro, outro, sounds, gameplay, atmosphere, story or objective, map bugs, game breaking bugs, puzzles, difficulty.. and I’m sure there’s many more. But the way we provide these views has definitely developed, from the mere paragraphs (some of which could be considered essays) and screenshots we were all used to posting on the main site for your usual map or mod, to demos, screenshots with annotations, live playthroughs (my favorite) and of course videos, as recording one’s playthrough became more and more popular.
We use an open feedback format, as this has obvious advantages over the compartmentalized or private feedback. Redundancy is also good as the author can easily miss what might seem insignificant at first, but noticed in great number would develop importance. The open feedback was first put into practice on Phillip’s blog, in an adjacent section of his main site, except the WordPress platform was less user friendly than the phpBB forum platform which we use today. And even on the forum, all the requirements and protocols involved, from the initial contact with the author to the closing of an iteration section or test, have let’s say transformed to fit everyone’s needs and to ease the entire process.
Ok, let’s say I’ve created a set of levels for Half-Life 2 and the time has come to get them play tested. Describe the process you use in the BTC when a level designer submits their work!
First, I would ask you why you came so late for a test when we can start with even 1 sketchy map. Then, I would get over myself and thank you for contacting us and deciding to use our help. You mentioned the author directly submitting their work, except that doesn’t always happen at first. They might have something at this stage, but not ready for submission to a broader audience (people besides them and their trusted friends), but that’s another topic. Basically I ask everyone to read our site for more info and for the contract bit, after which I try to answer any questions left and follow-ups and a reminder that we can do live tests is included.
We also ask for a good sense of total number and frequency of iterations while also suggesting sectioning long mods and/or assigning different testers at different stages. The next step is usually asking the author for the following information: physical availability, such as repository details, a working download link or an attachment containing a passworded archive (that we can upload or even host) for a working map or mod, basic info about the content, such as engine, game, genre, title (these are usually given by the author in the first approach, or we already have that if we were the ones contacting the author), number of maps to be tested in this iteration, stage of the maps, stage of the overall mod if present, number of total intended iterations, deadline, desired aspects to be looked for, any preferred testers. Forum registration is not mandatory, but a strong suggestion in order to properly collaborate, and the option of joining the Steam group is there for faster communication.
I test the link or archive without actually playing the content and create a new section on our forum once I have the basic info such as game, temp name, length, aspects and deadline and include any notes from the author. After some fiddling with the backend of the forum, I send out a call to arms through e-mail to testers according to the game in question, the forum link is also sent to the author if he decides to register and the actual testing begins. Each tester has their own thread for individual feedback but can contribute to other threads and discussions, as well. The author can then ask for additional feedback if necessary and decide whether to use it and how. The entire forum section remains open until a new iteration is presented or the work is actually released, even if in time it becomes inactive, as new testers can pick it up and still be helpful.
What happens when the testers and the map author disagree about a certain aspect of the level? Bonus points if the answer involves a round of counter-strike!
In the end it’s the author’s work and choice, we can only offer our feedback, it’s not mandatory to use it. Some of it might just be subjective, as you can imagine. Yet there have been sad cases where most of the objective feedback for fixable parts were actually ignored and so, in the final release, these weak points were noticed and mentioned in the players’ reviews. Things like that really bring us down since our main goal was not accomplished – to make a mod better.
Something you hinted at earlier is that map authors don’t like showing their work until it is “ready” and usually this means that the level is fairly developed in terms of details and polish before a
tester even sees it. How detrimental do you think this is to the level’s quality (if at all) and when does the BTC prefer to get its hands on a level?
I was actually hinting at the fact that most authors just don’t trust people other than their close friends, and this could be related, as this is how I imagine their thought process: they might not be so comfortable with handing out early versions in the fear of being copied, but rather a version closer to release so if it does get stolen, it might not have enough time to be released prior to the original work. But you have to be honest with yourself and admit that friends will always agree with you and your work, they could make the worst testers ever, proper and constructive criticism might never come out of them so you have to trust another set of people, one with experience and objectivity, to handle such an important step.
As for your point, it is indeed another issue, since we believe in early testing as mentioned above. You can imagine spotting an issue early on and having to correct it in one place affecting one thing, compared to noticing it very late in the development process and making a tone of work for yourself trying to mend or simply redo entire parts from scratch. Things like this do and should happen only if the team decides to simply change the direction in which the mod is going and not because let’s say this new model actually has an unfixable bug and needs to be taken out entirely, and the whole mechanic around it, plus level B1 and C2 since they won’t make sense without it now, not to mention come up with something just as new and exciting.
So to finish up, feel free to pimp the BTC some more! Any final words or shout outs you would like to do, now is the time! Thank you for taking the time to talk to Lambda Generation.
I tend to be modest so for this I will just let our work do the talking, for those ready to test their work at higher standards and release a quality product. Thank you for having us! It’s been an interesting journey down memory lane and hopefully we touched enough points of interest to get more of the right people involved.
To round out this article I also wanted to get the thoughts of a level designer who has been through the BTC process, the name Miigga may be familiar to anyone who enjoys Half-Life 2 custom maps. His level ‘Baryonic Predicament‘ has been a big hit in the community with its quirky puzzles and intense combat. He had this to say about the BTC testing process :
Before the Beta Testers Collective, I would mostly rely on myself to playtest my maps. This is not ideal, because as the mapper I know exactly what the player is supposed to do in order to progress in the level. I also had a few Steam friends I could trust to playtest my map and give me feedback. They were not experienced playtesters, which means that I would expect to only get one or two pieces of criticism from each.
My memory is a bit hazy here, but I do believe I was introduced to BTC by Phillip from PlanetPhillip, while I was working on my largest mapping project, Baryonic Predicament. That mod was then playtested by BTC (among others, including Daz) twice: once when the mod was 60-70% done and then once again when the map was almost completely done.
Having never had proper playtesting for my maps before, I was pleasantly surprised by the amount and detail of feedback I got from them. They provided me with lists of issues from each map. Many even provided demo files of their playthrough. Going through all of this information and making the fixes I deemed necessary took a while, but it was definitely worth it. Looking back at the changelogs I wrote, the playtesting caused me to make over sixty changes to my mod; some of them big, some of them small. I ended up with a significantly more polished end product than I otherwise would have.
The BTC taught me the importance of proper playtesting. I would now be hesitant to release a map or mod without having it tested by experienced playtesters.
Baryonic Predicament was tested by the BTC
So, if you are a designer currently working on a level for any of the Half-Life games and need some good constructive feedback, the Beta Testers Collective is the best place in town. You can start the testing process by visiting their website, do be sure to read all the fantastic information on the front page!
Kind of questionable of where this will go, but more about play testing.
Most players aren’t technical wizards, they don’t know how the engine will handle things, how things get done, etc. For them, the playthrough is mostly about “Did I enjoy it?” or “Was it even worth my time?”. Most things should be corrected by the map/mod maker right off the bat. This includes textures that are misaligned, lighting is too dark, ladders don’t work, spawns don’t work, etc.
Experience or enjoyment is hard to be “professional” about. Since humans work off of comparing a lot of things, (hot to cold, love to hate, etc.) a simple scale can be deployed (ex. 1-10). That all being said, it sure does help knowing how Source or other game engines handle things. I currently do QA for the greenlit mod No More Room in Hell. Performance is a key issue with enjoyability. As far as I remember, that’s what I spend most of my time on. Questions such as “Does my FPS drop when I look a certain way?” matter. Little technical know-how is required, and the more hardware that tries it, the more people can enjoy the game. Objective branches get fixed quickly since they will be obvious due to limited number of branches.
I guess I question it’s usefulness and what light it puts the developers in if they can’t do that whole process themselves.
Already applied to beta test. This looks like a great service that with increasing popularity should increase the overall level of quality of maps and mods. I’d be quite happy to be a part of that.