Dev Log #3: What's in a Technical Test?

[p]Hi again! We’re busy prepping for Pax West at the end of the month. If you’re in the Seattle area, come check out our first convention for Stars of Icarus! So this week we’re going to talk about something a little different and do a look into the realities of game production, and why developers do things like closed alphas, technical tests, and betas.[/p][p]We’re just wrapping up our very first Pre-Alpha technical test with a small segment of the community. We learned a ton from it. And we got to show off the game to new players for the first time, who had a blast! (If you didn’t get a chance to test things out, keep an eye out, we’re planning to have some larger tests in the near future.) Keeping early tests small is a critically necessary part of scaling well. So let’s dive into what a technical test is for, how it’s different from an alpha or beta test, and how our first one went! Hope it gives you a bit of insight into why you see games have these long and complicated pre-release periods.[/p][p][/p][h3]Breaking Things at Scale[/h3][p]Often the earliest development version the public is likely to see from a game is a “technical test” version. In short, a technical test like the one we just ran is one designed primarily to see what breaks. Generally you see them in multiplayer games, because scaling a multiplayer game is really, incredibly hard. You see scaling break games all the time. No matter how much time, money, or testing goes into a game launch, you can’t escape day 1 server issues. Developers scramble to put out endless small patches, put up long queue times, and do emergency downtime maintenance just to try and tread water. Years of incredibly passionate work by skilled developers, new indies and veteran AAA alike, struggle under an unpredictable and untestable strain of excited players. We’re certainly no strangers to the experience.[/p][p]Guns of Icarus had its own uniquely turbulent launch. After several months of closed and open beta periods, launch day approached in October of 2012. But meanwhile down the coast, so approached Hurricane Sandy. A huge influx of every beta tester, all the kickstarter backers, and all the new players just getting in on the excitement jumped on to shoot some airships out of the sky. It brought the GoI master server to its knees. And horribly, hilariously, it slowed the match start countdown to a fraction of real time. Every nominal second of countdown took 10 to 20 seconds to actually tick down. Myself (just a community moderator at the time), and the rest of the moderator team at the time, thankfully were able to manually kickstart matches and skip the timer with cheat codes. But randomly pinging in global chat to summon a mod was hardly a real fix, and the dev team was hard at work trying to figure out what broke and how to fix it. But out their windows, Manhattan was being hit with a hundred year hurricane. And while a bug fix was fast to find, deploying it required a server sitting offline in an office downtown, and the floodwaters were making the city streets do a passable impression of Venice. The closest programmer took it on themselves to retrieve the build machine from the office, convince security to let him into the building as the city flooded, and the day was saved. A great story! Stop by our booth at PAX to get a much better rendition from my boss! But also the kind of thing you really never want to have happen.[/p][p]So, you do everything in your power to try and prevent things from exploding. You plan, you review your processes, you test, and you scale up slowly to make sure you catch when and where things break. You try to remove as many single points of failure as possible, like the GoI build machine. And just as importantly, you find out things you didn’t even need to know.[/p][p]With our Stars of Icarus technical test we made two pretty good discoveries right away. The first was just a bug where we’d never actually request a second server from our backend when we actually needed it. So we had a max of just one game per region when we first launched this test. Our second discovery: we actually had no idea how we push out patches in a live environment without doing full downtime. It’s something we’d chatted about before, but never actually had a full plan for. And so we were pretty rapidly learning, experimenting, and breaking things in real time, within hours of opening the test. But we were doing it in a very low risk environment.[/p][p][/p][p][/p][h3]Just the Right Amount of Risk[/h3][p]Risk is why we do such a small scale initial test. We only alerted a small portion of our community about this test, and made them jump through a few hoops to get in, just to make sure we were only testing one step at a time. Keeping things small is a necessity, because we know it will break, we just don’t know how yet. So when things absolutely, inevitably go wrong, the worst we can do is disappoint the 200 people we first invited. Thankfully for these testers, our big issues were handled nearly seamlessly, but only because we managed that risk properly. If we were under the weight of hundreds to thousands of concurrents, the game would have been entirely unplayable. Even if we had launched during peak hours, instead of mid workday, we’d have caused a fair bit more trouble right away. With any larger a test, we would have learned a lot less in the process, as we scrambled for the quick and dirty fixes trying to keep things running under strain.[/p][p]Another risk we’re basically always managing at this stage of development is PR. They say all press is good press, and that’s often true, but we were definitely staying a little camera shy for this test. It’s not that we don’t want to show things off, we’d love to, but we’re still at a point in development where there’s going to be really notable differences between what’s visible now, and what we launch with. From experience, what we put out now will stick with the game for the rest of its life. We once put out a few early development screenshots of our game Embr, and we had players and press sharing them well into release day news coverage. It looked almost like a different game at that point! [/p][p]Since this technical test was our biggest experiment since announcement, we don’t know what might explode and look terrible. So we played it safe, which allowed us to stay focused on technical needs. There’s still a lot of people that haven’t heard of Stars of Icarus yet, and we want their first impression to be a good one![/p][p][/p][p][/p][p]This was up all the time, so even if someone didn’t read literally anything we said, this would still show up in the screenshots. If it leaked, we had it plastered on the image that it wasn’t meant to be shared yet.
[/p][h3]Mission Accomplished[/h3][p]So for us, the technical test was a huge success! We broke just the right amount of things. And from what we saw, our testers were having a good time too. Far from perfect, but we were happy to see some good laughs and good matches at this point in development. There were also still a few total game breakers for some people’s PCs, and a couple big bugs that could ruin a match. Again within expectations, but still frustrating to deal with for testers. But when it all worked, we got some great gameplay feedback as well, even if it wasn’t the focus of this test for us.[/p][p]So what’s next? Alpha, beta, release? Well we’re still working on scheduling all of those things. We wanted to see how this test went before we went any further. We need to take full stock of what issues we still have, what feedback we need to action, and what we want to test next on the tech end. But given how the test went, the next time you see us testing we’re likely to move onto closed alpha testing.[/p][p]Compared to this technical test, an alpha means practically: letting more people in, focusing on more of the gameplay and balance changes we’ve been working on, and further increasing scale to something more in the shape of the final game’s server and backend needs. An alpha to us means the major systems are in. And in the background we finish game content, and start balancing the game for the complete design.[/p][p]Further down the line, beta is when we really nail down everything that’s going into release, make our final changes and fixes, and send as many people at it as we can get, to really find even more ways to break things under stress. Because building a game is all about breaking things one step at a time!

We hope to see you there later this year, and keep an eye on our discord for the latest info.

See you in the stars,[/p][p]Matthew[/p][p]Muse Games
[/p]