For our next ‘Report In!’ developer interview, we check in with QA Lead Miloslav Cinko to discuss the role of Quality Assurance (QA) in Arma 3’s development!

We're often asked to let some of our people talk about what it's like to develop a game at BI. 'Report In!' gives you a more personal perspective on our team, a more detailed look at the way we go about our work, and fresh information about our ongoing projects.

The QA department is considered to be the guardian of gameplay, presentation, and overall quality. Talking to QA Lead Miloslav Cinko, we ask him about what it means to test a game like Arma 3, how they’re handling the Alpha and Beta, and the development of new features.

Introduction

[thumb class="pull-left" clear-xs]miloslav_cinko.jpg[/thumb] Tell the people a little about yourself. What's your role? How long have you been with Bohemia Interactive? Which projects have you contributed to and what is your favourite BI game or mission?

I am the QA Lead, which basically means that I’m responsible for bringing people the best game possible. I’ve been with Bohemia Interactive ever since it took Altar Games under its wing.

As for my favourite game, it’s a bit hard to say, but I’ll probably go with the British Armed Forces DLC. There are lots of reasons, starting with me being the QA Lead for BAF, having my own mission in there, alongside a couple of great weapons (I like bullpups) and the Jackal vehicles.

And can you give us a random fact about yourself?

I’m a ‘wargamer’ (Warhammer 40k), which kind of combines my hobbies, such as assembling and painting miniatures, playing tactical games and sci-fi in general.

A Bug’s Life

You’ve told us your role, but could you detail the QA process: when does something end up on your desk, what do you do with it, and what happens afterwards?

[thumb class="pull-left" clear-xs]report_in_qa_1.jpg[/thumb] This really depends on what this ‘something’ is. It might be a bug, a test request, or an evaluation request. However, it usually means I test it myself or assign it to one or more of my subordinates. We’ll be checking its current state and find out whether it influences anything else in the game (repairing one bug can actually result in more bugs). Ideally, we test it and then immediately close the issue (in some cases add it to our checklist for future builds). If we find new problems, we discuss the issue with the responsible developers and return it to them to be fixed. And then the cycle starts again.

You’re playing Arma 3 day in day out; how do you keep a fresh and objective eye?

I don’t think it can be avoided entirely, but what I try to do is find time for myself, I look at games from other developers, recent or past (but popular) game titles and compare them to our product. Also, I always speak my mind. I’m here to tell uncomfortable things to my colleagues and question their decisions. You may say it’s kind of like being devil's advocate.

Past Arma games have been known to be flawed at initial release. What is being done to ensure a more stable launch for Arma 3?

Well, in the past there was a lack of QA in several ways. Let’s take Arma 2 as an example. The first issue was manpower - the QA team back then (two people) could never test all that was needed.

Also, we hired an external contractor company, but they turned out to be too unreliable. The testers at that company were paid by the amount of bugs they found (with a daily maximum) and this resulted in a lot of duplicates and incoherent reproductions. Sometimes they were even reporting correct behavior as a bug. It wasted a lot of everyone’s time. All in all, we learned a lot from these past experiences and we are doing many things differently for Arma 3.

For starters, we now use more people with experience in commercial testing and playing Arma games. In Brno, the place where I work, I’ve picked the people in the QA team myself. Also, we co-operate with testers in the Prague offices. I visit them from time to time to give new briefings and we communicate on daily bases over the Internet. It’s a significant improvement.

[thumb class="pull-left" clear-xs]report_in_qa_2.jpg[/thumb] Secondly, we occasionally host focus group tests. This really helps us to ensure that the game is still headed in the right direction. Even though these sessions are limited to about once a year and only a few people can attend, we manage to gather a lot of useful information.

Last but not least, and perhaps most important of all, thanks to the Alpha participants we were able to track so many issues we would not be able to locate in any other way (e.g. massive multiplayer games, interfering third party software, and various hardware problems). It also enabled us to connect with players directly and see what the community really wants.

Hello there, Mr Feedback

So we’ve learned a little bit about the workflow, but so far it has mostly been about game bugs and errors. What about feedback on design decisions?

Yes, we do this as well. Either if we’re asked to, or, if we find a certain design decision to be wrong on our own. In the first scenario it usually means picking from several options and choosing the best one. In other cases, a dialogue with the design department is opened and the issue is discussed. There is not always an ideal solution, but there should always be an acceptable one.

Sometimes you guys must have some different opinions; is there ever a Clash of the Developers?

This can be difficult, but I agree with our Project Lead Joris-Jan van 't Land that QA should have the right to veto anything they find to be completely wrong. In the end, if I’m asked to decide, I always try to look at it from the perspective of a paying customer. If I have doubts, it simply means that it’s not ‘good enough’. But whenever there’s an argument within the team regarding certain decisions, we always try to maintain rational thought and look at the various options. So far we have always managed to find appropriate solutions.

Another responsibility of your department is to make sure the appropriate level of simulation remains intact. But how do you know what it feels like to be a soldier, drive a vehicle, or fly a chopper?

Well, can anyone beside soldiers/ drivers/ pilots really know? I personally don’t think so. I did play airsoft for some time, but I wouldn’t dare compare it to what actual soldiers experience. Nevertheless, when I started to form my QA team, I did search for people who have at least some experience with other games, driving and ranged weapons. However, as I said, this can never be enough. That is why I am truly grateful for our consultants, who are in active service in various armies around the world. They provide valuable feedback on things like weapon handling and vehicle behavior.

Going from A(lpha) to B(eta)

Let’s move to another major topic: you could say that QA gained thousands of external testers through the Arma 3 Alpha (and later Beta). How are you using people’s feedback?

First I would like to thank all of you who contributed to the Feedback Tracker. As I mentioned before, it gives us great deal of valuable information. Our admins and my co-developers keep a list of all issues and we try to resolve as many as possible. QA prioritizes crashes and blockers (something that prevents you from completing a mission) and then each developer has their own priorities. Unfortunately, our resources are limited, which is why we’re also not able to accommodate everyone's requests.

What things could still be changed and what cannot?

[thumb class="pull-left" clear-xs]report_in_qa_4.jpg[/thumb] This is a better question for our designers, but there are indeed engine, manpower and time limitations. Certain aspects can still be changed, but other things would have to be dropped.

I would say that any new engine feature is unlikely at this stage – at least until the full game has been released. However, somewhat smaller, polishing changes are all still supported.

Are there any differences between the Alpha and Beta in terms of QA/ feedback? What are you looking for?

The Alpha was our first release and there was a lot of pressure. It was our first moment of truth where we really showed Arma 3 to its fans. It’s been great, but also demanding. After the initial Alpha release, we did get more used to releasing regular updates, and the Beta doesn’t feel as ‘scary’ anymore. Don’t get me wrong though, it’s still a very big milestone. However, this time our pipelines are tried-and-tested.

Featuring new features

There are several new features in Arma 3. but there are a lot more. Can you tell us about some of the new features and how you’ve had a hand in their development?

The new transmissions were reviewed by me. In order to do this properly, I actually went for a ride in a real tank - just to experience its handling. To my surprise, it was incredibly agile.

Also, the new recoil (weapon imprecision) was chosen by me personally. I know it’s not for everyone, but using the MG in a Rambo style way is just not realistic.

There’s a lot more of course, and as you can imagine, there have been many long internal discussions (the inventory system, artillery computer application, and diving itself). Basically, we (QA) are included in most decisions.

General questions

And after a long day of making games, what do you like to do to relax and unwind?

You mean apart from spending time with my girlfriend? It really depends on my mood and level of exhaustion. I like sci-fi TV series, wargaming, driving, reading, and of course, gaming.

What do you have to do to become a QA tester and what skills are required?

I don’t have an exact how-to manual, but always make sure you know what you want to do, and where you want to work. You need to be self-aware during the job interview, as I’ll be asking these kinds of questions.

Naturally, you also have to understand the purpose of the product you wish to test. Perhaps even have a hobby or experience that is relevant to the particular game. This is not a must, but it helps. If you had to choose between guys that scored the same on the interview, but one of them plays airsoft, is interested in the Cold War and the Second World War, while the other one sings karaoke and cooks, who would you pick?

Lastly, basics of scripting are always welcome, since testing is not just about playing the game. And very importantly – you have to be perceptive, thorough and genuinely interested.

In an event of biblical proportions, a plague of bugs has ravaged the earth - infecting every video game alive. As Prime Minister of QAtar, you order a special taskforce to heal at least one. Which game would it be?

Are you aware of the fact that majority of developers of Arma are Arma fans? ;-) But if I was granted a second attempt, I would send them to heal ‘Dungeon Keeper’. I spent a lot of time with it when I was younger. It’s really a ‘cute’ game, where you build your own dungeon and have a Horned Reaper at your disposal. It was really good to be bad.