Next up in our latest series of Report In! dev interviews is Programming Lead Ondrej Martinak. Heading one of the most complex tasks in games development, Ondrej explains what it means to be a programmer on Arma 3, talks about some of the most significant challenges, and hints at what the Arma 3 Programming department will be working on in the upcoming year.
We're often asked to let some of our people talk about what it's like to develop a game at BI. 'Report In!' gives you a more personal perspective on our team, a more detailed look at the way we go about our work, and fresh information about our ongoing projects.
Introduction
Please tell the people a little about yourself. What's your role? How long have you been with Bohemia Interactive? Which projects have you contributed to and what is your favorite BI game or mission?
Hey, my name is Ondřej Martinák and I'm one of the Programming Leads on Arma 3. I joined Bohemia Interactive back in 2012 when the game was on its way to the public alpha release. I guess it won't be very surprising when I say that my favorite BI game is Arma 3. I love its visuals, and playing multiplayer missions with my friends and colleagues can provide some truly memorable moments. I usually like to play more action-packed missions, so I would probably choose Seize Feres or Seize Edoris from the official content selection, or Invade & Annex which is created by the community mission-makers, as my favorite scenarios.
And can you give us a random fact about yourself?
Contrary to popular belief, programmers are not machines that transform coffee into code. I actually start (and continue and end) every day with a cup of green tea. I don't really feel the need to infuse my body with a high amount of caffeine to get it running. True, green tea contains some caffeine as well and some people feel an influx of energy after drinking it, but it's not so taxing on the body and it doesn't affect the sleep regime as much. It certainly does not affect mine as I can drink a whole teapot and sleep happily half an hour later.
I usually drink jasmine tea during the work day as there is not much time to focus on proper preparation, but when I want to enjoy something richer I go to a local teahouse where they serve exquisite Japanese green tea - be it Gyokuro, Tamaryokucha or Sencha or the powdered tea Matcha which is usually used for the tea ceremony.
Code Magic
Programming seems incredibly complex to most of us, and somehow stands at the center of everything when it comes to software – including games. Can you explain a little bit about what programming actually is? And can you give a few concrete examples related to Arma 3?
Programming is a process of instructing a device (a computer in our case) to do certain tasks. In the case of Arma 3, these tasks might be as easy as checking whether the player has clicked a button in the main menu, or as complex as rendering a terrain with a 12 km view distance and thousands of objects all around.
Programs or games are written in programming languages that are specifically designed to simplify the process while providing features of the underlying device. There are many different types of languages as each one of them is trying to solve a different problem. Some languages aim to provide access to the lowest levels of the device and expose maximum of its performance while other languages offer a higher level of abstraction and faster development cycles.
What languages are being used for Arma 3?
The game and engine is written in C++ with bits of assembler here and there. This is not really surprising as C++ is the most popular choice for AAA games. Our newer tools and Launcher are written using C#/.NET as the performance there is not critical and C# with its underlying platform allows for fast development cycles.
We also use Python or Linux shell scripts to automate our building pipeline, data delivery or for creating small specialized tools that help with development and data analysis.
What does your typical day at the office look like?
First, I collect data that has accumulated since the previous evening (tasks, bug reports, new commits in the source code repository, e-mails, IMs, forums or twitter posts) and create a plan for the day outlining all the work I need to do.
The work itself is usually split into several parts. I check the progress on tasks that my department is currently working on. If there's some problem that blocks or slows down the development, I make sure It's gone as soon as possible. I rearrange the tasks if it's necessary to do some dependencies first so that we meet our deadlines efficiently and without delays. I also review the work when it's done and check that it's delivered to QA for testing.
Most of the work we do is cross-disciplinary, meaning there's more than one department involved in the development. It's my responsibility to ensure that we co-operate with others and no work is left in-between. For example, when we were prototyping weapon bipods, the first thing we did was to create a hardcoded animation of the bipod deployment. Then, our artists prepared a testing weapon with a new bipod model for us, which we have used to implement a generic deployment mechanism. When the implementation was done, we have delivered it to encoders to prepare the data for all other weapons.
When I make sure everything in the department is running smoothly, I write code myself. I just pick a task from the top of the current milestone's backlog as any other programmer would do, analyze the issue at hand and hack on it until it's fixed or implemented. This also includes doing research of current trends and technologies that we might utilize to further improve the game.
And how do you collaborate on the same code (e.g. version software, peer code reviews)?
We have a bunch of programmers spread across several studios who collaborate on the same codebase. So good organization is necessary. We use Subversion to version our code which is a pretty standard solution these days as it offers a simple centralized architecture that is easy to manage. We use several branches for the development. Our main dev branch is used for common day-to-day work and bug fixes and it's the branch that is used to create dev builds that are uploaded to Steam every day. We have special branches for stable and profile builds and when we are working on larger and potentially dangerous features, like a certain not-yet-announced gear mechanic that will simplify weapon manipulation, we use separate branches as well.
Apart from the programmers talking to each other regularly, we use Crucible for doing peer code reviews. Crucible is part of the Jira suite that we use to power our information system and it integrates nicely with our bug tracker, source code repository, and automated build system. Code reviews are a really great tool that encourage learning among team members and help us avoid many mistakes and bugs before the commits even reach QA.
The last line of defense is held by our automated build system. We create regular internal builds every hour that check for possible errors in the source code and run a limited set of automated tests. If a build fails responsible developers are notified immediately. Our nightly builds then run an extended set of automated tests that check correct behavior of the game as a whole.
Do you work with hardware partners, and what does it do for us / them?
Yes, we do work with hardware partners. Arma 3 uses PhysX for its physics simulation and NVIDIA has been most helpful when we have encountered a problem that we were not able to solve on our own. With AMD, we have closely co-operated on graphics optimizations and new rendering features. One of the results of this co-operation is the integration of a more efficient ambient occlusion algorithm called HDAO. We have also worked with Intel on further improving our graphics renderer and optimizing it both for integrated and dedicated graphics cards. Thanks to this co-operation, we now offer a better post-process anti-aliasing algorithm called CMAA.
We have also received samples of Steam Machines from Valve and Oculus Rift from Oculus VR and we are analyzing how best we can utilize these technologies in our future games.
And what aspects of your work do you find the most challenging? And what makes it the most rewarding?
The most challenging thing about coding Arma 3 is managing its complexity. The engine and especially the game has been evolving for many years now and it contains a lot of features that interact with each other. So whenever we are implementing a new feature, we have to make sure it interacts with all other parts of the game correctly, it doesn't break existing behavior, we have to make sure the AI understands the feature as well, and we need to ensure proper network synchronization. When all of this is done, the feature has to be tested and perhaps iterated so that a desired level of quality is achieved. All of this can make even a small change in the game a rather time consuming task.
On the other hand, when we do all of that properly and the feature is released in a public build it gives us great joy to see players enjoying it. Positive feedback from the people that play the game or invest their time in creating content for the game is one of the most rewarding elements in game development.
Tuning the engine
Although game engines are frequently discussed, there are probably still many people who do not exactly know what an engine does, and how it works. Can you give us a quick introduction?
A game engine sits at the core of each game, be it small indie games or huge AAA games that take hundreds of people to develop. Its purpose is to provide basic systems that interact with the operating system or computer (or console) hardware to provide games with the ability to load and cache files, render graphics, play sounds or send simulation data across a network. Game engines also often contain more specific structures like a representation of the game world and all its entities, character animations or artificial intelligence algorithms. Games are built on top of game engines as it is easier to use technology that is already prepared and thoroughly tested.
Some game engines are very simple and are essentially combined with the game itself. Other engines might be huge and complex beasts specialized in a narrow game genre. These engines often provide efficient tools for creating games of the genre they were designed for and offer a lot of common code that will speed up development of the game. The disadvantage is that it is usually difficult to use them for games of different genres.
This is covered by another group of game engines that are designed to be generic and can be used for creating all sorts of games. They typically support all the major desktop or mobile platforms to cover a wide variety of players. These engines are then licensed to individual game developers who can focus on creating game content instead of reinventing all the technologies from the ground up.
Arma 3 uses the latest generation of Bohemia Interactive’s in-house built Real Virtuality engine. Can you tell us how it compares to other game engines out there? What are its strengths, and what are its weaknesses?
Real Virtuality falls into the category of specialized game engines as it is very good for creating shooter games with huge open terrains. All games built with the RV engine feature open worlds, vehicles with proper physics simulation, or big multiplayer modes. Additionally, the engine itself is designed to be very open to modding so it is not only possible to create new weapons, vehicles, missions or maps, but total modifications as well. In the field of authentic military shooter games there are not many competitors that can rival Real Virtuality.
But as the engine is specialized the way it is, it wouldn't be easy to use it say for linear corridor-based games or games from a completely different genre.
And what third-party libraries are used, and what are the upsides / downsides?
We always carefully consider whether to use a third-party library or write our own solution. If you write the code in-house you can make it fit your engine or game perfectly, you always know what exactly it is doing and it's much easier to debug. On the other hand, using a third-party solution can save a lot of development time and the more widespread the library is the less bugs it contains. Sometimes, integrating the library into your environment can be difficult though and if you need to do that regularly to upgrade to new versions, it can become rather painful. There's also a big difference between open and closed source libraries as it is much easier to debug software where you can see all of the implementation details.
Real Virtuality uses several third-party and mostly open source libraries for some specific tasks like data compression/decompression, encryption and others. The biggest third-party libraries we use would be PhysX which powers our physics simulation, RotorLib that provides authentic flight simulation, volumetric clouds are brought by SimulWeather, and Steam is used both for distribution and social interaction. You can find a complete list of the libraries we use in the Field Manual inside Arma 3.
A recurring discussion topic among Arma 3 players is optimization and performance – particularly with regards to framerate. Can you explain a bit more about how this works for Arma 3, what can be improved and what’s being done - and perhaps if there’s anything players themselves could look out for?
As most of our players know, Arma 3 can look beautiful but at the same time it can be difficult to maintain a steady framerate - especially in multiplayer missions with lots of players and AI. It's a known problem for quite some time and we're continuously working on optimizations that aim to remedy the situation. One thing to keep in mind is that the performance gains come in regularly but in small steps which might be hard to see. When we look at them from a bigger perspective though, we can see that we have increased Arma 3's performance by approximately 35% in singleplayer missions and 35-40% in multiplayer missions since the game's release (these numbers can vary mission to mission of course).
To make it more difficult, Arma 3 is a sandbox game where a large portion of the game content is provided by community in the form of custom mods, missions or even maps. That means that, we as developers, must make the platform as fast as we can, but after that it is always up to the community authors to optimize their creations. The best thing we can do to help in this is to provide guidance in the form of documentation and profiling tools.
The topic of performance optimization is not a small one and we hope to cover it in more detail in a dedicated OPREP later.
The Future
Besides improvements and optimizations to existing tech, what can you tell us about new upcoming (Marksmen) features? From a programming perspective, how are these being developed?
Every new feature follows basically the same process. First, we take a design analysis (and often a scripted prototype) of the feature, and provide a feasibility feedback that would say if the feature can be implemented in our engine at all, if it needs to be simplified in some way, or if we can actually expand it. Next, we create a technical analysis which outlines the implementation, describes its architecture, and provides time estimates.
After all of these preparation tasks are done, we write code for the feature, prepare an internal build showcasing it, and discuss results with stakeholders. We continue iterating in this way until we reach a desired level of quality. The iterations are usually done in a matter of days which allows us to quickly react to bugs or changing requirements. This iteration phase might also include taking feedback from our players via our daily dev branch builds and forums or Feedback Tracker.
When the feature is deemed complete, it is delivered to our Quality Assurance (QA) Department for testing. This might again result in several iterations making sure there are no bugs and that the feature doesn't negatively impact other parts of the game.
Bipods?!
Bipods are one of the larger features we're working on for the next DLC, and it presents a nice challenge for our animation programmers. It is a mechanic that is not covered in many games and if it is, it's usually somewhat limited. Our take on this new weapon handling is coming together rather nicely and already promises interesting gameplay opportunities. I won't go into much detail here as all the juicy stuff will be covered by responsible developers in one of the upcoming OPREPs.
What about other features such as the launcher and the Steam Workshop integration?
Steam Workshop and accompanying tools like AddonBuilder, Publisher or Launcher are very important for Arma 3 because they simplify user content creation, sharing, and loading into the game. That is essentially the backbone of our modding support. The tools already support game addons but our plan is to take it much further.
We are currently working on extending addons that can contain only one file into larger mods that can contain many other files inside. This should allow community authors to use Steam Workshop for all of their content. The feature is in the last phase of development and should arrive in dev branch soon.
We also have plans for some rather unique features that should further motivate content creators to work on new game modifications, but for now we will keep them secret.
General Questions
After a long day of making games, what do you like to do to relax and unwind?
For me as a game developer, programming is not only work but also a passion. So when I get home I actually often fire up Visual Studio and write some more code. It usually means just playing with new technologies, trying different approaches, and prototyping various ideas.
But I think it won't be very surprising when I say that I love both creating and playing games. When I'm too tired to use my brain anymore, or I simply want to have some good old interactive fun, I play games - be it PC or console games or sometimes even the mobile sort. I play almost anything but a good shooter or RPG is always welcome.
In the rare time when I'm not sitting behind a computer, I like to go for a run. I never liked running when I was younger and I hated it in the physical education at school, but for some reason I really enjoy it now. Most people say that running is really good to clear your mind and let go of daily stress, but for me the one or two hours on the trail is a great opportunity to think about everything.
What advice would you give to people who want to become a (games) programmer?
As a programmer you have to learn a lot of theoretical background, of course. Be it computer architecture, specifics of different CPUs and GPUs, various programming languages, math, physics, etcetera. These you can learn at schools, in books, on the Internet or from your more experienced peers.
But a programmer is a creative profession like any other (except that you will be judged both by the means and results) which means you'll have to actually write code to learn anything. So be sure to write a ton of code and when you think you can't do it anymore write some more. It takes time to learn various idioms and patterns used in different languages and domains (especially if they are as complex as C++) and even more time to develop a clean style that will lead to efficient and maintainable code. Don't worry if it's not perfect, yet. It will be one day :).
There's one thing even more important than writing code though. Reading code. Not reading about code or how to code, but reading actual code. Beginning programmers might be surprised, but we spend much more time reading code and understanding it, than writing it. So go check out the interwebs, there are a lot of open sourced projects, games and game engines that you can learn from.
Somewhere in the glorious Nation of Optimization, an evil genius has enslaved a tribe of tea-fueled code monkeys – forcing them to develop a special piece of software that would kill every game engine in existence. With a limited supply of programming mana at your disposal, you manage to create a shield that will protect one game from destruction. Which would it be?
I would probably save Minecraft because it can serve as a platform for creating other games and thus providing us all with a glimmer of hope.