Generic Console development conference (Video+Transcript)

I've decided to open another thread (hopefully It's OK), since the other one was closed and I need a place to post the transcript. Let's try to keep it clean and civil this time. Without further ado:

VIDEO



TRANSCRIPT

Hi everyone, and welcome to this chat, all about the behind-the-scenes stuff of Assetto Corsa, a racing game developed by Kunos Simulazioni.

I'm Alessandro Piva, he's Fabrizio Brugnaro and we're Kunos' programmers.

Here's a brief summary *starts mentioning everything shown in the slide* (background, console porting, first port, performance, multithreading, assets conversion, build system, middleware, submission, GUI, new features, TRC, Testing, QA team and conclusions).

First of all, I'm Alessandro Piva.
I've started programming videogames back in the Commodore-64 era, when I was very young.
I've always participated in computer science competitions (Olympics, ACM, TopCoder).
I've obtained my triennial degree at the "La Sapienza" University, even if I started my academic path here, at "Rome 3" (note: that's where the conference was held). I moved over to "La Sapienza" because a computer-science degree wasn't available at "Rome 3". Then I worked for three years (2009-2012) at Rockstar North, Edinburgh, and then I started working for Kunos in 2014.

BACKGROUND

Kunos Simulazioni was founded by Marco Massarutto and Stefano Casillo in 2005 and It's alwasy been economically self-sustained. It wasn't a company who looked for venture capitalists' money, editors and such. It started from the bottom, with the founders' savings and, during the first years, It worked extensively on professional simulators, hardware and software systems, for manufacturers like Ferrari and Dallara, which are used and tested by their drivers.

Our team is made of about 25 people, but the core team is smaller, about 15 people.
Some times, we do a few Skype calls with other 7-8 guys. That's "the core of the core".
A small company has its pros: less burocracy and talking between us is much easier.

Assetto Corsa is a sim racing game, published on Steam "Early Access" in 2013.
The 1.0 build was released in December 2014 and, in this time frame, It grew a lot.
Now we're in March 2016 (note: pretty sure he meant 2017), I don't know if the 1.14 build is out yet, we're working on it. Current version features about 150 cars, 20 tracks.

Even if It's an indie game, much smaller compared to heavy-weights like Forza or GT, It was able to achieve some important things. It was the first PC game to feature a laserscanned Nordschleife, a very famous track. Do you know what laserscanning is? It's a technology used to create 3D scannings, which are used by our graphic artists to accurately build in-game circuits.

Assetto Corsa was born on PC, founders never thought about a console-port at the beginning. Then, in 2014, the interest in bringing Assetto to consoles sparked, without wanting to make it "easier", without simplifying it to reduce realism, that was our goal.

We kept supporting the PC version of the game. Console development must not slow it down or stop it. During the development process, we had to face challenges that were completely new to us: the submission process, for example but we'll talk about it later, or the quality assurance, a team of testers who play and test Assetto full-time.

CONSOLE PORTING AND FIRST PORT

Porting started, of course, with the base PC code, which made our work easier, since It's very small, about 80'000 lines of code, even smaller compared to some libraries, like Scaleform's. Our code is written in C++11, quite "portable". The non-portable parts, like graphics, inputs and window systems, for those we built an "abstraction layer". So we created a "window class", an "input class" and so on.

We proceeded step by step, without rushing to have everything ready at the early stages. The first working build on consoles was made in December 2014. There was no audio, no GUI, no networking, low framerate, lots of glitches due to wrong texture conversions, a bit unplayable because we did an incomplete port of the code necessary to handle inputs. The game started straight into a race, without menus or speedometer.

PERFORMANCE & MULTI-THREADING

Framerate was around 10-20 fps, in the best scenarios.
Performance was the elephant in the room. At the beginning, It was really poor. We're talking about 10-15 frames per seconds, even with just a few cars on the track: two, three or four. We took care of the most obvious performance issues at first, but the game was still running poorly, 20 frames at best, while achieving 60 FPS was our goal.

PS4 and Xbox One have a good performance level on the GPU-side. Graphics card is quite good. Problem lies in the CPU because both platforms feature an AMD processor, built around a mobile architecture. Sony told us that there's a 50% performance gap between console CPU and same-clock, desktop Intel CPUs. This is a very negative thing, even worse if you consider the low clock speeds of those processors (1.6 GHz).
And I'm not talking about PS4 Pro and Scorpio, I'm just talking about the base models.

So, we had to perform a quite deep "engine refactoring". Initially, before starting the console port, Assetto's code was built on two monolithic threads. One for the physics, elaborating one frame every 4 milliseconds (fixed 250Hz) and one graphic thread, built to go as fast as possible, while, of course, keeping in mind the GPU restrictions. The faster the GPU, the faster the thread. So, physics at 4 ms, graphics even at 16 ms, so 60 frames per second. Faster if your PC has a more powerful GPU.

The rendering code was very "OpenGL-style". There's an "object-tree" and each object is linked to a "render function". Each function makes its "drawing-calls" and the whole thing is strictly sequential, based on virtual functions (note: opposite of "data-oriented", but he wasn't able to provide a specific term). The engine architecture was very sequential.

But consoles' systems have their strenghts too. First thing, many cores available. On both consoles, we have six cores at our disposal. Then, the system's "primitives" (like syscalls, draw-calls, "traffic lights") and the graphics API both have a very low "overhead level".

Multi-threading is the absolute key to take advantage of CPU cores on PS4 and Xbox. As much as possible. So, we started to do an engine-refactoring, to make it "more multi-threaded". For example, we modified the physics engine, so that every car's step is done on separate threads. 16 cars? 16 different steps and each one of those steps apply different forces to the car's components. Then there's a big integration step, we use a physics-library called ODE for this, which apply every single force, like collisions, and at the end of this big step, we have the new positions for every car and its components. Making this step multi-threaded is very difficult, so we focused on the first ones. So, if we have 3 cores dedicated to the physics, every core is going to take care of 4-5 cars and things like that.

Then we also did an entire refactoring for the graphics engine as well, to make it "data-oriented". We don't have an object-hierachy that has virtual render functions. We still pass through the "grapho" of every object on the scene, but this "passing" creates a list of rendereable objects. It's a list, a data structure and for every thread, we scroll through this list to build the different "deferred-contexts". What's a deferred context?
When someone needs to render a graphic frame, he builds a list of draw commands that will be sent to the graphics card. The VGA performs them all, one by one. The deferred context is practically a mechanism to create sub-lists. There's the main list and It can be divided into different sub-lists: one for the shadows, one for the post-processing, one of the game's main geometry and so on. Those lists can be performed in separate threads. Again, multi-threading is essential. Different draw lists for shadows, reflections etc. on separate threads.

We also implemented a highly-flexible job system. Different threads, different jobs but still syncronized with each other thanks to traffic lights and things like that. A job system is something like that. You build n-threads, which are always listening and ready to perform different actions, different jobs with pre-defined inputs and outputs. There's a queue on the main thread, with a list of jobs and the n-threads, one by one, pull a job out of the queue and perform it. It's a very flexible system, because you can send those background threads, let's say, 10 rendering jobs, 4 audio jobs and one for the physics and the first thread takes care of one thing, then the next one does the same thing and so on. We don't have predefined threads because a flexible approach is better.

NOTE: He briefly describes what's shown on the slides. He talks about physics and graphics frames, car's steps mentioned above, cores usage (3 for physics, 2 for graphics, 1 for audio) and the jobs system mentioned above. More or less CPU cores simply make the whole thing faster or slower. More cores can process more threads, which can pull different jobs out of the queue in less time.

ASSETS

Besides performance, which was the biggest issue, there was another problem, an "engineering" one: the assets, the content. I mean, models, textures, sounds of the PC version, which needed to be converted or pre-processed for the other platforms. First of all, some times, the formats are simply different. For example, .DDS textures on PC, .GNF textures on PS4. Why different formats? Because Sony and Microsoft, who build and offer their SDKs for their consoles, also offer tools who try to optimize assets for their own platform, their hardware architecture. For example, about .GNF textures: the pixels don't follow the traditional order (row, row, row...column, column, column), but they're organized and arranged on a Peano's curve. This allows a better use of the system's cache.

In some cases, assets need to be generated automatically. Atlas textures, for example. Atlas are big textures, meant to include smaller texture groups, to reduce the numbers of textures' exchanges for the rendering phases. An automatic conversion process means that our artists don't need to manually convert textures or models. If someone finds a bug in the conversions' tools, and, of course, It has already happened, he can simply delete the glitched assets and rebuild them from scratch, automatically.
This can be done on a PC, remotely, because converting an entire game can take lots of hours.

BUILD SYSTEM

To convert assets, we created a build system, based on Ninja. What's Ninja? Ninja is a tool inspired by Make, which is a program used to build software or assets. What I mean is that, every "Make file" is a list of rules like "If this file has changed, then build me this other file with this command ". Every time you code, in C++, Java etc., you're creating a "Make file", like "if the source code has changed, use the compiler to create the executable".

Make is classic program within the Unix world and Ninja is very similar yet more focused on being fast and simple to use. A Ninja file, It's a list of rules...input --> command --> output and if Ninja sees that some files changed, then It generates the necessary commands to build different outputs accordingly.
If the input stays the same, then Ninja won't create the output file.
Ninja is very "scalable" and quite performing even with builds made of 40'000 files. There's this gigantic Ninja file, with all the game's textures and It can see which files have changed and It automatically converts them. Even with 40'000 files, as I said, It's still very quick.

Besides Ninja, the tool we used the most was Python. We created a dozen of Python scripts, to handle assets' conversion and pre-processing. We used Python's "Argparse" module extensively and things like opening images, generating texts, building processes became extremely easy.

If we look back, maybe using C# would have been a better idea, because strong-typing code languages could be easier to mantain and preserve while going forward, but that's just my personal opinion.
We also did stick to the Ninja+Python combination to generate the master disc and the different submission builds and packages.

MIDDLEWARES

Then we used some middlewares. As I said, they're big libraries which offer many features and functionalities to be used within the game. GUI: Scaleform, Audio: FMOD, Post-processing: Yebis. A million features, they try to do everything you could possibly think of. They can also be quite buggy. Sooner or later, you'll find a bug, which is more painful than finding a glitch in your own source code. You'll need to wait for "them" (note: software companies) to fix them, if you don't have access to their code.

Going forward, we'll evaluate again the necessity to use this kind of software and if they're still worth the trouble or not. FMOD and Yebis were worth it, Scaleform wasn't.

SUBMISSION

So, let's talk about submission. If you want to publish a game on consoles, your game needs to comply with a list of TRC (Tech Requirements Checklist), an extremely long list of technical and functional requirements, Sony and Microsoft want your game to comply with. STRICTLY. Otherwise your game cannot be released. Some of them are quite trivial. For example, your game needs a "pause" option. Others require deep architectural changes. For example, multi-users feature on Xbox. Then network protocols' cryptography. Microsoft asks for that. Or background game's installation. You need an initial part of the game ready to be played while the game is being installed.
Submission's timings are quite different compared to PC and Steam. You need to book a "slot", then create and upload your build, get the documentation ready, in different languages and so on.
The sad truth is that you really need someone, within the dev team or on the publisher side, to handle this submission process. It's extremely complex and we wouldn't have been able to do that alone.

Thank you for you time.

END OF ALESSANDRO'S PART

****************************

FABRIZIO TAKES THE MIC

Hi everyone, I'm Fabrizio, programmer at Kunos Simulazioni.
I'll try to quicken my pace, since my colleague took too much time *laughs*

Since I was a little kid, I've always played videgames. When I got to university, I decided to dedicate my career to this field. After my triennial degree in computer engineering, here at Rome 3, I had the chance to meet, here, someone who used to work for Kunos.
Since 2014, I worked for Kunos, handling the console porting.

I'll talk about UI porting, the biggest challenge, new features required by TRC and then briefly about testing and quality assurance.

GUI (Graphics User Interface)

The necessity to build a new interface was extremely clear. The one on PC was meant to be used with a mouse and that wasn't simply possible on consoles, where controllers are the main peripherals. Besides that, we had two .exes, one for the main menu, the other for the game. The main menu allows users to define some settings that will be shared and used by the other .exe. That's not acceptable on consoles, there must be just one .exe.

We evaluated two options: building a new UI editor, internally or using an existing middleware like Scaleform. We went for Scaleform. A new editor developed by us would have increased costs and time, that wasn't doable. Scaleform uses Adobe Flash technology, so Flash and Action Script knowledge was required. One progammer took care of Flash while I implemented Scaleform and all its functionalities into the game. That and the logic that allowed Flash to interact with the game itself.

We wanted to offer console players an experience as close as possible to the one on PC. But, the old PC interface wasn't built with consoles in mind. It features graphical elements, like sliders, buttons and drop-downs, which manage many different gameplay functionalities. We looked for all of these fuctions in the source code, extracted them, translated them and made them usable via a Flash-based interface. They were all over the place and that required lots of time and efforts. Some PC apps needed to be completely rewritten and adapted, when possible.

The game was supposed to be released in many countries. Many things needed to be translated and the built-in Scaleform translator became quite useful.

NEW FEATURES AND TRC (Technical Requirements Checklist)

New features were added on Sony's and Microsoft's request. Save systems, achievements, trophies and a few online services. The PC's save system was simple: just a few files in the game's folder, containing all the necessary informations.

Consoles have their own, peculiar systems. Cannot talk about them due to ND agreements. Let's say they're different and brought some restrictions. A few key parts of the save files, like replays, ghosts and setups were treated differently, to comply with Sony's and Microsoft's standards (note: slide says number of files and their sizes played a role in all of this. Fabrizio doesn't mention anything though).

TROPHIES & ACHIEVEMENTS

Trophies and achievements were completely new to the game, absent from the PC version. On one platform, It was easy, on the other, a long, time-consuming battle, but all in all a fun process to come up with challenges to keep players engaged. To unlock the most difficult ones, you need to be really, really, really good. Someone made it and I'm happy about that.

ONLINE SERVICES & MULTIPLAYER

In regards to online services, that was a real pain in the arse. Sony and Microsft have their rules and necessary checks, like Xbox Gold or PS+, if the user is banned or not, if he/she meets the age requirements, if the software is up-to-date etc. etc. etc. Both Sony and MS are very scrupulous and meticulous and they send tons of stuff back to you when It doesn't meet the necessary requirements.

On PC, you get a list of servers, you join one and you're good to go.
On consoles, you need one more step. Before joining a server, the player needs to be part of a "game session". Everything is transparent, user isn't even aware of this. This session is required due to online services and functions like inviting other players, voice chats etc. Those TRC cannot be ignored. They're essential.

QA (Quality Assurance) Team

QA team was something new as well for us. On PC, we don't have a dedicated team like this.
There's a small group of sim racing enthusiasts, who test the game for us. On consoles, you need a bigger QA team who knows the TRC very well. For every TRC, Sony and MS release a huge list of test-cases, all of them need to be checked and verified during the submission phase. The amount of work is huge and the QA team is, therefore, absolutely essential.

There was a trial stage, where we had to define the methods and the tools to collaborate with this QA team. After a while, everything became more fluid and easier to handle.
The QA team focused a lot on TRC, which is necessary but It wasn't as on PC, where enthusiasts pay attention to gameplay aspects, bugs and glitches. Unfortunately, the QA testers on consoles didn't have this kind of experience, some issues were spotted later.

CONCLUSIONS

The porting's main goal was realism. Physics needs to be as real as possible, on PC as on consoles.
It was really a huge challenge, undertaken by a very small team.
Main issue on technical side? Performance, definitely.
Main issue on non-technical side? TRC, submission and burocracy.

Thank you for your time and attention, my time's over.
-----------------------------------------------------------------------------------------------------------------

 

Nice post

 

This was a fascinating read. It's always interesting to read developers' take on moving from a PC environment, where you have extremely fast single-threaded performance to consoles, where you have very slow (relatively speaking) single-threaded performance. Many years ago, CPU engineers predicted this day would come. They watched as Moore's Law marched along, and they watched as the software industry assumed that single-threaded performance would track the same scale (it didn't). We're finally seeing a slowdown of progress, even on the PC side. This means that the lessons learned in console development will soon be a necessity on the PC side as well.

Their flexible threaded worker model sounds very cool. Using flexible threads that can do any job pretty much eliminates the task of queue monitoring and tuning. You never have to ask, do I have enough audio threads? Could I drop an audio thread and add a physics rendering thread?

Cool stuff, thanks for posting!

 

Very interesting read. Thanks!

 

Good read. Nice insight.

 

Best thread on forum!! Amazing stuff!!


Sent from my iPhone using Tapatalk

 

I love reading stuff like this