Hello dear biters and related species from unexplored planet full of life and natural resources. Recently I have been working on several high resolution graphics for your best friends - the tank and the player character. In this article I would like to show their updated visuals to you, as well as a sneak peek at how they are produced. The following text may contain traces of automation.
High Resolution Tank
Like most biter stories end, this story started with the tank. I just took the 3D scene that Pavel made in 2014. There was quite a bit of work required to make it all render correctly, but eventually I arrived at the desired result.
However there was one thing that really bothered me in the process - making the colour mask look good was rather impossible.
In general this is an issue that we already encountered earlier with the locomotive, cargo wagon and train stop. The cause was the RGBA values set for player colours, so I went ahead and changed all of them, I even added a “default” colour, so you can easily revert back to the single player default colour in 0.16 (which is also "Orange").
The dynamic colour algorithm (we call it tint) works in very weird ways, where if Alpha is set to 1.0, it kind of multiplies the mask by some selected colour. Alpha 0.0 somehow makes the whole mask sprite draw additively while colorized by the selected colour.
Regardless of the technical magic, the final verdict is that unless you are doing something extremely specific, the colour mask values should always be at 0.5 Alpha. You might have already noticed that the train colour sliders only allow you to change RGB values and not Alpha. That makes the colour much more stable - any RGB value you pick is going to work, once you start touching the Alpha it can look wrong and broken.
I would like to take this opportunity to ask modders to keep this in mind, as I have noticed multiple mods using 1.0 Alpha values in colour mask tint. The Alpha is black magic, keep it at 0.5 please.
Player character colours
The tank and the car colours are both much nicer now, but the player character colours seem a bit off...
The whole mask-works-with-0.5-alpha has one big condition - both the mask and the masked area of the sprite under it have to be desaturated. And the player graphics are already made to be orange. This is especially visible in the image above on colours like gray.This was never an issue since the player colours were using 1.0 Alpha, apart from the problem that all of the non-default colours looked horrible.
It might have been possible to do some quick hacks and just desaturate the player sprites where needed, but we found it more appropriate to take the opportunity to bring the player character into high resolution and fix it along the way.
High resolution player character
We were already aware that the player character is a very special task. Just the amount of spritesheets and Blend scenes already screams madness. Basically, the player character was split across many Blend files that Albert produced in 2015, each of them having one animation sequence. I honestly didn’t even ask how much work and time did it take for him to create all of this.
The game currently uses:
Idle player
Idle player with gun in hand
Mining player without axe
Mining player with axe
Running player
Running player with gun in hand
Player corpse
Each of these animations then have a variation for two armor levels, which can be drawn on top of the basic character. In total we are talking about over 4,000 individual sprites, double that number when we also consider high resolution.
From past experience we already know that re-rendering an entity in high resolution is almost never “just make it have more pixels”. Shaders and details start working differently, and it’s always necessary to do some changes. However, editing 21 Blend files at the same time does not sound very appealing or efficient, so our plan was to merge all of the animation into one huge Blend file. The aim is to have:
Blender file with 21 scenes, each rendering one animation.
Everything must render automatically.
All of the meshes linked so if we edit one, the rest automatically update.
All of the meshes use the same materials so we just edit one.
It must be possible to revisit the player character in the future.
Per usual this is much easier said than done, it is possible with the tools that Blender has, but would require an extreme amount of manual work. It would also take way too much time, and would be way too fragile.
Graphics workflow
Making the player character in high resolution would have been utterly insane without the use of scripting. In the following part of the article I will show you a few of the most useful Blender scripts that I wrote and used in the process.
1. Copy objects between Blender scenes The first task sounds simple - copy-paste the objects from the source Blend file to the new one - until you try to do it. In Blender you can Copy + Paste objects between scenes easily, the catch is that in the destination, they are all pasted to the same layer, depending on which layer is active. However the objects forget which layer they are from in the source scene. This is a big problem, but after short thinking I wrote a new object transferring tool.
It is a set of two scripts, the first one prepares objects in the source scene (it changes their names to have a prefix with layer numbers). The second script processes them in destination by reading the name, moving the object to the appropriate layer(s), and cleaning up the name as if nothing ever happened.
2. Link mesh data of identical objects The copy pasting is great, but every new paste creates a set of new unique objects with unique materials. If you do this 21 times, the amount of duplicate data which is supposed to be the same goes up like a rocket. Generally 3D software has a way to create duplicates which are linked so if you edit one, you also edit the other. However once you break the link, it’s not so easy to bring it back, because the program isn’t sure if the two objects are really the same. In Blender this is possible, but doing it one by one would be way too much work so it’s time for another script.
This one reads every object’s vertices one by one, and saves them to giant lists. It then compares it to another object, and if the other object is identical, it makes a link between them - if not, it keeps both of them unique and adds the new one to the list of vertex data to be compared with.
It’s actually surprisingly fast, considering that it filters through about 5 million vertices, but it still takes several minutes to process.
The result seems good except for one problem - some of the objects have identical mesh data, but special materials that we used for rendering shadows, which is solved by running another script, which searches for objects with the shadow maker material and removes them. This has to be done before linking the mesh data to make sure that our mesh data ends up with the correct materials.
3. Clean material slots The next step is to unify materials. This is already mostly taken care of because the mesh data already carries the used materials in it, but going through all of them and manually checking was required. It wasn’t helping that every object had defined about 20 material slots, even if it was using just a handful of them.
Whenever you have a mesh with multiple materials being used on different polygons, and you remove some polygons, chances are that you aren’t using all of the materials any more.
So I made myself a script which removes the unused material slots so I wouldn’t have to manually check so many of them. It’s not only more work but it’s also much easier to spot errors when you see only what you need to.
4. Generate Render Layers and Compositor Nodes With all of the objects correctly linked and materials properly editable, it’s time to figure out how to output this whole thing into sprites. As we already hinted in FFF-146, we are almost always using Render Layers going into Compositor nodes to output many different passes of the same scene.
Since we are keeping the layer structure of the original source, it was possible to set up the rendering system to do the same again. It was still a lot of work because it was not always consistent and many of the scenes required small but hard to spot tweaks.
One thing which helps at least a little bit, is pre-generating all of the render layers with their names and some basic setup, which is consistent between all scenes.
Render Layers aren’t much without a way to save them to the final images - this is handled by the compositor nodes. There is a special case for Ambient Occlusion and Shadow passes, but in general it’s absolutely consistent so scripting this part is great as it removes all need for manual attention to this. The script just reads the Render Layers and generates the nodes to fit them.
5. Rendering and final processing To render all this from the 3D scene, we are using a fancy system which allows us to render across multiple computers, as the animations would take over 40 hours to render on a single PC.
The rest is basically the same as with the combinators shown in FFF-194, so I won't go into details here.
Result
After all this, we get the player graphics in high resolution, with colour masks working correctly and also a lot of new Blender tools for ourselves to work with.
And here is a preview of how the colour masks work for 0.16.
All the things in this article combined took a lot of time to make, especially the high resolution version of the player character. In the long term we are likely going to do some more changes and adjustments to the player character models and animations. Thanks to the new source files we will be able to do that, and for now we are satisfied with how the player looks for 0.16.
Razer Chroma
Razer contacted us quite a while back and asked us if we could add Razer Chroma support to Factorio. Razer Chroma/Chroma Link means that the game can take control of any supported RGB devices (from mouse and keyboard to LED strips and chairs) and control the colors of the lights.
Damage taken animation (Keyboard, mouse, mousepad, headset, chroma link)
Player Healthbar when damaged (Keyboard, mouse, mousepad)
Player shields when damaged (Keyboard, mouse, mousepad)
Progress-bar during game loading, savegame loading, multiplayer map download multiplayer map loading, multiplayer catching up(Keyboard, mouse, mousepad, headset, chroma link)
Flash technology key on technology researched (Keyboard)
Flash map key when building are destroyed (Keyboard)
Custom Ripple animation when achievement is unlocked (Keyboard, mouse, mousepad, headset, chroma link)
People already want to control their light using the circuit network and Lua, so something like that could be cool to add in the future if there is demand for it.
Normally we wouldn't add features to 0.15 after it was marked stable, but since there will be some Razer events, we thought it's good idea to have Factorio in the list of supported games. So the feature was added today to 0.15 as an experimental update but it will be marked as stable next week if nothing is broken.
If you have anything to say you can let us know on our forum.
Hello dear biters and related species from unexplored planet full of life and natural resources. Recently I have been working on several high resolution graphics for your best friends - the tank and the player character. In this article I would like to show their updated visuals to you, as well as a sneak peek at how they are produced. The following text may contain traces of automation.
High Resolution Tank
Like most biter stories end, this story started with the tank. I just took the 3D scene that Pavel made in 2014. There was quite a bit of work required to make it all render correctly, but eventually I arrived at the desired result.
However there was one thing that really bothered me in the process - making the colour mask look good was rather impossible.
In general this is an issue that we already encountered earlier with the locomotive, cargo wagon and train stop. The cause was the RGBA values set for player colours, so I went ahead and changed all of them, I even added a “default” colour, so you can easily revert back to the single player default colour in 0.16 (which is also "Orange").
The dynamic colour algorithm (we call it tint) works in very weird ways, where if Alpha is set to 1.0, it kind of multiplies the mask by some selected colour. Alpha 0.0 somehow makes the whole mask sprite draw additively while colorized by the selected colour.
Regardless of the technical magic, the final verdict is that unless you are doing something extremely specific, the colour mask values should always be at 0.5 Alpha. You might have already noticed that the train colour sliders only allow you to change RGB values and not Alpha. That makes the colour much more stable - any RGB value you pick is going to work, once you start touching the Alpha it can look wrong and broken.
I would like to take this opportunity to ask modders to keep this in mind, as I have noticed multiple mods using 1.0 Alpha values in colour mask tint. The Alpha is black magic, keep it at 0.5 please.
Player character colours
The tank and the car colours are both much nicer now, but the player character colours seem a bit off...
The whole mask-works-with-0.5-alpha has one big condition - both the mask and the masked area of the sprite under it have to be desaturated. And the player graphics are already made to be orange. This is especially visible in the image above on colours like gray.This was never an issue since the player colours were using 1.0 Alpha, apart from the problem that all of the non-default colours looked horrible.
It might have been possible to do some quick hacks and just desaturate the player sprites where needed, but we found it more appropriate to take the opportunity to bring the player character into high resolution and fix it along the way.
High resolution player character
We were already aware that the player character is a very special task. Just the amount of spritesheets and Blend scenes already screams madness. Basically, the player character was split across many Blend files that Albert produced in 2015, each of them having one animation sequence. I honestly didn’t even ask how much work and time did it take for him to create all of this.
The game currently uses:
Idle player
Idle player with gun in hand
Mining player without axe
Mining player with axe
Running player
Running player with gun in hand
Player corpse
Each of these animations then have a variation for two armor levels, which can be drawn on top of the basic character. In total we are talking about over 4,000 individual sprites, double that number when we also consider high resolution.
From past experience we already know that re-rendering an entity in high resolution is almost never “just make it have more pixels”. Shaders and details start working differently, and it’s always necessary to do some changes. However, editing 21 Blend files at the same time does not sound very appealing or efficient, so our plan was to merge all of the animation into one huge Blend file. The aim is to have:
Blender file with 21 scenes, each rendering one animation.
Everything must render automatically.
All of the meshes linked so if we edit one, the rest automatically update.
All of the meshes use the same materials so we just edit one.
It must be possible to revisit the player character in the future.
Per usual this is much easier said than done, it is possible with the tools that Blender has, but would require an extreme amount of manual work. It would also take way too much time, and would be way too fragile.
Graphics workflow
Making the player character in high resolution would have been utterly insane without the use of scripting. In the following part of the article I will show you a few of the most useful Blender scripts that I wrote and used in the process.
1. Copy objects between Blender scenes The first task sounds simple - copy-paste the objects from the source Blend file to the new one - until you try to do it. In Blender you can Copy + Paste objects between scenes easily, the catch is that in the destination, they are all pasted to the same layer, depending on which layer is active. However the objects forget which layer they are from in the source scene. This is a big problem, but after short thinking I wrote a new object transferring tool.
It is a set of two scripts, the first one prepares objects in the source scene (it changes their names to have a prefix with layer numbers). The second script processes them in destination by reading the name, moving the object to the appropriate layer(s), and cleaning up the name as if nothing ever happened.
2. Link mesh data of identical objects The copy pasting is great, but every new paste creates a set of new unique objects with unique materials. If you do this 21 times, the amount of duplicate data which is supposed to be the same goes up like a rocket. Generally 3D software has a way to create duplicates which are linked so if you edit one, you also edit the other. However once you break the link, it’s not so easy to bring it back, because the program isn’t sure if the two objects are really the same. In Blender this is possible, but doing it one by one would be way too much work so it’s time for another script.
This one reads every object’s vertices one by one, and saves them to giant lists. It then compares it to another object, and if the other object is identical, it makes a link between them - if not, it keeps both of them unique and adds the new one to the list of vertex data to be compared with.
It’s actually surprisingly fast, considering that it filters through about 5 million vertices, but it still takes several minutes to process.
The result seems good except for one problem - some of the objects have identical mesh data, but special materials that we used for rendering shadows, which is solved by running another script, which searches for objects with the shadow maker material and removes them. This has to be done before linking the mesh data to make sure that our mesh data ends up with the correct materials.
3. Clean material slots The next step is to unify materials. This is already mostly taken care of because the mesh data already carries the used materials in it, but going through all of them and manually checking was required. It wasn’t helping that every object had defined about 20 material slots, even if it was using just a handful of them.
Whenever you have a mesh with multiple materials being used on different polygons, and you remove some polygons, chances are that you aren’t using all of the materials any more.
So I made myself a script which removes the unused material slots so I wouldn’t have to manually check so many of them. It’s not only more work but it’s also much easier to spot errors when you see only what you need to.
4. Generate Render Layers and Compositor Nodes With all of the objects correctly linked and materials properly editable, it’s time to figure out how to output this whole thing into sprites. As we already hinted in FFF-146, we are almost always using Render Layers going into Compositor nodes to output many different passes of the same scene.
Since we are keeping the layer structure of the original source, it was possible to set up the rendering system to do the same again. It was still a lot of work because it was not always consistent and many of the scenes required small but hard to spot tweaks.
One thing which helps at least a little bit, is pre-generating all of the render layers with their names and some basic setup, which is consistent between all scenes.
Render Layers aren’t much without a way to save them to the final images - this is handled by the compositor nodes. There is a special case for Ambient Occlusion and Shadow passes, but in general it’s absolutely consistent so scripting this part is great as it removes all need for manual attention to this. The script just reads the Render Layers and generates the nodes to fit them.
5. Rendering and final processing To render all this from the 3D scene, we are using a fancy system which allows us to render across multiple computers, as the animations would take over 40 hours to render on a single PC.
The rest is basically the same as with the combinators shown in FFF-194, so I won't go into details here.
Result
After all this, we get the player graphics in high resolution, with colour masks working correctly and also a lot of new Blender tools for ourselves to work with.
And here is a preview of how the colour masks work for 0.16.
All the things in this article combined took a lot of time to make, especially the high resolution version of the player character. In the long term we are likely going to do some more changes and adjustments to the player character models and animations. Thanks to the new source files we will be able to do that, and for now we are satisfied with how the player looks for 0.16.
Razer Chroma
Razer contacted us quite a while back and asked us if we could add Razer Chroma support to Factorio. Razer Chroma/Chroma Link means that the game can take control of any supported RGB devices (from mouse and keyboard to LED strips and chairs) and control the colors of the lights.
Damage taken animation (Keyboard, mouse, mousepad, headset, chroma link)
Player Healthbar when damaged (Keyboard, mouse, mousepad)
Player shields when damaged (Keyboard, mouse, mousepad)
Progress-bar during game loading, savegame loading, multiplayer map download multiplayer map loading, multiplayer catching up(Keyboard, mouse, mousepad, headset, chroma link)
Flash technology key on technology researched (Keyboard)
Flash map key when building are destroyed (Keyboard)
Custom Ripple animation when achievement is unlocked (Keyboard, mouse, mousepad, headset, chroma link)
People already want to control their light using the circuit network and Lua, so something like that could be cool to add in the future if there is demand for it.
Normally we wouldn't add features to 0.15 after it was marked stable, but since there will be some Razer events, we thought it's good idea to have Factorio in the list of supported games. So the feature was added today to 0.15 as an experimental update but it will be marked as stable next week if nothing is broken.
If you have anything to say you can let us know on our forum.
Hello, most of the team is out of the office today attending the Game Developers Session here in Prague, if you're around you can look out for some people wearing Factorio t-shirts.
As we were thinking about what to write in this Friday Facts, kovarex suggested "In the next Friday Facts we should write about how hard it is to write Friday Facts". Sometimes it is really difficult to find something interesting to write about. Thankfully we found some short things that that we thought you would like.
Passenger seat for vehicles
Just a minor multiplayer feature.
Resource generation and game balance
I wanted to have a look at how we generate resources and try to balance and improve it so it's a bit more fun. When playing the game, I noticed that I always need more iron than copper and I also felt that there is more copper than iron on the map, so I first wanted to look at that.
When we balance out any part of the game it's usually something like "that seems a bit low, let's increase it by 0.2-ish and see how it goes". While this worked surprisingly well so far, I like to take a more scientific approach and look at hard numbers when possible.
First I looked at resource requirements. In order to complete all non-infinite research you need:
60,445 Science pack 1
59,885 Science pack 2
48,600 Science pack 3
20,800 Production science pack
27,925 High-tech science pack
32,445 Military science pack
To make all this you need:
3.5 million copper and 5.2 million iron. Ratio: 0.67 (in normal mode)
10.4million copper and 10.9 million iron. Ratio: 0.95 (in expensive recipes mode)
I would say it's safe to assume that these are close to the number of resources needed to finish the game. Since you wont research everything, which will compensate for the cost of infrastructure and combat. Here I see that there is a different ratio of copper to iron, and the ratio is different in expensive mode.
Now going to the map generation, I generated some 2048x2048 maps and calculated the total number of resources on them. For some reason there was always more copper. This seemed very strange since the map generation settings had the same values for both copper and iron. It turns out that some resources are always on top of others. Coal was on top of copper that was on top of iron that was on top of stone that was on top of uranium (alphabetical order). This meant that there will be more copper on average. To be precise, the maps had 1.5 billion copper and 1.3 billion iron. That's a ratio of 1.15 caused by overlapping. I changed the order of resources around a bit so Iron is on top.
To balance all this out, the plan is to change some some recipes in expensive mode, so that the required copper to iron ratio is roughly the same in normal and expensive mode. Then the map generator settings will be tweaked so it also reflects that ratio (of course while keeping in mind the difference caused by overlapping resources).
Now some might argue that what's the point of all this. Making things balanced does not mean more fun, for example making the player and the tank overpowered and unbalanced in combat made them way more fun. But for resources I believe it's more fun to find value in almost every resource patch you find as you explore, instead of "oh, another copper patch, how useless", especially if the player assigns the same value to it and has some expectation that they are balanced. Balancing all this is not a big deal, but it's just a subtle attention to detail that might make the game 1% more fun, and polishing all these little things will happen more and more as we approach 1.0.
While I'm looking at the resource generation, there are more things that I plan to improve. For example:
Tweaking the resource density (average number of resources per tile).
Making resources much more spread apart.
Reworking how the staring area works so that it always contains a predictable amount of resources. This means that every new map is feasible.
Making sure the starting area is not covered by trees.
More optimizations
When we talk about game performance improvements it's almost always focused on the entity-update time as that is primarily what determines how fast the game can run. There is however one other important part and that's the "prepare" step that collects minimal information about the game to be rendered on screen. This step happens between the game being updated and the results being rendered on the screen which means the game has to be paused while it's run.
We haven't looked into improving this part of the game for quite some time because it runs in multiple threads and was always been "somewhat quick". Recently I decided to spend some time trying to improve it and found several easy optimizations. The end result being the prepare step now runs roughly 50% faster than it did before, leaving more time for the game logic and entity updates.
Lua API additions
With every major update we add keep improving the Lua mod API and 0.16 is no exception; between the larger tasks and bug fixing I've been working on requested Lua API additions. For those interested I've been keeping a public gist of the 0.16 changes and additions here.
I'm always reading the forums for new requests or changes to the API, so If you can make a valid argument for some new API feature (and provided it doesn't negatively impact the game performance when not used), please let us know on the Modding interface requests forum.
Let us know what you think by commenting in our usual topic at the forums.
Hello, most of the team is out of the office today attending the Game Developers Session here in Prague, if you're around you can look out for some people wearing Factorio t-shirts.
As we were thinking about what to write in this Friday Facts, kovarex suggested "In the next Friday Facts we should write about how hard it is to write Friday Facts". Sometimes it is really difficult to find something interesting to write about. Thankfully we found some short things that that we thought you would like.
Passenger seat for vehicles
Just a minor multiplayer feature.
Resource generation and game balance
I wanted to have a look at how we generate resources and try to balance and improve it so it's a bit more fun. When playing the game, I noticed that I always need more iron than copper and I also felt that there is more copper than iron on the map, so I first wanted to look at that.
When we balance out any part of the game it's usually something like "that seems a bit low, let's increase it by 0.2-ish and see how it goes". While this worked surprisingly well so far, I like to take a more scientific approach and look at hard numbers when possible.
First I looked at resource requirements. In order to complete all non-infinite research you need:
60,445 Science pack 1
59,885 Science pack 2
48,600 Science pack 3
20,800 Production science pack
27,925 High-tech science pack
32,445 Military science pack
To make all this you need:
3.5 million copper and 5.2 million iron. Ratio: 0.67 (in normal mode)
10.4million copper and 10.9 million iron. Ratio: 0.95 (in expensive recipes mode)
I would say it's safe to assume that these are close to the number of resources needed to finish the game. Since you wont research everything, which will compensate for the cost of infrastructure and combat. Here I see that there is a different ratio of copper to iron, and the ratio is different in expensive mode.
Now going to the map generation, I generated some 2048x2048 maps and calculated the total number of resources on them. For some reason there was always more copper. This seemed very strange since the map generation settings had the same values for both copper and iron. It turns out that some resources are always on top of others. Coal was on top of copper that was on top of iron that was on top of stone that was on top of uranium (alphabetical order). This meant that there will be more copper on average. To be precise, the maps had 1.5 billion copper and 1.3 billion iron. That's a ratio of 1.15 caused by overlapping. I changed the order of resources around a bit so Iron is on top.
To balance all this out, the plan is to change some some recipes in expensive mode, so that the required copper to iron ratio is roughly the same in normal and expensive mode. Then the map generator settings will be tweaked so it also reflects that ratio (of course while keeping in mind the difference caused by overlapping resources).
Now some might argue that what's the point of all this. Making things balanced does not mean more fun, for example making the player and the tank overpowered and unbalanced in combat made them way more fun. But for resources I believe it's more fun to find value in almost every resource patch you find as you explore, instead of "oh, another copper patch, how useless", especially if the player assigns the same value to it and has some expectation that they are balanced. Balancing all this is not a big deal, but it's just a subtle attention to detail that might make the game 1% more fun, and polishing all these little things will happen more and more as we approach 1.0.
While I'm looking at the resource generation, there are more things that I plan to improve. For example:
Tweaking the resource density (average number of resources per tile).
Making resources much more spread apart.
Reworking how the staring area works so that it always contains a predictable amount of resources. This means that every new map is feasible.
Making sure the starting area is not covered by trees.
More optimizations
When we talk about game performance improvements it's almost always focused on the entity-update time as that is primarily what determines how fast the game can run. There is however one other important part and that's the "prepare" step that collects minimal information about the game to be rendered on screen. This step happens between the game being updated and the results being rendered on the screen which means the game has to be paused while it's run.
We haven't looked into improving this part of the game for quite some time because it runs in multiple threads and was always been "somewhat quick". Recently I decided to spend some time trying to improve it and found several easy optimizations. The end result being the prepare step now runs roughly 50% faster than it did before, leaving more time for the game logic and entity updates.
Lua API additions
With every major update we add keep improving the Lua mod API and 0.16 is no exception; between the larger tasks and bug fixing I've been working on requested Lua API additions. For those interested I've been keeping a public gist of the 0.16 changes and additions here.
I'm always reading the forums for new requests or changes to the API, so If you can make a valid argument for some new API feature (and provided it doesn't negatively impact the game performance when not used), please let us know on the Modding interface requests forum.
Let us know what you think by commenting in our usual topic at the forums.
Hello, I wanted to write about the things I'm improving in our GUI library, but I realized, that the important part is to explain what is the motivation to do so. So let me present the history of Factorio GUI.
Version 0.X
Back in the day when Factorio started, I was quite clueless. I needed some GUI library for allegro, and agui was the only one I could find, so I started using it. As in most of the story, I didn't really pay much attention to GUI, it was just the pesky part of the code which I needed to work in order for the game to function. There weren't much GUIs back then, and most of them were done by manually placing bunch of elements so it is possible to interact with it. The look and positioning was quite random.
The previous strategy stopped being possible quite soon, and we realized that we can't just manually place widgets in a window, so we started to use the layout functionality to build the GUI, so it doesn't break immediately once it contains just a little bit of different data. At that time, it was good.
Version 0.5
It is hard to believe now, but at that time, I was insisting, that the GUI is OK, and we don't need to improve it, but luckily, I was persuaded by Albert and Tomas that we need to give the GUI a better look. That was how was the look similar to the current one created. We needed to add hierarchical graphical style system to be able to control the looks and positioning of individual elements, so we did it. We also needed to add several additional functions to the layout mechanism as the need to make these two windows have the same height automatically and similar things. So we made specialised features for that by bending some of the original GUI library code. It usually worked, and when it didn't we made little hacks here and there to make it work. It also created new problems, like the random margins of elements that were supposed to be aligned with something. As all of this was made by unique specific numbers of widths/heights of elements, it could never work right, as all the style values were scaled depending on the resolution.
Here for example, the progress bar style was specified to have a somewhat correct width, and we didn't even care or understand how bad it is.
Version 0.6
It was identified at this time, that we need some mechanism to solve these kind of problems, so I created the stretchable functionality. As I was clueless about the library internals, I was afraid to break anything, the usage of it was quite awkward, it had to be used together with the functionality of mechanism from 0.5 to stretch windows to have the same width/height. You needed to set correctly some of the style values and some of the C++ object values to make it work properly. You had to set the stretchable not only the element but all of the parent elements etc. But it worked ...
Current day
Small hacks here and there accumulated. When you use stretch layout and set align to right doesn't work? Well just set the layout to go from right to left and put things in opposite order. Isn't the layout doing what you need? Just calculate the needed dimensions manually and force them in the style. And much more ...
New people started joining our team and they didn't know the subtle things you needed to push to make it work and lot of new code was created with even more hacks. The GUI started to be a crazy complex beast. At that time, one of the new programmers wanted to fix some of the hacks. The result was, that he wanted to fix everything, and after 9 months of repeated inability to show anything, he was fired. After that time, GUI was considered even more scary. Things like this started to appear all over the place in the desparate hopes of making it work:
Some people have even shown their disgust by naming their variables:
GUI refactoring
Now we are facing the need to make the GUI update (first part was presented in one of the recent fffs). At this point, you have some idea of how painful would it be to just glue the needed additions to our current GUI. It would be painful complexity squared.
This is why I decided to roll up my sleeves and just dive into it. I started rewriting and clean-up the core parts of how the layout works. At this is point, I have a branch that has 52k lines of changes compared to master, and the internal workings of the GUI have been revised a lot. A lot of the mechanisms that had to be hacked painfully work automatically and there is much less code and knowledge needed to write the GUI in the new branch. But this also means, that we have to go through every single GUI and remove the previous hacks by the new functionality, which is a lot of work.
To aid in the GUI simplification work, we created a simple GUI debug view, to let us quickly see how are the layouts stacked. I will leave it in the game, as it could be also useful for modders when they need to debug their GUI structures.
On top of that, I started to write GUI tests, as GUI layout is the great example of piece of software, where fixing one thing breaks some other without you knowing. These changes will also change the way modders specify GUI structures (mainly related to the align and stretching), but it wouldn't be us if we didn't break mod compatibility in a new release.
As always, let us know what you think on our forums.
Hello, I wanted to write about the things I'm improving in our GUI library, but I realized, that the important part is to explain what is the motivation to do so. So let me present the history of Factorio GUI.
Version 0.X
Back in the day when Factorio started, I was quite clueless. I needed some GUI library for allegro, and agui was the only one I could find, so I started using it. As in most of the story, I didn't really pay much attention to GUI, it was just the pesky part of the code which I needed to work in order for the game to function. There weren't much GUIs back then, and most of them were done by manually placing bunch of elements so it is possible to interact with it. The look and positioning was quite random.
The previous strategy stopped being possible quite soon, and we realized that we can't just manually place widgets in a window, so we started to use the layout functionality to build the GUI, so it doesn't break immediately once it contains just a little bit of different data. At that time, it was good.
Version 0.5
It is hard to believe now, but at that time, I was insisting, that the GUI is OK, and we don't need to improve it, but luckily, I was persuaded by Albert and Tomas that we need to give the GUI a better look. That was how was the look similar to the current one created. We needed to add hierarchical graphical style system to be able to control the looks and positioning of individual elements, so we did it. We also needed to add several additional functions to the layout mechanism as the need to make these two windows have the same height automatically and similar things. So we made specialised features for that by bending some of the original GUI library code. It usually worked, and when it didn't we made little hacks here and there to make it work. It also created new problems, like the random margins of elements that were supposed to be aligned with something. As all of this was made by unique specific numbers of widths/heights of elements, it could never work right, as all the style values were scaled depending on the resolution.
Here for example, the progress bar style was specified to have a somewhat correct width, and we didn't even care or understand how bad it is.
Version 0.6
It was identified at this time, that we need some mechanism to solve these kind of problems, so I created the stretchable functionality. As I was clueless about the library internals, I was afraid to break anything, the usage of it was quite awkward, it had to be used together with the functionality of mechanism from 0.5 to stretch windows to have the same width/height. You needed to set correctly some of the style values and some of the C++ object values to make it work properly. You had to set the stretchable not only the element but all of the parent elements etc. But it worked ...
Current day
Small hacks here and there accumulated. When you use stretch layout and set align to right doesn't work? Well just set the layout to go from right to left and put things in opposite order. Isn't the layout doing what you need? Just calculate the needed dimensions manually and force them in the style. And much more ...
New people started joining our team and they didn't know the subtle things you needed to push to make it work and lot of new code was created with even more hacks. The GUI started to be a crazy complex beast. At that time, one of the new programmers wanted to fix some of the hacks. The result was, that he wanted to fix everything, and after 9 months of repeated inability to show anything, he was fired. After that time, GUI was considered even more scary. Things like this started to appear all over the place in the desparate hopes of making it work:
Some people have even shown their disgust by naming their variables:
GUI refactoring
Now we are facing the need to make the GUI update (first part was presented in one of the recent fffs). At this point, you have some idea of how painful would it be to just glue the needed additions to our current GUI. It would be painful complexity squared.
This is why I decided to roll up my sleeves and just dive into it. I started rewriting and clean-up the core parts of how the layout works. At this is point, I have a branch that has 52k lines of changes compared to master, and the internal workings of the GUI have been revised a lot. A lot of the mechanisms that had to be hacked painfully work automatically and there is much less code and knowledge needed to write the GUI in the new branch. But this also means, that we have to go through every single GUI and remove the previous hacks by the new functionality, which is a lot of work.
To aid in the GUI simplification work, we created a simple GUI debug view, to let us quickly see how are the layouts stacked. I will leave it in the game, as it could be also useful for modders when they need to debug their GUI structures.
On top of that, I started to write GUI tests, as GUI layout is the great example of piece of software, where fixing one thing breaks some other without you knowing. These changes will also change the way modders specify GUI structures (mainly related to the align and stretching), but it wouldn't be us if we didn't break mod compatibility in a new release.
As always, let us know what you think on our forums.
To make PvP fun, and fair, it is pretty important that all the teams have a fair start. We do this for the normal game by using some special logic to generate a specific starting area. While the map generation didn't support this, I worked around the issue by copy-pasting all the starting area chunks to new starting areas for all players. This looked very odd though, and since it was done in the Lua script, was very slow.
So the solution is to fix the map generator to create multiple starting points, which with all the refactoring work by TOGoS, was not so hard to do. Whereas before all the logic calculated its own 'distance from the start' from (0,0), now it looks for the closest starting area, and calculates the distance from that. This actually fixes and simplifies a lot of the PvP script, as I also don't have to worry about clearing away huge biters nests from each teams front door.
This is a much cleaner solution, as the new starting areas are seamlessly integrated into the map.
Multithreading issues
We were always saying, that we keep the full update multithreading as the last ace in the sleeve to be used after the normal optimisations become too difficult. Apart from the plan to split update into chunks and update them separately, there are also smaller and simpler things we can do. We have for example this part of the update step, where we update trains, electric network and belts, and we update it one after another like this:
But these tasks are almost independent as neither trains or belts use electricity and belts don't interact with trains. There would be some details that would have to be solved, like Lua API calls and inserter activation order, but other than that it would be pretty straightforward. So I tried to run these things in parallel and measured how fast it would be:
To my surprise, the parallel version didn't speed things up, it was actually even slower. First I thought that it is caused by the fact, that the threads steal each others cache or the task is too short to be parallelized etc., but then I realized, that we already do the "prepare logic" in parallel. The prepare logic gathers all the data (sprite draw orders) for rendering and it is doing in parallel up to 8 threads. The task is iterating through the game world, and it is also quite short. Yet still the parallelization works pretty well there. After a few days (and many cppcon videos about multithreading later), I got the answer to my question. The answer is the same as many other answers related to performance, it is caused by cache invalidation.
There is some shared cache for all the threads (L3 I believe), and some smaller caches specific for each of the cores (L2 and L1 I believe). The cache is divided into some kind of pages. The problem is that as long as I don't control the memory allocation, the data of 3 objects that might be completely independent can end up in the same page of memory cache. So lets imagine this situation:
Even if two different cores work with the same part of memory, they don't slow each other down as long as they only read the memory (Which is the case of our prepare logic as it is read only operation) as each core has its own copy of the memory page. The problem arises when the operation is not read only:
Each thread has its own copy of the page, but whenever something is changed, the other copies of the same page need to be invalidated and updated. This means that the threads are invalidating each others cache all the time, which slows the whole process so much that it is slower than the non-parallel solution.
How do we solve it? The solution would probably be to organize the memory allocation in such a way, that everything that is on the same chunk, or related to some specific task would have to use its own memory allocator that keeps things in one place together in the memory. Every chunk would have its own piece of memory dedicated to entities in the chunk, trains would have all its data in one piece of memory together etc. This way, it shouldn't happen that two tasks would work on one piece of memory at the same time so the multithreading would actually speed things up.
The problem with this is, that it requires big changes in the code as not only the entities itself but all their dynamically allocated data would have to be always allocated depending on the entity position. Whenever the entity moves between chunks, all its data structures would have to be re-allocated in the new chunk. All pointers to any data structures in entity would have to be properly updated as they move around etc. This means, that once (if) we decide to do this step, it would probably be one big update related only to this and this is not something we have time to do before 1.0.
As always, let us know what you think on our forums.
To make PvP fun, and fair, it is pretty important that all the teams have a fair start. We do this for the normal game by using some special logic to generate a specific starting area. While the map generation didn't support this, I worked around the issue by copy-pasting all the starting area chunks to new starting areas for all players. This looked very odd though, and since it was done in the Lua script, was very slow.
So the solution is to fix the map generator to create multiple starting points, which with all the refactoring work by TOGoS, was not so hard to do. Whereas before all the logic calculated its own 'distance from the start' from (0,0), now it looks for the closest starting area, and calculates the distance from that. This actually fixes and simplifies a lot of the PvP script, as I also don't have to worry about clearing away huge biters nests from each teams front door.
This is a much cleaner solution, as the new starting areas are seamlessly integrated into the map.
Multithreading issues
We were always saying, that we keep the full update multithreading as the last ace in the sleeve to be used after the normal optimisations become too difficult. Apart from the plan to split update into chunks and update them separately, there are also smaller and simpler things we can do. We have for example this part of the update step, where we update trains, electric network and belts, and we update it one after another like this:
But these tasks are almost independent as neither trains or belts use electricity and belts don't interact with trains. There would be some details that would have to be solved, like Lua API calls and inserter activation order, but other than that it would be pretty straightforward. So I tried to run these things in parallel and measured how fast it would be:
To my surprise, the parallel version didn't speed things up, it was actually even slower. First I thought that it is caused by the fact, that the threads steal each others cache or the task is too short to be parallelized etc., but then I realized, that we already do the "prepare logic" in parallel. The prepare logic gathers all the data (sprite draw orders) for rendering and it is doing in parallel up to 8 threads. The task is iterating through the game world, and it is also quite short. Yet still the parallelization works pretty well there. After a few days (and many cppcon videos about multithreading later), I got the answer to my question. The answer is the same as many other answers related to performance, it is caused by cache invalidation.
There is some shared cache for all the threads (L3 I believe), and some smaller caches specific for each of the cores (L2 and L1 I believe). The cache is divided into some kind of pages. The problem is that as long as I don't control the memory allocation, the data of 3 objects that might be completely independent can end up in the same page of memory cache. So lets imagine this situation:
Even if two different cores work with the same part of memory, they don't slow each other down as long as they only read the memory (Which is the case of our prepare logic as it is read only operation) as each core has its own copy of the memory page. The problem arises when the operation is not read only:
Each thread has its own copy of the page, but whenever something is changed, the other copies of the same page need to be invalidated and updated. This means that the threads are invalidating each others cache all the time, which slows the whole process so much that it is slower than the non-parallel solution.
How do we solve it? The solution would probably be to organize the memory allocation in such a way, that everything that is on the same chunk, or related to some specific task would have to use its own memory allocator that keeps things in one place together in the memory. Every chunk would have its own piece of memory dedicated to entities in the chunk, trains would have all its data in one piece of memory together etc. This way, it shouldn't happen that two tasks would work on one piece of memory at the same time so the multithreading would actually speed things up.
The problem with this is, that it requires big changes in the code as not only the entities itself but all their dynamically allocated data would have to be always allocated depending on the entity position. Whenever the entity moves between chunks, all its data structures would have to be re-allocated in the new chunk. All pointers to any data structures in entity would have to be properly updated as they move around etc. This means, that once (if) we decide to do this step, it would probably be one big update related only to this and this is not something we have time to do before 1.0.
As always, let us know what you think on our forums.
