I finished the item stack optimisations mentioned in FFF-198, and was able to do some performance tests. First I tested how many stacks on a big map actually need to use an externally allocated object (Item), and how many of them are plain. On the huge map I tested, it turned out that only 36K out of 1M stacks need the Item object. These were mainly science packs, as they need it for the progress of how used-up they are (and now when I think about it, it could also be omitted by only using the objects for science packs that are partially used up already). Overall factory performance was increased approximately 2% by this. It is nothing huge, but every bit matters.

One of the programmer that has read access to the code (Zulan), came up with a pull request that improves performance in Factorio by prefetching memory in the update loops ahead.

The problem when normally updating objects is, that CPU asks for memory representing the object. The memory is slow, at least compared to the CPU cache or the CPU speed. The memory transfer speed itself is not that slow, but the waiting (latency) time between ordering and receiving it is. This means, that what very often happens is, that CPU orders data of next entity from the memory, then it waits for quite a long time to get it, and then it does its logic. The memory prefetching partially solves it by doing this:

• Order data of the next entity from memory (prefetch)
• Do the logic of the current entity in the meantime
• Go back to start

The overall measured performance improvements vary between 6-10%, which is certainly a nice addition.

As flexible and powerful as it is, we have always felt there was one key missing to the puzzle. The main issue is that requester chests cannot provide their items to any other member of the logistics system. Trying to workaround this by putting an inserter to a passive provider, just leads to the robots moving the items in a loop. This is also a nuisance trying to supply construction robots with materials, as they can only collect them from storage or provider chests, and they are only typically located in the main base areas.



It is easy to design a system to resupply far out areas using trains to directly put items in provider chests, but if they are in the same logistic network we encounter the same loop as before. We were also concerned of people segregating their logistic networks for more control, it seems to us it was a workaround to a problem we should fix. The solution is the buffer chest, which functions as both a requester and passive provider chest.



You can see the buffer can act as an 'in-between' for storage/provider chests and the requesters. This leads to a solution of the main annoyances we identified.

Typically when you set up all the provider chests, they are spread out across your whole factory. When you return to base for a resupply, you end up waiting for a long time while the bots travel from all over the factory with items. By using a buffer chest, you can setup a dedicated 'supply area', where the buffer chest will already contain all the typical items, and the bots can quickly top-up your inventory.



Another problem is when you have a large perimeter defence, and you want it to be maintained by the construction robots. When the main base is so far away, it can take a long time for robots to arrive with repair packs, so the biters might be able to break through. Using the buffer chest, it will be easy to setup nearby supplies to quickly repair the walls when needed.



Last, but not least use-case is when you want to dedicate part of your storage for specific things. The reason can be either just OCD, or the fact, that you can make sure that too much coal for example won't make it impossible to store enough of iron ore in your storage system.



Here we present you the regular dose of new entities updated for high resolution for the hopefully fully high-res friendly 0.16 release.



As always, leave us any feedback or comments on our forum

I finished the item stack optimisations mentioned in FFF-198, and was able to do some performance tests. First I tested how many stacks on a big map actually need to use an externally allocated object (Item), and how many of them are plain. On the huge map I tested, it turned out that only 36K out of 1M stacks need the Item object. These were mainly science packs, as they need it for the progress of how used-up they are (and now when I think about it, it could also be omitted by only using the objects for science packs that are partially used up already). Overall factory performance was increased approximately 2% by this. It is nothing huge, but every bit matters.

One of the programmer that has read access to the code (Zulan), came up with a pull request that improves performance in Factorio by prefetching memory in the update loops ahead.

The problem when normally updating objects is, that CPU asks for memory representing the object. The memory is slow, at least compared to the CPU cache or the CPU speed. The memory transfer speed itself is not that slow, but the waiting (latency) time between ordering and receiving it is. This means, that what very often happens is, that CPU orders data of next entity from the memory, then it waits for quite a long time to get it, and then it does its logic. The memory prefetching partially solves it by doing this:

• Order data of the next entity from memory (prefetch)
• Do the logic of the current entity in the meantime
• Go back to start

The overall measured performance improvements vary between 6-10%, which is certainly a nice addition.

As flexible and powerful as it is, we have always felt there was one key missing to the puzzle. The main issue is that requester chests cannot provide their items to any other member of the logistics system. Trying to workaround this by putting an inserter to a passive provider, just leads to the robots moving the items in a loop. This is also a nuisance trying to supply construction robots with materials, as they can only collect them from storage or provider chests, and they are only typically located in the main base areas.



It is easy to design a system to resupply far out areas using trains to directly put items in provider chests, but if they are in the same logistic network we encounter the same loop as before. We were also concerned of people segregating their logistic networks for more control, it seems to us it was a workaround to a problem we should fix. The solution is the buffer chest, which functions as both a requester and passive provider chest.



You can see the buffer can act as an 'in-between' for storage/provider chests and the requesters. This leads to a solution of the main annoyances we identified.

Typically when you set up all the provider chests, they are spread out across your whole factory. When you return to base for a resupply, you end up waiting for a long time while the bots travel from all over the factory with items. By using a buffer chest, you can setup a dedicated 'supply area', where the buffer chest will already contain all the typical items, and the bots can quickly top-up your inventory.



Another problem is when you have a large perimeter defence, and you want it to be maintained by the construction robots. When the main base is so far away, it can take a long time for robots to arrive with repair packs, so the biters might be able to break through. Using the buffer chest, it will be easy to setup nearby supplies to quickly repair the walls when needed.



Last, but not least use-case is when you want to dedicate part of your storage for specific things. The reason can be either just OCD, or the fact, that you can make sure that too much coal for example won't make it impossible to store enough of iron ore in your storage system.



Here we present you the regular dose of new entities updated for high resolution for the hopefully fully high-res friendly 0.16 release.



As always, leave us any feedback or comments on our forum

• Mod name in mod info pane will no longer be localised. more
  
• Optional mod dependencies now show as orange when invalid. more
  

• Fixed crash when trying to load replay that is not compatible with current version.
  
• Fixed ghosts might emit light. more
  
• Fixed removing land mines didn't make any sound. more
  
• Fixed creating window larger than screen. more
  
• Improved performance of rendering uranium ore. more
  

• Bonus UI now shows additional force modifiers more
  
• simple-entity-with-owner (and simple-entity-with-force) now supports apply_runtime_tint in sprite definition.
  

• simple-entity-with-owner exposes color property through LuaEntity::color. Set it to {r=0, g=0, b=0, a=0} to use color of entity's force.
  
• Fixed LuaEntityPrototype::distribution_effectivity would return value of supply_area_distance instead. more
  

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Mod name in mod info pane will no longer be localised. more
  
• Optional mod dependencies now show as orange when invalid. more
  

• Fixed crash when trying to load replay that is not compatible with current version.
  
• Fixed ghosts might emit light. more
  
• Fixed removing land mines didn't make any sound. more
  
• Fixed creating window larger than screen. more
  
• Improved performance of rendering uranium ore. more
  

• Bonus UI now shows additional force modifiers more
  
• simple-entity-with-owner (and simple-entity-with-force) now supports apply_runtime_tint in sprite definition.
  

• simple-entity-with-owner exposes color property through LuaEntity::color. Set it to {r=0, g=0, b=0, a=0} to use color of entity's force.
  
• Fixed LuaEntityPrototype::distribution_effectivity would return value of supply_area_distance instead. more
  

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

I decided to write about the results of the item stack optimisations explained in the FFF-198, so I rushed today to finish its implementation, just to find out that the task affects an even bigger part of the code than I expected, Items are related to many things in Factorio :)

After many hours of rewriting and fixing, I can compile it and even start a game, but most of the things are broken. It is quite funny to see some of the basic item interactions to be broken. Now I'm making commits like "Now I can split stacks", "Now I can merge stacks", etc. It reminds me the old days. In conclusion, the details of the optimization will have to wait for next week, and since it is after 10pm, this Friday facts will be somewhat shorter :)

I can at least present you the continued work on the updated high resolution graphics. The update of circuit connectors not only provides them in high resolution, but as it is possible to see it in more detail, the graphics can show more accurately what it represents. Specifically, if the connector is only reading the state of the machine (blue LED), controlling its behaviour (red/green LED) or both.

You can also clearly see how weird it looks when we combine a low res entity (roboport, chests, liquid tank) with a high res connector, but this is just a temporary state and most of these entities should be high-res compatible when the release is ready.



You can also notice (on the chests for example), that the green and red connectors are not vertically aligned, which might look slightly weird, but it is on purpose. In the current version (0.15), when two entities are in one column and they are using both the green and red cable, they overlap perfectly so it is impossible to see both of the cables at the same time as shown below.



Additionally, the green LED in this example doesn't make any sense, as the chest will always have only read mode, which is also addressed in the new graphics.

As always, leave us any feedback or comments over on our forum

I decided to write about the results of the item stack optimisations explained in the FFF-198, so I rushed today to finish its implementation, just to find out that the task affects an even bigger part of the code than I expected, Items are related to many things in Factorio :)

After many hours of rewriting and fixing, I can compile it and even start a game, but most of the things are broken. It is quite funny to see some of the basic item interactions to be broken. Now I'm making commits like "Now I can split stacks", "Now I can merge stacks", etc. It reminds me the old days. In conclusion, the details of the optimization will have to wait for next week, and since it is after 10pm, this Friday facts will be somewhat shorter :)

I can at least present you the continued work on the updated high resolution graphics. The update of circuit connectors not only provides them in high resolution, but as it is possible to see it in more detail, the graphics can show more accurately what it represents. Specifically, if the connector is only reading the state of the machine (blue LED), controlling its behaviour (red/green LED) or both.

You can also clearly see how weird it looks when we combine a low res entity (roboport, chests, liquid tank) with a high res connector, but this is just a temporary state and most of these entities should be high-res compatible when the release is ready.



You can also notice (on the chests for example), that the green and red connectors are not vertically aligned, which might look slightly weird, but it is on purpose. In the current version (0.15), when two entities are in one column and they are using both the green and red cable, they overlap perfectly so it is impossible to see both of the cables at the same time as shown below.



Additionally, the green LED in this example doesn't make any sense, as the chest will always have only read mode, which is also addressed in the new graphics.

As always, leave us any feedback or comments over on our forum

• Fixed compatibility problem with several antivirus programs. more
  
• Fixed seed in map-gen-settings.json would be ignored when creating map on headless server. more
  
• Fixed that connecting to a multiplayer game with a large blueprint library might be difficult. more
  
• Fixed that using capsules would open an Entity's GUI when clicked. more
  
• Fixed that --window-size=maximized wouldn't work on Linux. more
  
• Fixed that changing reactor consumption(production) values through a mod didn't update its production until rebuilt. more
  
• Fixed that blueprints would sometimes stop transferring.
  
• Fixed crash when opening item/container and at the same time the controller is set to some that doesn't have inventory. more
  
• Fixed 3 possible crashes related to getting malformed network packet over the network.
  
• Maybe fixed a biter path cache-related crash. more
  
• Fixed that bad_alloc and similar low level erros were catched internally, so we couldn't get proper stack trace of those.
  
• Limited the size of a train chart tag when the map is zoomed in. more
  
• Possible rare crash fix related to building rails and viewing preview of entities right after that. more
  
• Limited technology cost multiplier to maximum of 1000. more
  

• The log method also specifies the mod that wrote that, not only script file.
  
• Added LuaEntityPrototype::distribution_effectivity read.
  
• Added LuaEntityPrototype::time_to_live read.
  
• Added LuaControl::following_robots read.
  
• Added LuaPlayer::pipette_entity().
  
• Added LuaEntity::can_be_destroyed().
  
• Added script_raised_destroy reserved event ID.
  
• Added script_raised_built reserved event ID.
  
• Added script_raised_revive reserved event ID.
  
• Changed LuaEntity::time_to_live to also work for combat robots.
  
• Changed LuaEntityPrototype::fluid_capacity read to also work on fluid-wagon.
  
• Changed LuaEntityPrototype::turret_range read returns nil instead of error if not turret.
  
• Changed LuaEntity::train to return nil if entity is not rolling stock.
  
• Added LuaEntityPrototype::explosion_beam read.
  
• Added LuaEntityPrototype::explosion_rotate read.
  

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Fixed compatibility problem with several antivirus programs. more
  
• Fixed seed in map-gen-settings.json would be ignored when creating map on headless server. more
  
• Fixed that connecting to a multiplayer game with a large blueprint library might be difficult. more
  
• Fixed that using capsules would open an Entity's GUI when clicked. more
  
• Fixed that --window-size=maximized wouldn't work on Linux. more
  
• Fixed that changing reactor consumption(production) values through a mod didn't update its production until rebuilt. more
  
• Fixed that blueprints would sometimes stop transferring.
  
• Fixed crash when opening item/container and at the same time the controller is set to some that doesn't have inventory. more
  
• Fixed 3 possible crashes related to getting malformed network packet over the network.
  
• Maybe fixed a biter path cache-related crash. more
  
• Fixed that bad_alloc and similar low level erros were catched internally, so we couldn't get proper stack trace of those.
  
• Limited the size of a train chart tag when the map is zoomed in. more
  
• Possible rare crash fix related to building rails and viewing preview of entities right after that. more
  
• Limited technology cost multiplier to maximum of 1000. more
  

• The log method also specifies the mod that wrote that, not only script file.
  
• Added LuaEntityPrototype::distribution_effectivity read.
  
• Added LuaEntityPrototype::time_to_live read.
  
• Added LuaControl::following_robots read.
  
• Added LuaPlayer::pipette_entity().
  
• Added LuaEntity::can_be_destroyed().
  
• Added script_raised_destroy reserved event ID.
  
• Added script_raised_built reserved event ID.
  
• Added script_raised_revive reserved event ID.
  
• Changed LuaEntity::time_to_live to also work for combat robots.
  
• Changed LuaEntityPrototype::fluid_capacity read to also work on fluid-wagon.
  
• Changed LuaEntityPrototype::turret_range read returns nil instead of error if not turret.
  
• Changed LuaEntity::train to return nil if entity is not rolling stock.
  
• Added LuaEntityPrototype::explosion_beam read.
  
• Added LuaEntityPrototype::explosion_rotate read.
  

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

Hello,
the 0.15 has been declared stable. Unfortunately we found some smaller problems, so there is going to be at least one bugfix release. One of the problems we discovered yesterday, is a glitch in the blueprint transferring logic that results in the transfers stopping forever when a player that is just transferring his blueprint into the game leaves. I'm quite surprised that I found it out myself when I was testing something else, and we didn't have a single bug report regarding it.

This is a great example to show why complexity of software grows faster than linearly with the amount of features. We have mods and we have the blueprint library. Both systems work when used separately, but new problems come when you use these two features together. The bug report is basically describing that when you have blueprints with modded content, and you switch to vanilla (or different mod setup), all the mod related content in the blueprint library is removed upon opening any game. This makes these blueprints useless if you want go back, enable the mods and use the blueprints again.

This is can be very annoying for the player, and after quick a discussion we decided to solve it by having blueprint library separated for every mod configuration and by allowing players to transfer library contents from different mod configurations to the current one on demand. We decided to not push this into 0.15 as we are too afraid to break things now, so it is going to be 0.16 feature.

The second major problem with the blueprint library is the UX, especially in multiplayer. When the blueprint/book, is big and is being transferred while the player is already holding it, it can't be moved to inventory/quickbar, and the 'drop here to export to main inventory' button does nothing but print a message to the console.

The plan is to make a special item type, that will be basically a promise of blueprint and use it in this case. Once the blueprint transfer into the game is finished, the item will change into the blueprint. When the player related to it will leave, or remove the blueprint from his library, the item will just disappear. It can even have nice info into tooltip about what is going on. This is (obviously) also going to be in 0.16 not 0.15.

Generally speaking, the blueprint library is a new thing to Factorio, so there several other UX tweaks and polishes we want to do.

The rail segment visualisation that I described in FFF 198 is now completed.The main problem I had to solve is how to color the segments to make sure, that two neighbour segments don't have the same color. Some of you probably know, that this is well known thing called graph coloring. For plain graphs (which the rail segments are), it has been proven long time ago, that 4 colors is enough to color every graph, but the proof and algorithm is not simple. But for 6 colors, the algorithm is pretty easy, the only downside is, that it is designed to color the graph as a whole, which is fine when the game is being loaded, but not acceptable when adding a single signal to a huge rail network, I needed something incremental.

In the meantime I wanted to test the visualisation graphics proposal from Vaclav, so I made a very simple logic that picks the lowest available color index that is not already taken by neighbour rails to see how it looks. I was surprised to find out that in normal rail system, it doesn't use more than 3 colors most of the time, and it is very rare to see the 5th color . The conclusion is simple, this is good enough. Although it is possible to construct rail system that results in our algorithm putting two of the same colors next to each other, it is not really a problem, as it doesn't break anything, and in a normal game, the chance of it happening is close to 0. The takeaway is that it is important to realize how strict you need to depending on the situation.

Here is an example of intersection with a missing signal, how fast can you find it the old way?



How fast can you find it when you see the visualisation?



I guess that this topic is closed, the fact that 2-3 colors are used most of the time makes the visualisation not look too colorful and it will help both new players to understand the concept of blocks as well as help experienced players when signalling an intersection to see what is going on at a glance.

There's always something else that can be optimized. Last week we purchased some new hardware because I wanted to test if 'throwing money at it' was a viable method of improving compilation times (and the speed at which we can develop/fix things). We purchased a brand new i9-7900X CPU and compatible hardware with great results. After some difficulties getting everything setup it was almost 150% faster than the previous setup I was using. After that significant improvement in compilation times that got me wondering what else I do multiple times per day development wise that I could speed up. That led to game startup time, map loading/saving, and map generation.I grabbed a relatively large save file with a large factory setup (Colonelwill's current map) and set out to get a baseline. The save file is 45.7 MB compressed and 132 MB uncompressed. In 0.15.31 it takes 3.89 seconds to load and 2.46 seconds to save.

I think I can safely say that nobody likes waiting for the game to save (myself included); it's distracting and breaks you out of whatever you're currently doing in the game. While working on potential improvements to save time I measured exactly what happens when I tell the game to save and the results gave me a little more appreciation for just how fast saving actually is.

Technical/numbers
The way saving works in Factorio is through a process called serialization. We go over the entire loaded map beginning to end and write out each piece that's needed to restore the map. Every time something needs to be saved it calls the seralizer method 'write' with the data and how many bytes it should serialize. On the 132 MB of data that makes up the uncompressed map the 'write' method on the seralizer is called 84'272'542 times. Of those 84 million calls to write 99% are < 10 bytes.
The largest was only 121 bytes (called 84 times). In total 152'472'991 bytes were processed making for an average of 1.8 bytes per call. The entire process completed in 2.46 seconds for an average of 334'257 calls per millisecond. The serializer we use for saving the map uses several memory buffers that it cycles internally as it writes one buffer to disk in parallel with the rest of the map saving logic so there wasn't much performance to gain in this logic. But, seeing the numbers gave me a better picture of just how much work happens when the map is saved. In the end I found a few small things to improve on such that saving the map in 0.16 now only takes 2.25 seconds (a 9% improvement). Not what I hoped for but an improvement is an improvement.

Map load time
When looking at the time spent loading maps I found that we had a similar memory buffer system to saving but with 1 major difference: when the buffer ran out it would stop and refill the buffer from disk. That was an obvious slow spot with an easy fix: use multiple buffers and load them from disk in parallel with the map deserialization. I also found some inefficiencies in how trains were loaded causing them to run O(Rails * Trains) operations every time the map was loaded which I fixed to just O(Trains). The end result being what took 3.89 seconds to load in 0.15 now takes 2.8 seconds to load in 0.16 (a 28% improvement).

Game startup time - Graphics
On this new computer with normal graphics quality Factorio takes 9.84 seconds to reach the main menu. I think that's pretty good for a game these days but as we work on Factorio we close and restart the game 100s of times per day. Every time we change something, test a save file from a bug report, or as I was doing: testing out changes to see how they impact performance I have to wait for the game to start and it's wasted time. There's a disabled config option that takes all of the loaded graphics the game uses and packs them into 1 large file that can be read at startup. It mostly helps when you frequently restart the game - which is exactly what we do every day. With that enabled the game launches in 5.31 seconds (a 46% improvement).

Game startup time - Mods
We don't tend to use mods while we're working on new features so some performance problems related to them go unnoticed. One bug report recently had 111 mods the person was using which exposed a slowdown in processing all of the mods at startup. When mods are loaded we track what a mod changes, to record the history of what mods change what game objects. Originally we did this using the C++ Boost library "property tree" class, but it turned out to be faster to write our own logic using Lua than use the Boost class. Since that change we wrote our own version of the property tree class and in testing it showed to be even faster than doing the logic inside Lua. In 0.15.31 it took 28.72 seconds to process the mods and now in 0.16 it only takes 20.2 seconds (a 29% improvement). Not only does this benefit the players but it helps us because we don't have to spend as much time waiting for the game to load when working on modded bug reports.

Game startup time - Sounds
With the loading of graphics and mod data no longer being the slowest parts the next thing that started taking the majority time was loading sounds. I realized, after my work on improving map loading, that there wasn't any reason I couldn't just load sounds in parallel with the graphics. It was a 20 minute 'fix' and now by the time graphics are finished loading, sounds have long finished meaning it can proceed straight to the main menu.

Game startup time - Conclusion
With graphics being loaded from the atlas-cache, mod data processing being almost twice as fast, and sounds loaded in parallel I can now launch the game and be at the main menu ready to play in just 2.962 seconds (a 44% improvement). The change was interesting to get used to over the past week but now when I need to test things in 0.15 I find myself annoyed at how long everything takes compared to the 'new hotness' of 0.16.

Map generation - New maps
Unless I'm working on some specific save file I generate maps almost as frequently as I restart the game while working on bug fixes/new features. I've never considered generating a new map particularly slow but it never felt fast either. 0.15 already generates the map in the background while the game runs to prevent stuttering while new chunks are generated but the starting area when making a new map is not. I changed it so the starting area would generate in multiple threads if possible (while still maintaining determinism). The end result being: 0.15.31 generates a new map in 1.83 seconds and 0.16 generates a new map in 0.8 seconds (a 56% improvement).

Conclusion
Overall the start-game/load/generate game experience is measurably faster. Most importantly it feels noticeably faster, and makes for a better overall experience when we don't have to wait for things. With the belt optimizations and everything mentioned here, 0.16 is already promising to improve performance in multiple areas and full 0.16 development hasn't even begun, with other ideas we've talked about in previous Friday Facts still to be implemented promising even better performance.

As always, leave us any feedback or comments over on our forum

Hello,
the 0.15 has been declared stable. Unfortunately we found some smaller problems, so there is going to be at least one bugfix release. One of the problems we discovered yesterday, is a glitch in the blueprint transferring logic that results in the transfers stopping forever when a player that is just transferring his blueprint into the game leaves. I'm quite surprised that I found it out myself when I was testing something else, and we didn't have a single bug report regarding it.

This is a great example to show why complexity of software grows faster than linearly with the amount of features. We have mods and we have the blueprint library. Both systems work when used separately, but new problems come when you use these two features together. The bug report is basically describing that when you have blueprints with modded content, and you switch to vanilla (or different mod setup), all the mod related content in the blueprint library is removed upon opening any game. This makes these blueprints useless if you want go back, enable the mods and use the blueprints again.

This is can be very annoying for the player, and after quick a discussion we decided to solve it by having blueprint library separated for every mod configuration and by allowing players to transfer library contents from different mod configurations to the current one on demand. We decided to not push this into 0.15 as we are too afraid to break things now, so it is going to be 0.16 feature.

The second major problem with the blueprint library is the UX, especially in multiplayer. When the blueprint/book, is big and is being transferred while the player is already holding it, it can't be moved to inventory/quickbar, and the 'drop here to export to main inventory' button does nothing but print a message to the console.

The plan is to make a special item type, that will be basically a promise of blueprint and use it in this case. Once the blueprint transfer into the game is finished, the item will change into the blueprint. When the player related to it will leave, or remove the blueprint from his library, the item will just disappear. It can even have nice info into tooltip about what is going on. This is (obviously) also going to be in 0.16 not 0.15.

Generally speaking, the blueprint library is a new thing to Factorio, so there several other UX tweaks and polishes we want to do.

The rail segment visualisation that I described in FFF 198 is now completed.The main problem I had to solve is how to color the segments to make sure, that two neighbour segments don't have the same color. Some of you probably know, that this is well known thing called graph coloring. For plain graphs (which the rail segments are), it has been proven long time ago, that 4 colors is enough to color every graph, but the proof and algorithm is not simple. But for 6 colors, the algorithm is pretty easy, the only downside is, that it is designed to color the graph as a whole, which is fine when the game is being loaded, but not acceptable when adding a single signal to a huge rail network, I needed something incremental.

In the meantime I wanted to test the visualisation graphics proposal from Vaclav, so I made a very simple logic that picks the lowest available color index that is not already taken by neighbour rails to see how it looks. I was surprised to find out that in normal rail system, it doesn't use more than 3 colors most of the time, and it is very rare to see the 5th color . The conclusion is simple, this is good enough. Although it is possible to construct rail system that results in our algorithm putting two of the same colors next to each other, it is not really a problem, as it doesn't break anything, and in a normal game, the chance of it happening is close to 0. The takeaway is that it is important to realize how strict you need to depending on the situation.

Here is an example of intersection with a missing signal, how fast can you find it the old way?



How fast can you find it when you see the visualisation?



I guess that this topic is closed, the fact that 2-3 colors are used most of the time makes the visualisation not look too colorful and it will help both new players to understand the concept of blocks as well as help experienced players when signalling an intersection to see what is going on at a glance.

There's always something else that can be optimized. Last week we purchased some new hardware because I wanted to test if 'throwing money at it' was a viable method of improving compilation times (and the speed at which we can develop/fix things). We purchased a brand new i9-7900X CPU and compatible hardware with great results. After some difficulties getting everything setup it was almost 150% faster than the previous setup I was using. After that significant improvement in compilation times that got me wondering what else I do multiple times per day development wise that I could speed up. That led to game startup time, map loading/saving, and map generation.I grabbed a relatively large save file with a large factory setup (Colonelwill's current map) and set out to get a baseline. The save file is 45.7 MB compressed and 132 MB uncompressed. In 0.15.31 it takes 3.89 seconds to load and 2.46 seconds to save.

I think I can safely say that nobody likes waiting for the game to save (myself included); it's distracting and breaks you out of whatever you're currently doing in the game. While working on potential improvements to save time I measured exactly what happens when I tell the game to save and the results gave me a little more appreciation for just how fast saving actually is.

Technical/numbers
The way saving works in Factorio is through a process called serialization. We go over the entire loaded map beginning to end and write out each piece that's needed to restore the map. Every time something needs to be saved it calls the seralizer method 'write' with the data and how many bytes it should serialize. On the 132 MB of data that makes up the uncompressed map the 'write' method on the seralizer is called 84'272'542 times. Of those 84 million calls to write 99% are < 10 bytes.
The largest was only 121 bytes (called 84 times). In total 152'472'991 bytes were processed making for an average of 1.8 bytes per call. The entire process completed in 2.46 seconds for an average of 334'257 calls per millisecond. The serializer we use for saving the map uses several memory buffers that it cycles internally as it writes one buffer to disk in parallel with the rest of the map saving logic so there wasn't much performance to gain in this logic. But, seeing the numbers gave me a better picture of just how much work happens when the map is saved. In the end I found a few small things to improve on such that saving the map in 0.16 now only takes 2.25 seconds (a 9% improvement). Not what I hoped for but an improvement is an improvement.

Map load time
When looking at the time spent loading maps I found that we had a similar memory buffer system to saving but with 1 major difference: when the buffer ran out it would stop and refill the buffer from disk. That was an obvious slow spot with an easy fix: use multiple buffers and load them from disk in parallel with the map deserialization. I also found some inefficiencies in how trains were loaded causing them to run O(Rails * Trains) operations every time the map was loaded which I fixed to just O(Trains). The end result being what took 3.89 seconds to load in 0.15 now takes 2.8 seconds to load in 0.16 (a 28% improvement).

Game startup time - Graphics
On this new computer with normal graphics quality Factorio takes 9.84 seconds to reach the main menu. I think that's pretty good for a game these days but as we work on Factorio we close and restart the game 100s of times per day. Every time we change something, test a save file from a bug report, or as I was doing: testing out changes to see how they impact performance I have to wait for the game to start and it's wasted time. There's a disabled config option that takes all of the loaded graphics the game uses and packs them into 1 large file that can be read at startup. It mostly helps when you frequently restart the game - which is exactly what we do every day. With that enabled the game launches in 5.31 seconds (a 46% improvement).

Game startup time - Mods
We don't tend to use mods while we're working on new features so some performance problems related to them go unnoticed. One bug report recently had 111 mods the person was using which exposed a slowdown in processing all of the mods at startup. When mods are loaded we track what a mod changes, to record the history of what mods change what game objects. Originally we did this using the C++ Boost library "property tree" class, but it turned out to be faster to write our own logic using Lua than use the Boost class. Since that change we wrote our own version of the property tree class and in testing it showed to be even faster than doing the logic inside Lua. In 0.15.31 it took 28.72 seconds to process the mods and now in 0.16 it only takes 20.2 seconds (a 29% improvement). Not only does this benefit the players but it helps us because we don't have to spend as much time waiting for the game to load when working on modded bug reports.

Game startup time - Sounds
With the loading of graphics and mod data no longer being the slowest parts the next thing that started taking the majority time was loading sounds. I realized, after my work on improving map loading, that there wasn't any reason I couldn't just load sounds in parallel with the graphics. It was a 20 minute 'fix' and now by the time graphics are finished loading, sounds have long finished meaning it can proceed straight to the main menu.

Game startup time - Conclusion
With graphics being loaded from the atlas-cache, mod data processing being almost twice as fast, and sounds loaded in parallel I can now launch the game and be at the main menu ready to play in just 2.962 seconds (a 44% improvement). The change was interesting to get used to over the past week but now when I need to test things in 0.15 I find myself annoyed at how long everything takes compared to the 'new hotness' of 0.16.

Map generation - New maps
Unless I'm working on some specific save file I generate maps almost as frequently as I restart the game while working on bug fixes/new features. I've never considered generating a new map particularly slow but it never felt fast either. 0.15 already generates the map in the background while the game runs to prevent stuttering while new chunks are generated but the starting area when making a new map is not. I changed it so the starting area would generate in multiple threads if possible (while still maintaining determinism). The end result being: 0.15.31 generates a new map in 1.83 seconds and 0.16 generates a new map in 0.8 seconds (a 56% improvement).

Conclusion
Overall the start-game/load/generate game experience is measurably faster. Most importantly it feels noticeably faster, and makes for a better overall experience when we don't have to wait for things. With the belt optimizations and everything mentioned here, 0.16 is already promising to improve performance in multiple areas and full 0.16 development hasn't even begun, with other ideas we've talked about in previous Friday Facts still to be implemented promising even better performance.

As always, leave us any feedback or comments over on our forum