Hello,

another friday and another Facts. It has been 3 years already without a single friday facts missing. I didn't expect that!

Our automated integration test count just reached 800. We have several modes to run these. One of the modes is to generate crc hash values of all the steps, which can be compared with values when the tests are started on different platform, or release type.
When doing this, I noticed that there are several problems related to difference between 32 and 64 bits and after solving few of them, I realized that doing this kind of work might be actually not useful usage of our time.

So I made a list of reasons to not have 32 bit support anymore:

• Extra issues related to determinism between 32 bit and 64 bit systems that would have to be solved.
• Release times (compile, upload, updates, server storage space)
• Occasional bugs that we don't notice as no one uses 32 bit system here.
• Confusion of some people that download 32 bit by accident and run out of memory soon.

As steam allows me to check the hardware survey of users playing factorio, I did that and found out that less than 1% of people playing Factorio have 32 bit system. That is not that much considering, that a lot of them have different device capable to run the game as well. I believe, that saving our time so we can spend it on things the 99% of our players appreciate more is the right decision.

The conclusion is to disable multiplayer in 32 bit version right away (0.14.10), so we don't have to deal with desync reports related to 32 versus 64 bit systems and since 0.15 we won't release 32 bit version at all.

Desyncs are not that common, but it still happens from time to time, and if it is a desync loop (anyone new that joins the game desyncs until server is restarted) it is definitely annoying problem. It is probably caused by last few determinism oversights which are harder to find, but had to be solved anyway. There are 2 fundamental problems with our desync reports now:

1) When the desync happens in multiplayer game, the player needs to tell the server that it desynced. This takes time as the packet has to travel to the server. Then the client has to update the game until it is in the same tick as the server at the time of receiving the desync alarm. At this point, client and server have the map in the same tick, and this is the version saved in the desync report. The problem is, that it is quite long time since the first original difference between server and client happened, so huge amount of other differences accumulated as the butterfly effect was spreading the difference. It is mostly nearly impossible the find the original cause of the problem from such a desync report most of the time.

We will solve this problem by a change in the server logic. Whenever the server encounters the desync, it will try to check the save-load stability locally. We actually use this in the tests. This means that the server will save the current version of the map, it then loads and saves second instance of the map from it and compares if it stays the same. This report will not contain any accumulated changes from running different version of the save, so only the original cause of the problems will appear in the diff. This will help us to discover the cyclic desync loops, as they are almost always related to the fact that load-save changes some internal state.

2) We have a special tool that can add human readable tags to the binary files, so we can actually see what kind of data is different. This code is not part of the binary we release, so we have to add these tags to the desync report by loading the save and saving it with the tags. The problem is, that some of the desync issues (especially those related to desync loop) are related to the save-load stability, so the information gets lost in this process. So we will make a change, that in the future versions of Factorio we release will be able to save the desync report in the human readable format directly.

I hope that these changes should make it possible for us to solve the rest of the desync problems with much less effort.

The rail sprites we used had one big problem. To save some video memory we were flipping the pictures:





The problem is, that as the pictures need to be flippable, they need to be very flat, or it would look too weird. We decided to fix it for 0.15 while doing the high res version of the rails.

This is a work in progress of the new rail sprites. Please note that it is at a very raw stage that doesn't even have textures, but it can be used to demonstrate how important it is to have specific pictures for different rotations:



We will show the final version of rails in some of the future friday facts.

As always, if you have any comments or otherwise, please let us know on our forums.

Hello,

another friday and another Facts. It has been 3 years already without a single friday facts missing. I didn't expect that!

Our automated integration test count just reached 800. We have several modes to run these. One of the modes is to generate crc hash values of all the steps, which can be compared with values when the tests are started on different platform, or release type.
When doing this, I noticed that there are several problems related to difference between 32 and 64 bits and after solving few of them, I realized that doing this kind of work might be actually not useful usage of our time.

So I made a list of reasons to not have 32 bit support anymore:

• Extra issues related to determinism between 32 bit and 64 bit systems that would have to be solved.
• Release times (compile, upload, updates, server storage space)
• Occasional bugs that we don't notice as no one uses 32 bit system here.
• Confusion of some people that download 32 bit by accident and run out of memory soon.

As steam allows me to check the hardware survey of users playing factorio, I did that and found out that less than 1% of people playing Factorio have 32 bit system. That is not that much considering, that a lot of them have different device capable to run the game as well. I believe, that saving our time so we can spend it on things the 99% of our players appreciate more is the right decision.

The conclusion is to disable multiplayer in 32 bit version right away (0.14.10), so we don't have to deal with desync reports related to 32 versus 64 bit systems and since 0.15 we won't release 32 bit version at all.

Desyncs are not that common, but it still happens from time to time, and if it is a desync loop (anyone new that joins the game desyncs until server is restarted) it is definitely annoying problem. It is probably caused by last few determinism oversights which are harder to find, but had to be solved anyway. There are 2 fundamental problems with our desync reports now:

1) When the desync happens in multiplayer game, the player needs to tell the server that it desynced. This takes time as the packet has to travel to the server. Then the client has to update the game until it is in the same tick as the server at the time of receiving the desync alarm. At this point, client and server have the map in the same tick, and this is the version saved in the desync report. The problem is, that it is quite long time since the first original difference between server and client happened, so huge amount of other differences accumulated as the butterfly effect was spreading the difference. It is mostly nearly impossible the find the original cause of the problem from such a desync report most of the time.

We will solve this problem by a change in the server logic. Whenever the server encounters the desync, it will try to check the save-load stability locally. We actually use this in the tests. This means that the server will save the current version of the map, it then loads and saves second instance of the map from it and compares if it stays the same. This report will not contain any accumulated changes from running different version of the save, so only the original cause of the problems will appear in the diff. This will help us to discover the cyclic desync loops, as they are almost always related to the fact that load-save changes some internal state.

2) We have a special tool that can add human readable tags to the binary files, so we can actually see what kind of data is different. This code is not part of the binary we release, so we have to add these tags to the desync report by loading the save and saving it with the tags. The problem is, that some of the desync issues (especially those related to desync loop) are related to the save-load stability, so the information gets lost in this process. So we will make a change, that in the future versions of Factorio we release will be able to save the desync report in the human readable format directly.

I hope that these changes should make it possible for us to solve the rest of the desync problems with much less effort.

The rail sprites we used had one big problem. To save some video memory we were flipping the pictures:





The problem is, that as the pictures need to be flippable, they need to be very flat, or it would look too weird. We decided to fix it for 0.15 while doing the high res version of the rails.

This is a work in progress of the new rail sprites. Please note that it is at a very raw stage that doesn't even have textures, but it can be used to demonstrate how important it is to have specific pictures for different rotations:



We will show the final version of rails in some of the future friday facts.

As always, if you have any comments or otherwise, please let us know on our forums.

• Changes
  
  • Server started with --start-server-load-scenario will now save to saves/<scenario name> upon exit.
    
  • Multiplayer usernames can only consist of letters, numbers and -_. characters.
    
  
  
• Bugfixes
  
  • Possible fix for a server not responding error.
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Changes
  
  • Server started with --start-server-load-scenario will now save to saves/<scenario name> upon exit.
    
  • Multiplayer usernames can only consist of letters, numbers and -_. characters.
    
  
  
• Bugfixes
  
  • Possible fix for a server not responding error.
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Changes
  
  • Disabled multiplayer support in the 32 bit version of the game.
    
  • Multiplayer map downloading GUI shows average over the last 2 seconds instead of 20 seconds.
    
  
  
• Bugfixes
  
  • Potential fix of the crash after desync.
    
  • Fixed desync related to electric network statistics of accumulators.
    
  • Fixed assertion fail in tightspot level-05 more
    
  • Removed 2 redundant research prerequisites. more
    
  • Fixed non even selection boxes of locomotive versus cargo wagon. more
    
  • Fixed that Factorio would crash when unable to delete a mod. more
    
  • Fixed that the chart would display user's names from other surfaces. more
    
  • Fixed blueprint tiles alignment inside preview/editor. more
    
  • Fixed player limit disallowing connection attempt in public server list. more
    
  • Fixed mod browser filters and ordering being lost. more
    
  • Fixed freshly installed mods not being marked in the mod browser. more
    
  • Fixed signals sometimes not finding their child signals properly. more
    
  • Fixed up/down keys in schedule gui were not selecting properly. more
    
  • Additional keys ignored when the console is opened. more
    
  • Fixed that aliens would ignore very narrow paths they could barely fit through. more
    
  • Fixed that numbers were sometimes rounded differently on 32 bit systems in lua.
    
  • Fixed that the game would freeze when game.speed was set to a value less than 0.2. more
    
  • Fixed cargo wagon air resistance being smaller than that of the locomotive.
    
  • Fixed of loading "require_user_verification" from server-settings (It was called verify_user_identity at one place and require_user_verification at other).
    
  • Fixed underground belt fast replace when replacing the same belt multiple times. more
    
  • Fixed rendering of turret range when construction network area overlay is also being drawn. more
    
  • Temporary decrease of decorative entities counts.
    
  • Fixed server max upload speed.
    
  • Fixed that Factorio wouldn't give an error when --start-server-load-scenario was combined with --start-server or --start-server-load-latest. more
    
  • Fixed beam would apply one extra damage tick. more
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Changes
  
  • Disabled multiplayer support in the 32 bit version of the game.
    
  • Multiplayer map downloading GUI shows average over the last 2 seconds instead of 20 seconds.
    
  
  
• Bugfixes
  
  • Potential fix of the crash after desync.
    
  • Fixed desync related to electric network statistics of accumulators.
    
  • Fixed assertion fail in tightspot level-05 more
    
  • Removed 2 redundant research prerequisites. more
    
  • Fixed non even selection boxes of locomotive versus cargo wagon. more
    
  • Fixed that Factorio would crash when unable to delete a mod. more
    
  • Fixed that the chart would display user's names from other surfaces. more
    
  • Fixed blueprint tiles alignment inside preview/editor. more
    
  • Fixed player limit disallowing connection attempt in public server list. more
    
  • Fixed mod browser filters and ordering being lost. more
    
  • Fixed freshly installed mods not being marked in the mod browser. more
    
  • Fixed signals sometimes not finding their child signals properly. more
    
  • Fixed up/down keys in schedule gui were not selecting properly. more
    
  • Additional keys ignored when the console is opened. more
    
  • Fixed that aliens would ignore very narrow paths they could barely fit through. more
    
  • Fixed that numbers were sometimes rounded differently on 32 bit systems in lua.
    
  • Fixed that the game would freeze when game.speed was set to a value less than 0.2. more
    
  • Fixed cargo wagon air resistance being smaller than that of the locomotive.
    
  • Fixed of loading "require_user_verification" from server-settings (It was called verify_user_identity at one place and require_user_verification at other).
    
  • Fixed underground belt fast replace when replacing the same belt multiple times. more
    
  • Fixed rendering of turret range when construction network area overlay is also being drawn. more
    
  • Temporary decrease of decorative entities counts.
    
  • Fixed server max upload speed.
    
  • Fixed that Factorio wouldn't give an error when --start-server-load-scenario was combined with --start-server or --start-server-load-latest. more
    
  • Fixed beam would apply one extra damage tick. more
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

Hello,
we have written a lot about multiplayer in the past weeks, it started by the reports at the beginning of the rewrite, and now it finishes with our reports of mega games being played. But as most of our time still goes to bug hunting to achieve 0.14 stable as soon as possible, I just can't help myself:

As we watched the stream of big multiplayer we mentioned in the last fff, we decided to participate and help organize another big multiplayer event. Even though the server provider we rented from for the event was DDoS'd at that time, it was still a success. During the event, we did profiling as the game was running, and we learned a lot of valuable information.



The Factorio internals always have to be programmed with knowledge of quantities. Transport belts need to be optimised a lot, as there are tons of those on the map and they eat a lot of the CPU time. On the other hand, the time to simulate the players was never really a problem, since there were always at most a few players on the map at the same time. This paradigm was changed a lot when we profiled a replay with 300 players and found out that big part of the CPU time is spent on things like searching for gates to be opened or searching for items to be looted etc. These things are not hard to optimise, but we just didn't know we should do it until now, so it is work in progress.

Several other needs emerged, like changing the server settings (password/max upload rate/max players) on the fly as the game is played. Postponement of the map saving for incoming players, so the server saves once in a while for more people at the same time, instead of saving every 3 seconds when someone wants to join a crowded game. The admin options also had to be improved etc. etc.
Then it comes to further and further optimisations of the network traffic so more and more people can play at the same time. The problem with this is, that we could do it forever, we wanted 20 or at most 50 players to be able to play in the new multiplayer, but we reached 350 and now we are optimising it so more than 400 could join, and if we achieved that, we would go for 1000. But are we still doing something that improves the gameplay, or just trying to win the race with ourselves at this point? We need to realize that our goal is not to make some MMO, but to have a good single player game, that also have a decent multiplayer. So the conclusion is, that if the game supports 200 players with a good connection it is more than enough. It can go higher but we won't aim to increase it much anymore.
After hour or two, the main bottleneck is not the players or the network traffic. The factory grows at a cosmic speed in these megagames, so the CPU time needed to simulate it becomes the problem. By changing our focus to general factory simulation optimisations, we not only make it possible to play massive multiplayer games better, but we also allow to build much bigger bases in singleplayer as well. The belts optimisations and multithreading are worked on, but far from being finished.

As we have been asked many times, I probably need to answer again: Yes, we will add the liquid wagons in 0.15, but it won't be that simple. As we want to have the pump and wagon connection to be organic, we need the pump to connect to the wagon. But as the geometry of our trains is wicked the connection points on the wagon won't be regular in vertical directions. We are making a 2d game so we have to prepare 6 different animations for every relative position of the pump. Albert had doubts if it is worth it as we always have to try to keep our video memory bill reasonable, but I believe it will add a nice touch to the game.




One of the 0.13 changes was change of the map generation. As part of it, it happened that the game is suddenly generating a lot of more of the decorative entities. It often looks like this:

It looks nicer, but the problem is that there could be tens of millions of these on big maps, which can multiply the size of the save and ram required to hold it.There is even a mod that removes them completely to solve the issue. We were discussing solutions of this, and we will probably just change decoratives so they are not regular entities anymore. They would be a special type of an object that is not saved in the map, but generated on the fly when the player walks around. They would be saved per chunk in a very compressed format so it only takes 3 bytes of memory per decorative instead of approximately 300 bytes when it exists as an entity. This will also improve performance, as when some code is checking entities in some perimeter, it wouldn't have to iterate through tons of these decoratives uselessly. As these changes are too big to be added into the 0.14, we will probably just reduce decorative generation in 0.14 and do this change for 0.15.

As always, if you have any comments or otherwise, please let us know on our forums.

Hello,
we have written a lot about multiplayer in the past weeks, it started by the reports at the beginning of the rewrite, and now it finishes with our reports of mega games being played. But as most of our time still goes to bug hunting to achieve 0.14 stable as soon as possible, I just can't help myself:

As we watched the stream of big multiplayer we mentioned in the last fff, we decided to participate and help organize another big multiplayer event. Even though the server provider we rented from for the event was DDoS'd at that time, it was still a success. During the event, we did profiling as the game was running, and we learned a lot of valuable information.



The Factorio internals always have to be programmed with knowledge of quantities. Transport belts need to be optimised a lot, as there are tons of those on the map and they eat a lot of the CPU time. On the other hand, the time to simulate the players was never really a problem, since there were always at most a few players on the map at the same time. This paradigm was changed a lot when we profiled a replay with 300 players and found out that big part of the CPU time is spent on things like searching for gates to be opened or searching for items to be looted etc. These things are not hard to optimise, but we just didn't know we should do it until now, so it is work in progress.

Several other needs emerged, like changing the server settings (password/max upload rate/max players) on the fly as the game is played. Postponement of the map saving for incoming players, so the server saves once in a while for more people at the same time, instead of saving every 3 seconds when someone wants to join a crowded game. The admin options also had to be improved etc. etc.
Then it comes to further and further optimisations of the network traffic so more and more people can play at the same time. The problem with this is, that we could do it forever, we wanted 20 or at most 50 players to be able to play in the new multiplayer, but we reached 350 and now we are optimising it so more than 400 could join, and if we achieved that, we would go for 1000. But are we still doing something that improves the gameplay, or just trying to win the race with ourselves at this point? We need to realize that our goal is not to make some MMO, but to have a good single player game, that also have a decent multiplayer. So the conclusion is, that if the game supports 200 players with a good connection it is more than enough. It can go higher but we won't aim to increase it much anymore.
After hour or two, the main bottleneck is not the players or the network traffic. The factory grows at a cosmic speed in these megagames, so the CPU time needed to simulate it becomes the problem. By changing our focus to general factory simulation optimisations, we not only make it possible to play massive multiplayer games better, but we also allow to build much bigger bases in singleplayer as well. The belts optimisations and multithreading are worked on, but far from being finished.

As we have been asked many times, I probably need to answer again: Yes, we will add the liquid wagons in 0.15, but it won't be that simple. As we want to have the pump and wagon connection to be organic, we need the pump to connect to the wagon. But as the geometry of our trains is wicked the connection points on the wagon won't be regular in vertical directions. We are making a 2d game so we have to prepare 6 different animations for every relative position of the pump. Albert had doubts if it is worth it as we always have to try to keep our video memory bill reasonable, but I believe it will add a nice touch to the game.




One of the 0.13 changes was change of the map generation. As part of it, it happened that the game is suddenly generating a lot of more of the decorative entities. It often looks like this:

It looks nicer, but the problem is that there could be tens of millions of these on big maps, which can multiply the size of the save and ram required to hold it.There is even a mod that removes them completely to solve the issue. We were discussing solutions of this, and we will probably just change decoratives so they are not regular entities anymore. They would be a special type of an object that is not saved in the map, but generated on the fly when the player walks around. They would be saved per chunk in a very compressed format so it only takes 3 bytes of memory per decorative instead of approximately 300 bytes when it exists as an entity. This will also improve performance, as when some code is checking entities in some perimeter, it wouldn't have to iterate through tons of these decoratives uselessly. As these changes are too big to be added into the 0.14, we will probably just reduce decorative generation in 0.14 and do this change for 0.15.

As always, if you have any comments or otherwise, please let us know on our forums.

• Changes
  
  • Added admin field to server-settings.json, list of case-sensitive usernames that will become admins on connecting.
    
  • Admins are exempt from player count limit.
    
  
  
• Bugfixes
  
  • Fixed that the server could be running even if it was supposed to be stopped.
    
  • Fixed Modded key bindings would fire extra times if 2 mods had the same keybinding. more
    
  • Fixed that parsing players by index in scripts didn't work correctly. more
    
  • Fixed that Factorio wouldn't release memory back to the OS after unloading a large save on Linux. more
    
  • Fixed crash when restarting Factorio due to mod change when executable is read-only on Linux. more
    
  • Game will no longer be capped at 300UPS when using high game.speed and vsync. New cap is FPS*5*game.speed. more
    
  • Fixed possible crashes when setting combinator parameters.
    
  • Fixed that clients could send a lot of useless data when recovering after connectiontion problems while playing multiplayer game.
    
  • Fixed that recipe overload_multiplier wasn't used for furnaces. more
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Changes
  
  • Added admin field to server-settings.json, list of case-sensitive usernames that will become admins on connecting.
    
  • Admins are exempt from player count limit.
    
  
  
• Bugfixes
  
  • Fixed that the server could be running even if it was supposed to be stopped.
    
  • Fixed Modded key bindings would fire extra times if 2 mods had the same keybinding. more
    
  • Fixed that parsing players by index in scripts didn't work correctly. more
    
  • Fixed that Factorio wouldn't release memory back to the OS after unloading a large save on Linux. more
    
  • Fixed crash when restarting Factorio due to mod change when executable is read-only on Linux. more
    
  • Game will no longer be capped at 300UPS when using high game.speed and vsync. New cap is FPS*5*game.speed. more
    
  • Fixed possible crashes when setting combinator parameters.
    
  • Fixed that clients could send a lot of useless data when recovering after connectiontion problems while playing multiplayer game.
    
  • Fixed that recipe overload_multiplier wasn't used for furnaces. more
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.