Hello,

the 1st of June, which is the goal of the 0.13 release is starting to feel uncomfortably close, especially if I want to start play-testing 2 weeks before the release date. There are a lot of bigger and smaller tasks appearing all over the place. It is now the time of moving some of them to the next release again. Many of them are moved like this for years already.

I have to finish the Friday facts soon today, as we are going on a trip to Třeboň, where the Czech game of the year ceremony is held. Factorio is nominated on 3 categories, so there is a chance to win something, but mainly, we will have the opportunity to talk and party with other game developers, which doesn't happen often.

The final decision has been made and instead of a loader, we will have not heavy, but rapid inserter. To make things straightforward, I decided to limit the inserter stack size bonus to the rapid inserter only. Some people certainly won't like it, as they won't be able to use long handed inserters or burner inserters to take advantage of this bonus, but the advantages outweigh it.

• Streamlined definition of the inserter stack size bonus, now it increases rapid inserter capacity whatever is it doing.
  
• The rapid inserter will have its specific usages. As the rapid inserter currently waits until its hand is filled with items, it is not clearly better than fast inserter. When the bonus is 5, and it has 4 items in hand, it can wait minutes (or forever) for the last item, which might be undesirable especially with expensive, low throughput items. This implies that rapid inserters will eventually become stuck on mixed belts as long as some explicit mechanism to prevent it won't be there.

To make it useful, we will increase the maximum rapid inserter capacity (instead of inserter stack size bonus) from 4 to 11, so it can hold up to 12 items of the same kind.

Four fast inserters can be easily matched by one fully upgraded rapid inserter:


Four fully upgraded rapid inserters can almost fully compress an express belt:


Another one of the ease of use additions for 0.13, is the train station visualization. Whenever you are building next to a rail that leads to a station, you can see the future train wagon positions. This simplifies the building of the stations a lot.



The visualization is also helpful when building the train stop. You can immediately see, how many wagons can fit in the platform. It is combined with the indicator of possible train station placements. These are limited by the direction in a similar way as rail signals are. With this you won't build the station on the other side of the rail by accident anymore.



I encountered this cute picture by ironhand on our forums, and I just had to share it with you.



As always, let us know what you think on our forums.

Hello,

the 1st of June, which is the goal of the 0.13 release is starting to feel uncomfortably close, especially if I want to start play-testing 2 weeks before the release date. There are a lot of bigger and smaller tasks appearing all over the place. It is now the time of moving some of them to the next release again. Many of them are moved like this for years already.

I have to finish the Friday facts soon today, as we are going on a trip to Třeboň, where the Czech game of the year ceremony is held. Factorio is nominated on 3 categories, so there is a chance to win something, but mainly, we will have the opportunity to talk and party with other game developers, which doesn't happen often.

The final decision has been made and instead of a loader, we will have not heavy, but rapid inserter. To make things straightforward, I decided to limit the inserter stack size bonus to the rapid inserter only. Some people certainly won't like it, as they won't be able to use long handed inserters or burner inserters to take advantage of this bonus, but the advantages outweigh it.

• Streamlined definition of the inserter stack size bonus, now it increases rapid inserter capacity whatever is it doing.
  
• The rapid inserter will have its specific usages. As the rapid inserter currently waits until its hand is filled with items, it is not clearly better than fast inserter. When the bonus is 5, and it has 4 items in hand, it can wait minutes (or forever) for the last item, which might be undesirable especially with expensive, low throughput items. This implies that rapid inserters will eventually become stuck on mixed belts as long as some explicit mechanism to prevent it won't be there.

To make it useful, we will increase the maximum rapid inserter capacity (instead of inserter stack size bonus) from 4 to 11, so it can hold up to 12 items of the same kind.

Four fast inserters can be easily matched by one fully upgraded rapid inserter:


Four fully upgraded rapid inserters can almost fully compress an express belt:


Another one of the ease of use additions for 0.13, is the train station visualization. Whenever you are building next to a rail that leads to a station, you can see the future train wagon positions. This simplifies the building of the stations a lot.



The visualization is also helpful when building the train stop. You can immediately see, how many wagons can fit in the platform. It is combined with the indicator of possible train station placements. These are limited by the direction in a similar way as rail signals are. With this you won't build the station on the other side of the rail by accident anymore.



I encountered this cute picture by ironhand on our forums, and I just had to share it with you.



As always, let us know what you think on our forums.

• Bugfixes
  
  • Fixed tiles in blueprints. more
    
  • Fixed the duplicated message on load error in multiplayer. more
    
  • Fluid values are rounded to the closest value instead of rounding down when transmitted to circuit network. more
    
  • Fixed that switching brush shape in the map editor didn't update the selected icon highlight. more
    
  • Fixed that label/checkbox padding was applied twice, moving the rendered text to be out of place. more
    
  • Fixed that map (chart) wasn't properly updated when existing map was edited in the map editor. more
    
  • Fixed crash when the character would die from mining something. more
    
  • Fixed crash when robot can't find charging spot when stationing. more
    
  • Fixed crash in multiplayer that could happen when 2 people are editing the same train schedule. more
    
  • Unified the electric network statistics production/consumption values to be avarage of the shown timeframe. more
    
  • Solved small numerical text box editing issue. more
    
  
  
• Scripting
  
  • Added LuaStyle::cell_padding/horizontal_spacing/vertical_spacing.
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Bugfixes
  
  • Fixed tiles in blueprints. more
    
  • Fixed the duplicated message on load error in multiplayer. more
    
  • Fluid values are rounded to the closest value instead of rounding down when transmitted to circuit network. more
    
  • Fixed that switching brush shape in the map editor didn't update the selected icon highlight. more
    
  • Fixed that label/checkbox padding was applied twice, moving the rendered text to be out of place. more
    
  • Fixed that map (chart) wasn't properly updated when existing map was edited in the map editor. more
    
  • Fixed crash when the character would die from mining something. more
    
  • Fixed crash when robot can't find charging spot when stationing. more
    
  • Fixed crash in multiplayer that could happen when 2 people are editing the same train schedule. more
    
  • Unified the electric network statistics production/consumption values to be avarage of the shown timeframe. more
    
  • Solved small numerical text box editing issue. more
    
  
  
• Scripting
  
  • Added LuaStyle::cell_padding/horizontal_spacing/vertical_spacing.
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Fixed loading of electric network of pre 0.12 saves.
  
• Fixed blueprint building. Blueprints are migrated as long as you load the version 0.12.30 or earlier, existing bluepritns might be off if you resaved in 0.12.31 already.

• Lua interface to create blueprint now expects the entity positions to be relative to center exactly, so in rail-less blueprints, the position 0.0 translates to center of the tile when the blueprint is built.

Use the automatic updater if you can (check experimental updates in other settings) or download full installation at http://www.factorio.com/download/experimental.

• Fixed loading of electric network of pre 0.12 saves.
  
• Fixed blueprint building. Blueprints are migrated as long as you load the version 0.12.30 or earlier, existing bluepritns might be off if you resaved in 0.12.31 already.

• Lua interface to create blueprint now expects the entity positions to be relative to center exactly, so in rail-less blueprints, the position 0.0 translates to center of the tile when the blueprint is built.

Use the automatic updater if you can (check experimental updates in other settings) or download full installation at http://www.factorio.com/download/experimental.

This week's instance of FFF is brought to you by cube, your friendly neighbor
clueless network programmer. This post will be more technical than usual, so let us know if you found it interesting.


The week has been spent mostly on pushing forward in multiple of ongoing areas in the development for 0.13. Robert is working on Combinators, Posila and Vaclav on Fire and Flamethrower Turret, Martin is immersed in Factorio modportal and its integration and Tomas and Ondra have been tweaking some Multiplayer Matching server stuff. While Kovarex has spent most time on the 0.12 bugfixing. Albert is back home for a week so the full speed gfx work will happen from next week on.


Factorio has long had problems with map upload speed.
As usual the hard problems they always appear "out there" and never when you're looking for them,
so while we were getting a steady stream of bug reports about this problem, we
could never replicate it.

You might ask why do we even need to upload the whole map when the player might
end up seeing just a tiny part of the world anyway.
AAAAnd the answer is: That's just how it works :-).
Essentially it's because Factorio runs a lockstep simulation and every player needs
to have complete information about the whole map.
This has been discussed in slightly more detail in
an older blogpost.

Now that we know that we really must copy all the data, the question is how to do
it well.


As you probably already knew, the Internet works by passing small chunks of data
called packets
(or frames or segments or datagrams, depending on where exactly you look).
These are nothing special, just some bytes wrapped in an envelope made out of other bytes.
The network that these packets travel is a dangerous place, packet can get lost,
delayed, or even duplicated along the way.

To keep people sane and networks fast, someone came up with a way to hide all these
complications (because that's what programmers do) and invented the TCP protocol.
Instead of transferring unreliable packets, it provides a stream of bytes
and the machinery inside wraps pieces of the stream to packets and makes sure that
all bytes arrive as sent.

Which sounds suspiciously similar to moving files over the network, right?
Just pop the whole thing into the stream and let the elfs do their magic.


Unfortunately in Factorio nothing is ever easy :-).
Using TCP would have three important consequences: It would force players to open
TCP ports as well as UDP ports in firewalls and NAT servers, it would remove the
possibility of using NAT punching now or later with MP matching server and it would
cause problems with downloading from multiple players at the same time.

So we decided to have file transfers done over UDP, even though it means reimplementing
a significant part of TCP functionality.

To make downloading from several other players possible in peer to peer mode,
the file transfers work by requesting individual blocks from peers and waiting for
the data, instead of sending data and waiting for acknowledgements.
Also we don't really care about ordering of the data blocks as long as all of them
are eventually transmitted.


When we started implementing the file transfer all I had was a rough idea of
"we're sending some packets and waiting for confirmations, somehow we know how many unconfirmed
packets there should be at one time and that value is called cWnd".
So Wikipedia came to the rescue and told me that a prety decent algorithm to determine
cWnd is called CUBIC.
So I read an article or two, coded it up and ... it worked! Let's ship it!

A few releases, a few fixes, and a few desperate kludges later it worked very reliably
on my machine and the code got kind of forgotten.

About three weeks ago we decided to revisit the map transfers and I've spent two
fun weeks in TCP land.


The significant part of TCP functionality mentioned earlier is called congestion avoidance.

When you are sending some packets into the network you usually don't know in advance
how fast the network is or how long it will take to reach the other end and
if too many packets are stuffed into the connection not only some of them will be
dropped, but also packets belonging to other connections that go through the same
part of network will get dropped.
This situation is called network congestion and generally it's a thing we want to avoid,
because it slows the available transfer speed for everyone.

So the main idea behind congestion avoidance TCP is to send some amount data and
wait for acknowledgements before sending more.
If the acknowledgement arrives, we know that sending this amount of data is OK and
we try to send a little more next time, if on the other hand the packets start
disappearing we know we are sending too much and we should slow down.

Because of only a little complicated reasons, the increase of the window size
is a roughly linear function of time and the decrease is immediate halving, this
algorithm is called AIMD (additive increase, multiplicative decrease).
This way the amount of data in the network is oscillating between no congestion
at all and very minor congestion, slowly going up and then suddenly jumping down
to half speed.


Our first guess for the reasons of the slow transfer speeds was packet loss.
Both AIMD and CUBIC decrease the transfer speed drastically after a packet is lost
so when your microwave decides that the WiFi connection is its mortal enemy,
the congestion avoidance algorithm will slow down.

The measurements confirmed this, but compared to TCP, our file transfers were always
a little bit faster when packet loss was artificially added.

After an embarrassingly long period of experimentation I was left with two implementations.
AIMD, based mainly on RFC 5681 and
my own attempt slightly inspired by Sync-TCP and LEDBAT,
but other than that completely original.
And neither of them worked too well :-).

With my custom protocol that was not unexpected, it was mostly meant for experimenting
and to get a feeling for all the different behaviors the network can cause.

On the other hand why the AIMD implementation was not working was a mystery.
Finally after finding RFC 2525
(thank you, internet people from 1999), I figured out that all problems in my life
were caused by a missing fast retransmit / fast recovery mechanism.

Fast retransmit and fast recovery work by marking a packet as lost not when its
timeout expires, but when three packets sent later are confirmed instead of the
packet in question.
In this case the packet is retransmitted and fast recovery mode is entered instead of waiting
a whole second, overflowing the network with extra packets the whole time.

After these changes we've upgraded the algorithm to be very similar to
TCP Westwood which finally
gave us the improved behavior on lossy networks.

So TADAAA, here we are now.
The map transfer protocol is less tested than the previous one, but we feel more
certain about it, so let's hope that it's going to be the last change necessary
to this part of networking code.

Map download was slow, cube spent some time doing computer stuff, now it should be faster.

Let us know what you think at the forum.

This week's instance of FFF is brought to you by cube, your friendly neighbor
clueless network programmer. This post will be more technical than usual, so let us know if you found it interesting.


The week has been spent mostly on pushing forward in multiple of ongoing areas in the development for 0.13. Robert is working on Combinators, Posila and Vaclav on Fire and Flamethrower Turret, Martin is immersed in Factorio modportal and its integration and Tomas and Ondra have been tweaking some Multiplayer Matching server stuff. While Kovarex has spent most time on the 0.12 bugfixing. Albert is back home for a week so the full speed gfx work will happen from next week on.


Factorio has long had problems with map upload speed.
As usual the hard problems they always appear "out there" and never when you're looking for them,
so while we were getting a steady stream of bug reports about this problem, we
could never replicate it.

You might ask why do we even need to upload the whole map when the player might
end up seeing just a tiny part of the world anyway.
AAAAnd the answer is: That's just how it works :-).
Essentially it's because Factorio runs a lockstep simulation and every player needs
to have complete information about the whole map.
This has been discussed in slightly more detail in
an older blogpost.

Now that we know that we really must copy all the data, the question is how to do
it well.


As you probably already knew, the Internet works by passing small chunks of data
called packets
(or frames or segments or datagrams, depending on where exactly you look).
These are nothing special, just some bytes wrapped in an envelope made out of other bytes.
The network that these packets travel is a dangerous place, packet can get lost,
delayed, or even duplicated along the way.

To keep people sane and networks fast, someone came up with a way to hide all these
complications (because that's what programmers do) and invented the TCP protocol.
Instead of transferring unreliable packets, it provides a stream of bytes
and the machinery inside wraps pieces of the stream to packets and makes sure that
all bytes arrive as sent.

Which sounds suspiciously similar to moving files over the network, right?
Just pop the whole thing into the stream and let the elfs do their magic.


Unfortunately in Factorio nothing is ever easy :-).
Using TCP would have three important consequences: It would force players to open
TCP ports as well as UDP ports in firewalls and NAT servers, it would remove the
possibility of using NAT punching now or later with MP matching server and it would
cause problems with downloading from multiple players at the same time.

So we decided to have file transfers done over UDP, even though it means reimplementing
a significant part of TCP functionality.

To make downloading from several other players possible in peer to peer mode,
the file transfers work by requesting individual blocks from peers and waiting for
the data, instead of sending data and waiting for acknowledgements.
Also we don't really care about ordering of the data blocks as long as all of them
are eventually transmitted.


When we started implementing the file transfer all I had was a rough idea of
"we're sending some packets and waiting for confirmations, somehow we know how many unconfirmed
packets there should be at one time and that value is called cWnd".
So Wikipedia came to the rescue and told me that a prety decent algorithm to determine
cWnd is called CUBIC.
So I read an article or two, coded it up and ... it worked! Let's ship it!

A few releases, a few fixes, and a few desperate kludges later it worked very reliably
on my machine and the code got kind of forgotten.

About three weeks ago we decided to revisit the map transfers and I've spent two
fun weeks in TCP land.


The significant part of TCP functionality mentioned earlier is called congestion avoidance.

When you are sending some packets into the network you usually don't know in advance
how fast the network is or how long it will take to reach the other end and
if too many packets are stuffed into the connection not only some of them will be
dropped, but also packets belonging to other connections that go through the same
part of network will get dropped.
This situation is called network congestion and generally it's a thing we want to avoid,
because it slows the available transfer speed for everyone.

So the main idea behind congestion avoidance TCP is to send some amount data and
wait for acknowledgements before sending more.
If the acknowledgement arrives, we know that sending this amount of data is OK and
we try to send a little more next time, if on the other hand the packets start
disappearing we know we are sending too much and we should slow down.

Because of only a little complicated reasons, the increase of the window size
is a roughly linear function of time and the decrease is immediate halving, this
algorithm is called AIMD (additive increase, multiplicative decrease).
This way the amount of data in the network is oscillating between no congestion
at all and very minor congestion, slowly going up and then suddenly jumping down
to half speed.


Our first guess for the reasons of the slow transfer speeds was packet loss.
Both AIMD and CUBIC decrease the transfer speed drastically after a packet is lost
so when your microwave decides that the WiFi connection is its mortal enemy,
the congestion avoidance algorithm will slow down.

The measurements confirmed this, but compared to TCP, our file transfers were always
a little bit faster when packet loss was artificially added.

After an embarrassingly long period of experimentation I was left with two implementations.
AIMD, based mainly on RFC 5681 and
my own attempt slightly inspired by Sync-TCP and LEDBAT,
but other than that completely original.
And neither of them worked too well :-).

With my custom protocol that was not unexpected, it was mostly meant for experimenting
and to get a feeling for all the different behaviors the network can cause.

On the other hand why the AIMD implementation was not working was a mystery.
Finally after finding RFC 2525
(thank you, internet people from 1999), I figured out that all problems in my life
were caused by a missing fast retransmit / fast recovery mechanism.

Fast retransmit and fast recovery work by marking a packet as lost not when its
timeout expires, but when three packets sent later are confirmed instead of the
packet in question.
In this case the packet is retransmitted and fast recovery mode is entered instead of waiting
a whole second, overflowing the network with extra packets the whole time.

After these changes we've upgraded the algorithm to be very similar to
TCP Westwood which finally
gave us the improved behavior on lossy networks.

So TADAAA, here we are now.
The map transfer protocol is less tested than the previous one, but we feel more
certain about it, so let's hope that it's going to be the last change necessary
to this part of networking code.

Map download was slow, cube spent some time doing computer stuff, now it should be faster.

Let us know what you think at the forum.

• Bugfixes
  
  • Fixed crash when script errors occurs while loading game while in game. more
    
  • Fixed that attempting to connect to Factorio 0.12.30 with an old client wouldn't produce the correct error message. more
    
  • Fixed that modded roboports with no charging slots were still considering as charging candidates. more
    
  • Fixed that item icon variant for dark background was not used when showing cargo of logistic robots. more
    
  • Fixed crash with "Unblock sendto failed: The operation completed successfully." more
    
  • Human readable error notice when multiplayer connection wasn't successful. more
    
  • Fixed that headless server wouldn't save the map after Control-C. more
    
  • Fixed that attacking biters with a mining tool wouldn't aggro them.
    
  • Improved map download speed when connecting to multiplayer game. more
    
  • Fixed that assigning units to a unit group of a different force would corrupt the save. more
    
  • Fixed crash when using can_insert on the quickbar from the Lua APi.
    
  • Fixed tight spot scenario crash. more
    
  • Fixed sort order for recipes in the recipe display.
    
  • Fixed incorrect error message, when setting the number of filters on an inserter. more
    
  • Fixed flame from rafineries that are marked for deconstruction was frozen in mid air. more
    
  • Fixed that odd sized blueprints were shifted when rotated. more
    
  • Fixed crash when clearing modded blueprint that doesn't need item to be cleared. more
    
  • Fixed crash in arithmetic combinator when calculating -2147483648/-1. more
    
  • Fixed crash when assigning a character as a passenger of a vehicle when that character already is a passenger of another vehicle. more
    
  • Fixed dealing damage could sometimes corrupt game state and saves. more
    
  • Fixed modded save could not be loaded when a modded inserter gained abilities of smart inserter. more
    
  • Fixed unknown-key message on the third level of first-steps. more
    
  • Fixed cursor-split transfer (right click) not working when putting items into assembling machine stacks over normal stack size. more
    
  • Fixed remote.call() from within the same mod. more
    
  • Fixed inconsistent inserter behaviour when inserting into a splitter from the side. more
    
  • Fixed train trying to turn around on a single rail. more
    
  • Fixed internal electric network problems when mod specified irregular number as supply reach of power pole. more
    
  • Fixed crash when lua script appears in specific state in multiplayer. more
    
  • Fixed another crash than can happen when train destroys itself. more
    
  • Fixed rare multiplayer crash "server is missing in the crcs". more
    
  • Fixed several GUI problems on high DPI displays more
    
  
  
• Optimisations
  
  • Improved performance when using landmines. more
    
  
  
• Scripting
  
  • LuaUnitGroup::add_member now requires that the new member's force be the same as the UnitGroup's.
    
  • Added LuaUnitGroup::force read-only attribute.
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.

• Bugfixes
  
  • Fixed crash when script errors occurs while loading game while in game. more
    
  • Fixed that attempting to connect to Factorio 0.12.30 with an old client wouldn't produce the correct error message. more
    
  • Fixed that modded roboports with no charging slots were still considering as charging candidates. more
    
  • Fixed that item icon variant for dark background was not used when showing cargo of logistic robots. more
    
  • Fixed crash with "Unblock sendto failed: The operation completed successfully." more
    
  • Human readable error notice when multiplayer connection wasn't successful. more
    
  • Fixed that headless server wouldn't save the map after Control-C. more
    
  • Fixed that attacking biters with a mining tool wouldn't aggro them.
    
  • Improved map download speed when connecting to multiplayer game. more
    
  • Fixed that assigning units to a unit group of a different force would corrupt the save. more
    
  • Fixed crash when using can_insert on the quickbar from the Lua APi.
    
  • Fixed tight spot scenario crash. more
    
  • Fixed sort order for recipes in the recipe display.
    
  • Fixed incorrect error message, when setting the number of filters on an inserter. more
    
  • Fixed flame from rafineries that are marked for deconstruction was frozen in mid air. more
    
  • Fixed that odd sized blueprints were shifted when rotated. more
    
  • Fixed crash when clearing modded blueprint that doesn't need item to be cleared. more
    
  • Fixed crash in arithmetic combinator when calculating -2147483648/-1. more
    
  • Fixed crash when assigning a character as a passenger of a vehicle when that character already is a passenger of another vehicle. more
    
  • Fixed dealing damage could sometimes corrupt game state and saves. more
    
  • Fixed modded save could not be loaded when a modded inserter gained abilities of smart inserter. more
    
  • Fixed unknown-key message on the third level of first-steps. more
    
  • Fixed cursor-split transfer (right click) not working when putting items into assembling machine stacks over normal stack size. more
    
  • Fixed remote.call() from within the same mod. more
    
  • Fixed inconsistent inserter behaviour when inserting into a splitter from the side. more
    
  • Fixed train trying to turn around on a single rail. more
    
  • Fixed internal electric network problems when mod specified irregular number as supply reach of power pole. more
    
  • Fixed crash when lua script appears in specific state in multiplayer. more
    
  • Fixed another crash than can happen when train destroys itself. more
    
  • Fixed rare multiplayer crash "server is missing in the crcs". more
    
  • Fixed several GUI problems on high DPI displays more
    
  
  
• Optimisations
  
  • Improved performance when using landmines. more
    
  
  
• Scripting
  
  • LuaUnitGroup::add_member now requires that the new member's force be the same as the UnitGroup's.
    
  • Added LuaUnitGroup::force read-only attribute.
    

You can get experimental releases by selecting the 'experimental' beta branch under Factorio's properties in Steam.