Last week Outriders launched and it is a fun time, provided you can get online.
Perhaps that’s unfair because since Sunday 4th April, Outriders servers have been relatively stable, but many players, ourselves included, have been concerned that server issues would appear again as more folks tried to log in and lay waste to Enoch.
Thankfully, however, People Can Fly is working on the problem diligently and while it’s not out of the woods yet it has explained what the problem is and how it managed to fix the issue.
“We’re committed to full transparency with you. Today, just as we have been over the past year. So we won’t give you the expected “server demand was too much for us”. We were in fact debugging a complex issue with why some metric calls were bringing down our externally hosted database. We did not face this issue during the demo launch earlier this year,” explained the developer.
So what happened?
Calls to the game’s server weren’t being managed by RAM but were instead being managed by something People Can Fly calls swap-disk. We suspect this is in reference to virtual memory which makes use of physical storage space, we suspect this because People Can Fly says that this data management method was “too slow for the flow of this amount of data”.
Once there was a large enough queue of these server calls, the service failed.
“We spent over two days and nights applying numerous changes and improvement attempts: we both doubled the database servers and vertically scaled them by approximately 50 percent (“scale-up and scale out”). We re-balanced user profiles and inventories to new servers. Subsequent to the scale-up and scale-out, we also increased disk IOPS on all servers by approximately 40%. We also increased the headroom on the database, multiplied the number of shards (not the Anomalous kind) and continued to do all we were able to in order to force data into RAM,” explained the developer.
While these steps helped resolve some issues, the developer is still in search of a fix that sticks.
“At this moment in time we are still waiting for a final Root Cause Analysis (RCA) from our partners, but ultimately what really helped resolve the overloading issue was configuring our database cache cleaning, which was being run every 60 seconds. At this frequency the database cache cleaning operation demanded too many resources which in turn led to the above mentioned RAM issues and a snowball effect that resulted in the connectivity issues seen,” People Can Fly added.
The net result of these bodges are servers that are more resilient and that can sustain more concurrent users.
While it seems the developer still has a ways to go until servers are operating at full tilt, we’re impressed at People Can Fly’s transparency here.
Looter shooters have infamously suffered from day one issues and often the excuse handed to players is that server load was higher than expected, which is what we expected to hear here.
Rather, People Can Fly has been very transparent about what went wrong and we can’t help but appreciate that. Community engagement for a game such as this is incredibly important and the developers here are doing a stellar job so far.
This news came down as part of a developer news update you can read here.
One of the highlights there is a balance update being made to bullet abilities. This is bad news for Technomancer, Pyromancer and Trickster players though perhaps it will stop poor Devastators from being kicked from end-game activities because they don’t have bullet abilities.
It has been a crazy launch week for Outriders and we hope that the state of the game and its servers only improves from here on out because despite all the connectivity issues, we’re enjoying the game.