I Have a Need, a Need for Download Speed
Should you try to speed up consecutive Steam downloads with a cache server?
Thanks to Wendell from Level1Techs for his all of his help on this project and pointing us in the right direction!
A few years ago, we were fortunate enough to get a fiber internet connection installed at the PC Perspective office. Capable of 1Gbps download speeds and about 250Mbps upload, we were excited at the possibilities that laid ahead.
However, when you have access to a very fast internet connection, you begin to notice that the bottleneck has shifted from your connection to the servers on the other side of the content delivery networks (CDNs) that power the internet. While these CDNs have very fast links to the internet, they generally limit bandwidth so that there is more speed to go around to multiple people at the same time.
DL from Steam at 101 MB/s! So fast that the SSD is having trouble keeping up. Able to grab 30+ GB in under 5 mins. pic.twitter.com/ptHUyQVHJr— Ryan Shrout (@ryanshrout) September 3, 2014
A look back at what once was
One of the services that we found would max out our connection was Steam. Since we download a lot of PC games at the office, it was a nice benefit to have an internet connection as fast as our NICs could handle, and that the Steam CDNs would serve us at our maximum potential. In fact, the bottleneck shifted over to storage performance, as the random writing nature of Steam thrashed our SSDs at the time.
By no stretch of the imagination is 60MB/s slow.. but what happened to our 100MB/s!
Unfortunately, this has ceased to remain the case. At some point, Steam downloads started getting slower on our same internet connection. Not only did storage utilization during a Steam download start to increase, but also CPU usage, pointing to a potential change in how Steam distributed their data. While downloads on our high-end systems fell to around 50-60MB/s, systems with less CPU horsepower started to see speeds fall to 20-30MB/s. All hope was lost for fast game downloads.. or was it?
Recently, Wendell from Level1Techs mentioned on Twitter that they were running a local Steam caching server on their network with great success. After some guidance from Wendell, we decided to tackle this project and see if it would help our specific scenario.
The theory behind caching Steam downloads is simple. While you might not get the appearance of it from the Steam client, all that's happening is a bunch of HTTP requests to Valve's CDN. Simply speaking, this means, that once you know the addresses of the servers Steam uses for downloading, you can monitor the traffic coming from them, save the data to a cache file, and then when it is needed again, simply deliver the local copy instead of going out to the internet to fetch it.
In order to be able to hijack this traffic, you need to intercept the DNS requests for Valve's CDN and point them to your local HTTP server configured for delivering the data.
To accomplish this, you need to run two Docker containers. The first is the steamcache "generic" container, which provides a preconfigured nginx-powered HTTP proxy to intercept and store the relevant data.
Second, you need to be running the steamcache-dns container to redirect the DNS requests to your local server. This requires your computer sending all of it's DNS requests through to Docker container. This can be configured either by changing the local DNS server configuration on your computers or by changing the DNS server configuration in your router.
Running a local DNS server will provide additional speed benefits to activities such as web browsing since your computer no longer needs to make a request to an external DNS server for every possible HTTP request. This means a lower access time for all HTTP requests.
We aren't going to get into the minutia of configuring Docker for people who have never used it, as there are much better tutorials for that on the Internet by people who understand the process way better than we do. The notes on the steamcache website are a good place to start if you are thinking of doing this yourself.
It's worth noting that the steamcache generic container provides support for a multitude of other download services, like Uplay, Origin, and even Windows Updates. For now, we are sticking with strictly caching Steam downloads. We don't download a lot of games from Uplay or Origin, and when we do, they still manage to fully utilize our 1Gbps internet connection.
As time goes on, and the Steam caching proves to be stable, we might look into adding more cached services.
In order to run the docker containers, we turned to our new Ryzen 7 1700-powered QNAP TS-877 NAS (full review on this to come soon!). While it might seem like a bit of an oddity to run this sort of thing on a NAS, the desktop CPU, 10 Gigabit NIC, and available 2.5-in drive bays for SSDs made it a great pick.
For the cache storage, we are using two 1TB Samsung 840 EVO SATA SSDs in a RAID 0 configuration. While we wouldn't normally recommend a RAID 0 configuration in production, all this will be doing is holding a download cache of files we can easily get back if needed, so we aren't necessarily worried about the potentials of data loss.
Our Linux environment is provided by an Ubuntu 16.04 VM running in the KVM-based QNAP Virtualization Station. While the QNAP has native support for Docker containers running outside of a VM, we wanted to be able to move the VM image to other hardware if needed, and the QNAP Container Station software didn't seem to be very well documented at this point.
With our Steam caching server in place, we first started by downloading several games to a local machine. Since this is the initial download, and there is no data already cached, it will not be any faster than downloading games without the cache enabled. However, in our experiences, this seeding of data to the cache also isn't any slower than the normal process of downloading the game.
To evaluate the performance of the cache, we then downloaded the same games on another PC. In order to eliminate any potential performance impacts on our client, we used an overkill system with an 8-core/16-thread Core i7-5960X CPU, 10Gbit NIC, and an Intel Optane 900P SSD.
With a hard limit of about 250 MB/s (!!!), now, the bottleneck seems to be the on the server side. When downloading at around 250MB/s, the Ryzen 7 1700 CPU in our QNAP NAS seems to be maxing out a single thread of the processor, within our Ubuntu VM.
While we could move this setup to yet another dedicated server with faster single-threaded performance, for now, we are happy with our results. Considering most of our machines aren't equipped with 10Gbit NICs in the office yet, being able to max out our Gigabit connection again is good enough for now.
It does seem like a bit of overkill for an office of less than 5 people to have a Steam caching server, but given what we do it make sense for us. For organizers who are running a LAN party, this would be a great solution to prevent game downloads/updates from saturating your link to the Internet.