I recently started making an effort to make sure all of my digital purchases are backed up. Apps, eBooks, Music and Comics are fairly easy to deal with plus I don’t have much as compared to my Steam library.
My Steam library, while compared to other people, might not be huge but it’s the single largest collection of digital content I own and have zero backups of it. In my case my library totals around $6000 as of this writing and based on the ever fluctuation price of content on Steam.
Steam itself offers a method for backing up your games. You launch Stream, install a game and then click ‘Steam’ in the top left and ‘Backup and Restore Programs’. You’re then presented with a wizard that will guide you through the process of backing up your selected games. The flaws with this system are that it’s manual, would need to be re-run every time a game was updated if you care about having the latest patched version and requires you to install every single game in your library.
The solution I’ve come up with still requires you to have all of your games installed locally but it does address the rest of the problems.
Here is how I set everything up.
First I needed storage equal to double my Steam Library, half for the library itself and half for the backups. In reality I’ll need less but this is a good starting point. I used Steam Gauge to figure out my library is 2.29TB. I have a 4TB USB hard drive lying around, that should just barely do the trick.
Second I needed something to run all of this off of. Fortunately for me I have a homelab and a dedicated system for backing up all of my personal computers and my homelab itself. The system is a small form factor PC with a Core i3-8100, 16GB of RAM and is running Windows Server 2016 Standard. You can accomplish the same thing with any old laptop, desktop or even just use your gaming rig. It just needs to be powerful enough to run Windows 7 (or newer) and Steam. You can just use your gaming rig if you leave it on 24/7 and have all of your Steam games installed on it. I only install the games I’m playing or have yet to play on my gaming rig which is why I’m using a separate system.
Now that I have everything I need I got started:
- I attached the USB drive to the system I am going to use to store all of my Steam games and perform the backups
- I formatted the USB drive with NTFS
- (Optional) If you want to make your storage got further and don’t mind a performance impact you can enable compression. To enable compression right click the drive, choose properties, and check mark ‘Compress this drive to save disk space’. I recommend doing this before you start downloading your Steam library. Since I’m using Windows Server 2016 and not a regular version of Windows I chose to do something different. More on that here.
- I then downloaded and installed the Steam client on the USB drive and logged into it
- I then queued up every single game in my library to download/install and waited…… like 3 days
While the download was happening I did some quick testing and confirmed that I can have Steam running on a second PC in my home downloading games and still use Steam on my gaming rig to play games. This is ideal because now I don’t have to worry about scheduling game updates to occur on the backup server when I most likely won’t be play games on my gaming rig.
Three days later my entire library was downloaded and I can get into the automation part. We need to accomplish a few things:
- Automatically launch Steam on a schedule so games will patch and then
gracefullyexit Steam upon completion
- Run The Steam Backup Tool against the Steam library so new games are backed up and games that were patched are re-backed up
- Make sure neither 1 or 2 happen at the same time because the Steam Backup Tool won’t run if Steam is running
To accomplish #1 I chose to use Windows Tasks.
Launching Steam is straight forward, just create a scheduled task in Windows that launches “<STEAM INSTALL LOCATION>\Steam.exe” and be sure to configure the task to ‘Run whether user is logged in or not’. I set mine to run at 9:00am and configure Steam to schedule game updates between 9:00am and 1:00pm and then configured the task to automatically terminate the application (Steam) if it runs for more than 4 hours. This will take care of automatically starting Steam, letting updates download and closing Steam.
After the task was created I disabled it. I want to run a full backup of everything first before enabling the automatic start/exit of Steam. The first full backup is going to take a long time and if Steam launches while it’s running bad things will likely happen.
To accomplish #2 I again chose to use Windows Tasks.
I decided I am going to backup my Steam games to the same drive they are installed on. This is because my backup software (Veeam) can then copy those files to my actual backup hard drive giving me the advantage of compression, deduplication, proper incrementals and the ability to replicate the data later to the cloud all using the Veeam software. This ends up wasting a ton of storage but that’s OK in my case. This 4TB hard drive has been doing nothing for a year. For the average person you’re probably going to want to choose a different location to save your backups.
The Steam Backup Tool has a command line (or CLI) version of it’s executable. The flags are:
usage: steamBackupCLI [options]
-h, -?, --help show this message and exit.
-O, --out-dir=VALUE (required) Set backup directory
-S, --steam-dir=VALUE Do not automatically detect Steam directory, use
this directory instead
-2, --lzma2 Use LZMA2 compression.
-C, --compression=VALUE Set compression level. Possible values 0 - 5:
0 : Copy
1 : Fastest
2 : Fast
3 : Normal
4 : Maximum
5 : Ultra
-B, --backup Update backup
Update games that have been changed since the
last backup, EXCLUDING games that have not been
backed up yet.
-L, --library Update library
Update games that have been changed since the
last backup, INCLUDING games that have not been
backed up yet.
-D, --delete Delete all backup files before starting
ignored when either update library or update
backup parameter is used
-T, --threads=VALUE Thread count
LZMA: number of concurrent instances,
LZMA2: number of threads used
I created my scheduled task to run the backups to occur at 2:00pm (1 hour after Steam should automatically exit). The scheduled task is just like the Steam one created earlier:
C:\Program Files (x86)\Steam Backup Tool\v1_8_6\steamBackupCLI.exe -O E:\Steam-Backups -S E:\Steam -2 -C 5 -L -T 2
I chose to use LZMA2 compression (because why not), Ultra compression because I don’t care how long this takes and want the least amount of disk space to be used, I want anything newly installed to get backed up and anything that’s been patched to be re-backed up and I limited the backup to 2 threads. The server I am running this on only has 4 threads and I want to leave 2 for the OS and Veeam to be able to continue doing their job while these backups run.
You can barely see in the second image there but all of the parameters for the backup after “steamBackupCLI.exe” should go into the “Add Arugments” text box. Microsoft really needs to update this UI to make things more visible.
Before enabling this scheduled task I ran it manually since the first backup is going to take the longest amount of time.
The way I have things scheduled right now backups need to complete within 17 hours otherwise Steam might launch itself during the backups. I might end up changing my schedule so Steam launches on Mondays for 24 hours and then I run backups Tuesday and let it run to completion giving the backups 6 whole days to run. I don’t think it will come to this but I will update this posting if it turns out things take longer than expected.
That should be that. All you need to do moving forward is remember to manually run Steam on the system taking your backups and install new games you buy so they get backed up. I was not able to find a way to automate this. Install the games while Steam is running between 9:00am and 1:00pm if you used my schedule or make sure backups have completed running before manually launching Steam.
Using deduplication instead of compression
Since I’m running this on Windows Server 2016 I have the option to use deduplication instead of compression. De-duplication occurs on a schedule you set and I believe is less CPU intensive than compression. I do not know if it’s more efficient though. My results were fairly impressive with roughly 27% of my Steam library being redundant data that could be deduplicated.