How to (almost) automatically backup your Steam library

Update 2018-11-12

It appears a recent Steam update broke the way I was originally launching Steam via the scheduled task. I’ve updated the post accordingly with a different method of accomplishing the same task.

 

Original Post

I recently started making an effort to make sure all of my digital purchases are backed up. Apps, eBooks, Music and Comics are fairly easy to deal with plus I don’t have much as compared to my Steam library.

My Steam library, while compared to other people, might not be huge but it’s the single largest collection of digital content I own and have zero backups of it. In my case my library totals around $6000 as of this writing and based on the ever fluctuation price of content on Steam.

Steam itself offers a method for backing up your games. You launch Stream, install a game and then click ‘Steam’ in the top left and ‘Backup and Restore Programs’. You’re then presented with a wizard that will guide you through the process of backing up your selected games. The flaws with this system are that it’s manual, would need to be re-run every time a game was updated if you care about having the latest patched version and requires you to install every single game in your library.

The solution I’ve come up with still requires you to have all of your games installed locally but it does address the rest of the problems.

I found this utility called “The Steam Backup Tool” (latest version mirrored locally just in case) which takes care of the majority of the automation part of this process.

Here is how I set everything up.

First I needed storage equal to double my Steam Library, half for the library itself and half for the backups. In reality I’ll need less but this is a good starting point. I used Steam Gauge to figure out my library is 2.29TB. I have a 4TB USB hard drive lying around, that should just barely do the trick.

Second I needed something to run all of this off of. Fortunately for me I have a homelab and a dedicated system for backing up all of my personal computers and my homelab itself. The system is a small form factor PC with a Core i3-8100, 16GB of RAM and is running Windows Server 2016 Standard. You can accomplish the same thing with any old laptop, desktop or even just use your gaming rig. It just needs to be powerful enough to run Windows 7 (or newer) and Steam. You can just use your gaming rig if you leave it on 24/7 and have all of your Steam games installed on it. I only install the games I’m playing or have yet to play on my gaming rig which is why I’m using a separate system.

Now that I have everything I need I got started:

  1. I attached the USB drive to the system I am going to use to store all of my Steam games and perform the backups
  2. I formatted the USB drive with NTFS
  3. (Optional) If you want to make your storage got further and don’t mind a performance impact you can enable compression. To enable compression right click the drive, choose properties, and check mark ‘Compress this drive to save disk space’. I recommend doing this before you start downloading your Steam library. Since I’m using Windows Server 2016 and not a regular version of Windows I chose to do something different. More on that here.
  4. I then downloaded and installed the Steam client on the USB drive and logged into it
  5. I then queued up every single game in my library to download/install and waited…… like three days

While the download was happening I did some quick testing and confirmed that I can have Steam running on a second PC in my home downloading games and still use Steam on my gaming rig to play games. This is ideal because now I don’t have to worry about scheduling game updates to occur on the backup server when I most likely won’t be play games on my gaming rig.

Three days later my entire library was downloaded and I can get into the automation part. We need to accomplish a few things:

  1. Automatically launch Steam on a schedule so games will patch and then gracefully exit Steam upon completion
  2. Run The Steam Backup Tool against the Steam library so new games are backed up and games that were patched are re-backed up

To accomplish #1 I chose to use Windows Tasks.

Launching Steam is straight forward, just create a scheduled task in Windows that launches “<STEAM PATH>\Steam.exe” and be sure to configure the task to ‘Run only when the user is logged on’. I set mine to run at 9:00am on Monday. This will take care of automatically starting Steam, letting updates download over the next few days.

After the task was created I disabled it (right click, disable). I want to run a full backup of everything first before enabling the automatic start/exit of Steam. The first full backup is going to take a long time and if Steam launches while it’s running bad things will likely happen.

To accomplish #2 I again chose to use Windows Tasks and some PowerShell.

I decided I am going to backup my Steam games to the same drive they are installed on. This is because my backup software (Veeam) can then copy those files to my actual backup hard drive giving me the advantage of compression, deduplication, proper incrementals and the ability to replicate the data later to the cloud all using the Veeam software. This ends up wasting a ton of storage but that’s fine for my use case. This 4TB hard drive I’m using has been doing nothing for a year. For the average person you’re probably going to want to choose a different location to save your backups.

The Steam Backup Tool has a command line (or CLI) version of it’s executable. The flags are:

The backup command I settled on is:

I chose to use LZMA2 compression [-2] (because why not), Ultra compression [-C 5] because I don’t care how long this takes and want the least amount of disk space to be used, I want anything newly installed to get backed up and anything that’s been patched to be re-backed up [-L] and I limited the backup to 2 threads [-T 2] because server I am running this on only has 4 threads and I want to leave 2 for the OS and Veeam to be able to continue doing their job while these backups run.

The next part was a bit tricky to figure out due to the design of SteamBackupCLI.exe. It turns out the application is expecting a command prompt or PowerShell console window to be created when it runs so it can resize the window and dump all of it’s output to the screen. Trying to run SteamBackupCLI in the background is a no-go which makes it slightly annoying to get running in an automated fashion but not impossible. I’ve submitted a bug to the developer in hopes they will update SteamBackupCLI and offer a command line flag to dump all output to a log file instead of expecting a console window to be opened.

What this means for us is we have to remain logged in to the system that is going to run SteamBackupCLI. In my case I use Remote Desktop (RD) to get into my server so all I need to do is login to the server after a reboot and just click the ‘x’ to close RD with out actually logging out. If you’re using your regular desktop you need to do is make sure you log into as the user running the backup task after a reboot. You can likely “switch desktops” or lock the screen and everything will continue to function but I have not tested this.

Here is the PowerShell script I wrote to kill Steam, wait 30 seconds and then start SteamBackupCLI minimized:

This script assumes you installed Steam Backup Tool in “C:\Program Files (x86)\Steam Backup Tool\v1_8_6\“. If you didn’t, update the 3rd line of this script accordingly.

I saved this script as “startSteamBackup.ps1” in “C:\Scripts” on my system. You can save it anywhere you want, just remember where.

I configured the backup to run on Wednesday at 9:00am which will give the backup 4 days to run before Steam will try to launch itself again on Monday. It is very important with this task that you select ‘Run only when the user is logged on’. If you choose the other option this task will crash on launch as mentioned earlier in this post.

The parameters in the last screenshot are:

Program/script: powershell.exe
Add arguments (optional): -ExecutionPolicy Bypass C:\Scripts\startSteamBackup.ps1
Start in (optional): C:\Scripts

After the task was created I disabled it (right click, disable).

Before enabling the scheduled tasks I manually ran the backup command “steamBackupCLI.exe -O E:\Steam-Backups -S E:\Steam -2 -C 5 -L -T 2” once since this was going to take longer than the 4 days I’m allowing with the schedule. Once the initial backup was done I then enabled both tasks (right click, enable).

 

 

(Bonus) Using deduplication instead of compression

Since I’m running this on Windows Server 2016 I have the option to use deduplication instead of compression. De-duplication occurs on a schedule you set and I believe is less CPU intensive than compression. I do not know if it’s more efficient though. My results were fairly impressive with roughly 27% of my Steam library being redundant data that could be deduplicated.

 

 

Update 2018-11-10

After one full run of backups

Script to sync Domain Controller SSL Certificates to a specific host

We have an application that uses LDAP over SSL to authenticate users via Active Directory. The server running the application is a member of the domain and has the domains Root CA installed in it’s local certificate store.

Technically the Root CA should be good enough for the server and any applications on it to trust the SSL certificates on our domain controllers because they are signed by that Root CA. Not the case for this application.

We have four Domain Controllers each with a different SSL certificate that expires yearly and each with a different expiry date. Exporting and importing these certificates manually is going to be a huge annoyance.

I wrote a PowerShell script to handle doing it automatically for us. This script is being run against 2012 R2 Domain controllers which is why I use the PowerShell Module exporting the certificates and the target is  2008 R2 which is why the import is handled via ‘certutil’. You could easily swap these out in the script to suite your needs.

 

Migrating DHCP from 2003 to 2012 R2

This post will likely fall on deaf ears since no one out there is still running Windows Server 2003 right?

Well we are for our oldest DHCP server. Better yet it’s 2003 Standard (non R2) which means I can’t installed Active Directory Management Gateway Service which would allow me to remotely access DHCP on the 2003 server via PowerShell 5.1 running on a different box. Newer versions of PowerShell include support for working with DHCP. PowerShell 2.0 (latest available for Server 2003) does not support these commands.

Googling around found me the standard recommended way of migrating DHCP subnets.

On the source machine run:

This will export all of the current leases and reservations in the 192.168.0.0 scope into a text file, you can then transfer the text file over to your new DHCP server and run the following to import it:

The downside to this method is that it causes a temporary outage of your DHCP server during the import/export. I just migrated 80 odd subnets during the day and the outages were so short no one noticed.

Alright so that was the easy part.

In addition to 80 standard DHCP scopes we have 4 superscopes that also need to be migrated.

Attempting to migrate the superscopes using the above method failed. When I attempted to import the scope into the destination DHCP server I got the error “TLS supported but not configured”.

The first post I found for this error in Google links to a Technet discussion where someone states “No, you cannot direct migrate windows server 2003 DHCP/DNS to windows server 2012 DHCP/DNS.”. Clearly not an accurate statement since I’d just migrated 80 standard scopes.

So here I am. I cannot move 4 super scopes using the netsh method, I cannot use PowerShell because the 2003 server is to old and I do not want to upgrade it to 2003 R2 for multiple reasons (did I mention it’s a Domain Controller to?).

What I ended up doing was building a new Windows 2012 R2 Standard box, joined it to our domain so I could remotely access it and have the benefit of domain logins for accessing resources across our network and then installed DHCP on it. Immediately after the DHCP Server installation completed I went into the Windows Firewall and blocked DHCP just in case.

Then on the old DHCP server I ran this command:

I then transferred the all.txt file over to the DHCP server I just built and ran this:

and ended up with a complete copy of my old DHCP server on my temporary DHCP server including my superscopes, no errors.

Now I can use PowerShell to finish this up. On the temporary DHCP server I ran this:

transferred SUPERSCOPE1.xml to the new DHCP server and ran:

And bingo. Superscope successfully migrated from 2003 to 2012R2 with a small middle step.

If you screw up or need to do this in batches over time you can quickly and easily wipe everything out on the temporary DHCP server by doing the following:

  1. Stop the DHCP service
  2. Delete the contents of C:\Windows\system32\dhcp
  3. Start DHCP servoce

You’ll end up with a blank DHCP server that you can re-import a fresh copy of your old DHCP server into.

After vCenter 6.5 U1g newly cloned VMs report “HA not enabled”

Our environment is:

  • vCenter 6.5 U1g (8024368) running on a Windows Server 2008 R2 Standard VM (we’re moving to an appliance once we go 6.7)
  • vSphere 5.5 7504623, 8 node cluster

Since upgrading to the latest vCenter every new VM we clone into our production cluster reports “The virtual machine failed to become vSphere HA Protected and HA may not attempt to restart it after a failure”

After contacting VMware Support they informed me that this is a known issue with the versions of vCenter and vSphere we’re using and to upgrade to vSphere 6.0 or 6.5.

Since vSphere 5.5 is EOL in September no patches are coming to resolve this.

It also turns out the issue is a GUI issue only. The VMs displaying this error are still protected via HA and will recover fine if a host goes down.

A workaround for the issue is to disable/enable HA. Doing so will clear the error.

How to replace the boot volume on a RaspberryPi

Just ran into what I think is called the “rainbow screen of death” where a fully functioning RaspberryPi running Raspbian stopped working on me after a reboot.

I know my MicroSD card is good quality and I wasn’t mucking with the boot file system before the reboot so it baffled me why things would just stop working.

I ran then MicroSD card through SpinRite and that didn’t solve the problem. If I connected the microSD card to another PC all the files were there.

Based on my research I found this happens from time to time where something goes wrong with the /boot volume. I could have just re-loaded and re-setup my RPi (it would have taken less time at this point) but I decided I wanted to try and fix it with out losing all of my data.

This is what worked for me:

  1. Image a fresh spare microSD card with the same distro of Raspbian you’re using on the non-functional microSD
  2. Connect the non-functional microSD to the same PC you have your freshly imaged microSD connected
  3. In Windows there is going to be one partition (disk) under My Computer you can actually view, that’s the boot partition
  4. On the non-functioning microSD copy ‘cmdline.txt’ and ‘config.txt’ to some place safe on your computer
  5. Copy the entire contents of the boot partition from the new microSD to the bad one overwriting all files when prompted
  6. Copy the backups of ‘cmdline.txt’ and ‘config.txt’ on to the bad microSD

If you’ve never run ‘rpi-update’ then you’re done. You should be able to put the microSD into your RPi and it will boot again.

If you have run ‘rpi-update’ you have a bit more work to do.

  1. Go to https://github.com/raspberrypi/firmware/releases and download the release that matches the date of the last time you ran rpi-update
    Note: If you don’t know when this is you might be able to boot the RPi now, login locally and dig through your ‘history’ or look at the date/time stamps on the files in /boot
  2. Extract the downloaded file
  3. Copy the contents of firmware-#.#######\boot into \boot on your bad microSD overwriting all files when prompted

Now you’re done. You should be able to put that bad microSD back into a RPi and have it boot

Once your RPi is backup and running I highly recommend manually running a ‘rpi-update’

References