SolarWinds Web Helpdesk 12.7.2 post-upgrade problems

Recently after upgrading our WHD from 12.7.1 to 12.7.2 we started experiencing two issues that turned out to be related.

Adding comments to tickets could generate an error stating: “Something went wrong. Please contact SolarWinds support for assitance or review the Web Hep Desk logs.”

and when trying to upload attachments to tickets: “Your upload failed: Connection reset by peer: Amount read didn’t match content-length”

After some back and forth with SolarWinds support they said the issue is related to the version of Tomcat they bundled with WHD 12.7.2. The bundled version is 9.0.31 and SolarWinds recommends manually upgrading it to 9.0.34 to fix the issue.

The procedure I followed was:

# Login to your WHD server and become root

# Make a temp directory, go into it and download Tomcat 9.0.34
mkdir temp
cd temp
wget -c https://muug.ca/mirror/apache-dist/tomcat/tomcat-9/v9.0.34/bin/apache-tomcat-9.0.34.tar.gz

# Stop WHD
service stop webhelpdesk

# Backup your existing WHD Tomcat directory just in case
cd /usr/local/webhelpdesk/bin
tar -cf tomcatBackup-20200423.tar tomcat
gzip -9 tomcatBackup-20200423.tar

# Copy Tomcat 9.0.34 overtop of Tomcat 9.0.31 while preserving WHD related content
cd ~/temp
cd apache-tomcat-9.0.34
cp -Raf bin conf lib NOTICE RELEASE-NOTES /usr/local/webhelpdesk/bin/tomcat/

# Reboot your server or manually start WHD back up
shutdown -r now
# OR
service webhelpdesk start

Once we did this the errors above went away.

How to (almost) automatically backup your Steam library

Update 2018-11-12

It appears a recent Steam update broke the way I was originally launching Steam via the scheduled task. I’ve updated the post accordingly with a different method of accomplishing the same task.

 

Original Post

I recently started making an effort to make sure all of my digital purchases are backed up. Apps, eBooks, Music and Comics are fairly easy to deal with plus I don’t have much as compared to my Steam library.

My Steam library, while compared to other people, might not be huge but it’s the single largest collection of digital content I own and have zero backups of it. In my case my library totals around $6000 as of this writing and based on the ever fluctuation price of content on Steam.

Steam itself offers a method for backing up your games. You launch Stream, install a game and then click ‘Steam’ in the top left and ‘Backup and Restore Programs’. You’re then presented with a wizard that will guide you through the process of backing up your selected games. The flaws with this system are that it’s manual, would need to be re-run every time a game was updated if you care about having the latest patched version and requires you to install every single game in your library.

The solution I’ve come up with still requires you to have all of your games installed locally but it does address the rest of the problems.

I found this utility called “The Steam Backup Tool” (latest version mirrored locally just in case) which takes care of the majority of the automation part of this process.

Here is how I set everything up.

First I needed storage equal to double my Steam Library, half for the library itself and half for the backups. In reality I’ll need less but this is a good starting point. I used Steam Gauge to figure out my library is 2.29TB. I have a 4TB USB hard drive lying around, that should just barely do the trick.

Second I needed something to run all of this off of. Fortunately for me I have a homelab and a dedicated system for backing up all of my personal computers and my homelab itself. The system is a small form factor PC with a Core i3-8100, 16GB of RAM and is running Windows Server 2016 Standard. You can accomplish the same thing with any old laptop, desktop or even just use your gaming rig. It just needs to be powerful enough to run Windows 7 (or newer) and Steam. You can just use your gaming rig if you leave it on 24/7 and have all of your Steam games installed on it. I only install the games I’m playing or have yet to play on my gaming rig which is why I’m using a separate system.

Now that I have everything I need I got started:

  1. I attached the USB drive to the system I am going to use to store all of my Steam games and perform the backups
  2. I formatted the USB drive with NTFS
  3. (Optional) If you want to make your storage got further and don’t mind a performance impact you can enable compression. To enable compression right click the drive, choose properties, and check mark ‘Compress this drive to save disk space’. I recommend doing this before you start downloading your Steam library. Since I’m using Windows Server 2016 and not a regular version of Windows I chose to do something different. More on that here.
  4. I then downloaded and installed the Steam client on the USB drive and logged into it
  5. I then queued up every single game in my library to download/install and waited…… like three days

While the download was happening I did some quick testing and confirmed that I can have Steam running on a second PC in my home downloading games and still use Steam on my gaming rig to play games. This is ideal because now I don’t have to worry about scheduling game updates to occur on the backup server when I most likely won’t be play games on my gaming rig.

Three days later my entire library was downloaded and I can get into the automation part. We need to accomplish a few things:

  1. Automatically launch Steam on a schedule so games will patch and then gracefully exit Steam upon completion
  2. Run The Steam Backup Tool against the Steam library so new games are backed up and games that were patched are re-backed up

To accomplish #1 I chose to use Windows Tasks.

Launching Steam is straight forward, just create a scheduled task in Windows that launches “<STEAM PATH>\Steam.exe” and be sure to configure the task to ‘Run only when the user is logged on’. I set mine to run at 9:00am on Monday. This will take care of automatically starting Steam, letting updates download over the next few days.

After the task was created I disabled it (right click, disable). I want to run a full backup of everything first before enabling the automatic start/exit of Steam. The first full backup is going to take a long time and if Steam launches while it’s running bad things will likely happen.

To accomplish #2 I again chose to use Windows Tasks and some PowerShell.

I decided I am going to backup my Steam games to the same drive they are installed on. This is because my backup software (Veeam) can then copy those files to my actual backup hard drive giving me the advantage of compression, deduplication, proper incrementals and the ability to replicate the data later to the cloud all using the Veeam software. This ends up wasting a ton of storage but that’s fine for my use case. This 4TB hard drive I’m using has been doing nothing for a year. For the average person you’re probably going to want to choose a different location to save your backups.

The Steam Backup Tool has a command line (or CLI) version of it’s executable. The flags are:

usage: steamBackupCLI [options]

Parameters:
  -h, -?, --help             show this message and exit.
  -O, --out-dir=VALUE        (required) Set backup directory
  -S, --steam-dir=VALUE      Do not automatically detect Steam directory, use
                               this directory instead
  -2, --lzma2                Use LZMA2 compression.
  -C, --compression=VALUE    Set compression level. Possible values 0 - 5:
                                0 : Copy
                                1 : Fastest
                                2 : Fast
                                3 : Normal
                                4 : Maximum
                                5 : Ultra
  -B, --backup               Update backup
                               Update games that have been changed since the
                               last backup, EXCLUDING games that have not been
                               backed up yet.
  -L, --library              Update library
                               Update games that have been changed since the
                               last backup, INCLUDING games that have not been
                               backed up yet.
  -D, --delete               Delete all backup files before starting
                               ignored when either update library or update
                               backup parameter is used
  -T, --threads=VALUE        Thread count
                               LZMA:  number of concurrent instances,
                               LZMA2: number of threads used

The backup command I settled on is:

C:\Program Files (x86)\Steam Backup Tool\v1_8_6\steamBackupCLI.exe -O E:\Steam-Backups -S E:\Steam -2 -C 5 -L -T 2

I chose to use LZMA2 compression [-2] (because why not), Ultra compression [-C 5] because I don’t care how long this takes and want the least amount of disk space to be used, I want anything newly installed to get backed up and anything that’s been patched to be re-backed up [-L] and I limited the backup to 2 threads [-T 2] because server I am running this on only has 4 threads and I want to leave 2 for the OS and Veeam to be able to continue doing their job while these backups run.

The next part was a bit tricky to figure out due to the design of SteamBackupCLI.exe. It turns out the application is expecting a command prompt or PowerShell console window to be created when it runs so it can resize the window and dump all of it’s output to the screen. Trying to run SteamBackupCLI in the background is a no-go which makes it slightly annoying to get running in an automated fashion but not impossible. I’ve submitted a bug to the developer in hopes they will update SteamBackupCLI and offer a command line flag to dump all output to a log file instead of expecting a console window to be opened.

What this means for us is we have to remain logged in to the system that is going to run SteamBackupCLI. In my case I use Remote Desktop (RD) to get into my server so all I need to do is login to the server after a reboot and just click the ‘x’ to close RD with out actually logging out. If you’re using your regular desktop you need to do is make sure you log into as the user running the backup task after a reboot. You can likely “switch desktops” or lock the screen and everything will continue to function but I have not tested this.

Here is the PowerShell script I wrote to kill Steam, wait 30 seconds and then start SteamBackupCLI minimized:

Get-Process -Name "Steam" | Stop-Process -Force
Start-Sleep -Seconds 30
cd "C:\Program Files (x86)\Steam Backup Tool\v1_8_6\"
start powershell -WindowStyle Minimized ".\steamBackupCLI.exe -O E:\Steam-Backups -S E:\Steam -2 -C 5 -L -T 2"

This script assumes you installed Steam Backup Tool in “C:\Program Files (x86)\Steam Backup Tool\v1_8_6\“. If you didn’t, update the 3rd line of this script accordingly.

I saved this script as “startSteamBackup.ps1” in “C:\Scripts” on my system. You can save it anywhere you want, just remember where.

I configured the backup to run on Wednesday at 9:00am which will give the backup 4 days to run before Steam will try to launch itself again on Monday. It is very important with this task that you select ‘Run only when the user is logged on’. If you choose the other option this task will crash on launch as mentioned earlier in this post.

The parameters in the last screenshot are:

Program/script: powershell.exe
Add arguments (optional): -ExecutionPolicy Bypass C:\Scripts\startSteamBackup.ps1
Start in (optional): C:\Scripts

After the task was created I disabled it (right click, disable).

Before enabling the scheduled tasks I manually ran the backup command “steamBackupCLI.exe -O E:\Steam-Backups -S E:\Steam -2 -C 5 -L -T 2” once since this was going to take longer than the 4 days I’m allowing with the schedule. Once the initial backup was done I then enabled both tasks (right click, enable).

 

 

(Bonus) Using deduplication instead of compression

Since I’m running this on Windows Server 2016 I have the option to use deduplication instead of compression. De-duplication occurs on a schedule you set and I believe is less CPU intensive than compression. I do not know if it’s more efficient though. My results were fairly impressive with roughly 27% of my Steam library being redundant data that could be deduplicated.

 

 

Update 2018-11-10

After one full run of backups

Patterson EagleSoft v18 crashing during auto-backups

We’re running Patterson EagleSoft v18.10.05 on a Windows 2008 R2 Standard server.

We have auto-backup configured to run when ever the database starts up and to keep 14 backups:

We have a scheduled task on the server that runs “D:\EagleSoft\Shared Files\techaid.exe -stop” at 3:00am and then “D:\EagleSoft\Shared Files\techaid.exe -start” at 3:15am to trigger a nightly automatic backup while the database isn’t in use.

We just noticed that backups haven’t been occurring since 2017-12-31. Funny how they stopped 2018-01-01.

Then around 2018-02-12 the EagleSoft database would stop as planned at 3:00am and then crash/fail to start at 3:15am with no backup being generated. We’d then have to manually start EagleSoft’s database the next day so users could access it.

We found files like these in the EagleSoft auto backup directory:

  • DotNetZip-5tusrxsj.tmp
  • DotNetZip-fybwnf0z.tmp
  • DotNetZip-hete0py2.tmp
  • DotNetZip-zjlqhwj0.tmp

After contacting Patterson for support they informed us this is a known issue with EagleSoft v18 and the fix was to upgrade to v19.

I dug into the backup zip files and found they contain only three files:

  • Eaglesoft.Server.Configuration.data
  • PattersonPM.db
  • PattersonPM.log

All of which are located in “<EAGLESOFT INSTALL DIR>\Data”.

Some further digging showed there were additional LOG and DB files in the Data DIR that aren’t being backed up by auto backup.

Upgrading right now isn’t exactly an option for us so I wrote a PowerShell script to replace EagleSofts built-in auto backup. We run full backups of the server with Veeam so this script is just a little extra insurance for the database specifically.

Note: This script requires PowerShell v5.0 or newer. As of this writing v5.1 appears to be the newest and can be installed on Windows Server 2008 R2 or newer.

#
# Written by: Eric Schewe ([email protected])
# Created on: 2018-02-19
#
# This script replaces the auto backup feature built into EagleSoft which appears to have broken as of 2018-01-01 in v18.
# Patterson recommends upgrading to v19 to resolve the issue. In the mean time this script can replace the auto backup.
#
# This script will stop the EagleSoft database, copy the database files, start EagleSoft and the compress the database copies.
#
# This script requires PowerShell v5.0 or newer
#
# Important: Make sure auto backup is disabled or EagleSoft may not start up properly after this script runs
#
# Installation: 
#   1. Configure this as a scheduled task that runs on your server when no one is using EagleSoft
#   2. Set $eagleSoftDirectory and $backupsLocation
#   3. Set $backupsToKeep to how many days of backups you want to keep
#

# &lt;-------- Start User Configuration --------&gt;

# Where is EagleSoft installed?
# Example: "C:\EagleSoft"
$eagleSoftDirectory = "C:\EagleSoft" 

# Where do you want the backups to go?
# Example: "D:\EagleSoft-Backups"
$backupsLocation = "D:\EagleSoft-Backups"

# How many days of backups do you want to keep?
$backupsToKeep = 14

# &lt;-------- End User Configuration --------&gt;


$goToRunBackups = $true; # Initialize variable
$date = (Get-Date -Format yyyy-MM-dd) # Todays date
$dateTime = (Get-Date -Format yyyyMMdd-HHmmss) # Right now

# Make sure the backup location exists
if(!(Test-Path -Path $backupsLocation )){
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Error: Backup destination directory does not exist or cannot be accessed"
    $goToRunBackups = $false;
}
# Make sure the EagleSoft installation dir is set correctly
if(!(Test-Path -Path "$($eagleSoftDirectory)\Shared Files\techaid.exe" )){
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Error: Cannot find techaid.exe. Make sure EagleSoft installation directory is correct"
    $goToRunBackups = $false;
}


if ($goToRunBackups -eq $true) {

    # Setup some variables
    $scratchDir = "$($backupsLocation)\scratchDir-$($dateTime)"
    $backupFilename = "EagleSoft-Backup-$($dateTime).zip"
    $databasesToBackup = (Get-Item -Path "$($eagleSoftDirectory)\Data\*.db")

    # Create a scratch directory for us to use
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Creating scratch directory"
    New-Item -Force -ItemType directory -Path $scratchDir | Out-Null

    # Stop the EagleSoft database
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Stopping EagleSoft"
    &amp; "$($eagleSoftDirectory)\Shared Files\techaid.exe" -stop

    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Waiting 60 seconds for everything to stop"
    Start-Sleep 60

    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Copying database and log files"
    foreach ($database in $databasesToBackup) {

        Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Copying $($database.Name)"
        Copy-Item -path "$($eagleSoftDirectory)\Data\$($database.Name)" -destination $scratchDir

        # Check if a matching log file exists and grab it to
        $logFile = $($database.Name) -replace ".db", ".log"
        if(Test-Path -Path "$($eagleSoftDirectory)\Data\$logFile"){

            Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Copying $logFile"
            Copy-Item -path "$($eagleSoftDirectory)\Data\$logFile" -destination $scratchDir

        }

    }

    # Backup configuration file
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Copying configuration file"
    Copy-Item -path "$($eagleSoftDirectory)\Data\Eaglesoft.Server.Configuration.data" -destination $scratchDir

    # Start the EagleSoft database
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Starting EagleSoft"
    &amp; "$($eagleSoftDirectory)\Shared Files\techaid.exe" -start

    # Compress the backed up files
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Compressing files"
    Compress-Archive -Path $scratchDir -DestinationPath "$($backupsLocation)\$($backupFilename)"

    # Clean-up scratch directory
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Removing scratch directory and files"
    Remove-Item -path $scratchDir -recurse -force

    # Clean-up old backups
    Write-Host "$(Get-Date -Format "yyyy-MM-dd HH:mm:ss") - Purging old backups"
    Get-ChildItem $backupsLocation | Where-Object { $_.LastWriteTime -lt $(Get-Date).AddDays("-$($backupsToKeep)") } | Remove-Item

}

 

You can save this as ‘eagleSoftBackups.ps1‘ (or anything really) on your EagleSoft server and then use ‘Scheduled Tasks’ to schedule it to run nightly or how ever often you want.