Backing up a VM with a PCIe device attached to it with Veeam

In a previous post I talked about installing a Quadro P620 into my ESXi host so I could attach it to my Plex VM. This worked out great except my Veeam backups started failing.

There is a limitation in VMware vSphere where you can’t take a Snapshot of a VM with a PCIe device passed through to it.

One option is to install the Veeam Agent for the OS you’re running and use it to take guest based backups. This isn’t ideal though in my opinion. I would much rather keep my host based backups of the VM. Fortunately this is a easy solution to this problem.

Shut off the VM before taking the Veeam backup and then power it back on after the backup is complete.

To get this working you need to install the VMware PowerShell Module on your Veeam server. To do this perform the following steps:

  1. Right click on the PowerShell shortcut and choose ‘Run as Administrator’
  2. Run the following commands:
    Find-Module -Name VMware.PowerCLI
    Install-Module -Name VMware.PowerCLI -Scope AllUsers
    Get-Command -Module *VMWare*
    Set-PowerCLIConfiguration -Scope AllUsers -ParticipateInCeip $false -InvalidCertificateAction Ignore
  3. You should see a large list of VMware PowerShell commands output which means you’ve successfully installed the module

Next up you need to make sure your Veeam Services are running under a Service Account with the appropriate permissions in vCenter. I believe this is normally a best practice and chances are you’ve all already done this. In my case I’d installed Veeam as a local service. Don’t know why but to fix it I just flipped over the following Windows Services to run as my backup operator account which had Domain Admin, Backup Operator, Local Admin on the Veeam Server and Administrator on vSphere permissions already.

The services were:

  • Veeam Backup Enterprise Manager
  • Veeam Backup Service
  • Veeam Broker Service
  • Veeam Cloud Connect Service
  • Veeam Guest Catalog Service
  • Veeam RESTful API Service

I then rebooted my Veeam server.

I already have my vCenter service joined to my domain but I did run into an issue where single sign-on wasn’t working properly. If I attempted to connect to my vCenter server via PowerShell using “Connect-VIServer <VCENTER SERVER FQDN>” I would be prompted for credentials which shouldn’t be happening since the account I’m logged in as is an Administrator in vCenter.

Turned out I needed to add my AD Group that gives users Administrative access to the vCenter Global Permissions list:

  1. Login to vCenter as an administrator
  2. Click ‘Menu’ and ‘Administrator’
  3. Click ‘Global Permissions’
  4. Click ‘Add’
  5. Change the ‘User’ field to your domain, search for the user or security group (I recommend security groups) and select it, make sure the role is ‘Administrator’ and check ‘Propagate to children’ and click ‘Ok’

After doing this I could run “Connect-VIServer <VCENTER SERVER FQDN>” and not be prompted for credentials.

Now that all the prep-work is done we can re-configure our backup job in Veeam.

First we’re going to need two scripts, one to shutdown the VM and one to boot it back up. I’ve saved these scripts on my Veeam server in “C:\Scripts\<VM FQDN>\”

The shutdown script is “shutdown.bat”, be sure to search and replace “VCENTER FQDN” and “VM FQDN” with your values:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -executionpolicy bypass -Command Connect-VIServer -Server "VCENTER FQDN"; Shutdown-VMGuest -VM "VM FQDN" -Server VCENTER FQDN -Confirm:0; do{$vm=Get-VM -Name "VM FQDN"}while ($vm.PowerState -eq \"PoweredOn\")

The startup script is “startup.bat”, be sure to search and replace “VCENTER FQDN” and “VM FQDN” with your values:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -executionpolicy bypass -Command Connect-VIServer -Server "VCENTER FQDN"; Start-VM -VM "VM FQDN" -Server "VCENTER FQDN"; Start-Sleep -s 90

Once you’ve created these fire up the Veeam console and re-configure the VMs job:

  1. Launch Veeam
  2. Find the backup job for your VM, right click on it and choose ‘Edit’
  3. Go to ‘Storage’
  4. Click ‘Advanced’
  5. Go to ‘Scripts’
  6. Checkmark ‘Run the following script before the job:’ and select your “shutdown.bat” script
  7. Checkmark ‘Run the following script after the job:’ and select your “startup.bat” script
  8. Click ‘Ok’
  9. Click ‘Finish’
  10. Perform a test run of the job, you can monitor the start-up/shutdown in vCenter

That’s it. Minor inconvenience but it works. Hopefully vSphere 7 will allow for snapshots on VMs with pass-through devices configured.

References

Some Microsoft Storage Spaces Benchmarks

My backup server has a ASRock H370M-ITX/AC motherboard in it and at the time of these benchmarks 3x6TB Seagate Ironwolf SATA disks.

I run Veeam and used the SATA disks as my backup repository.

My original configuration was 2x6TB Ironwolfs in a RAID0 (using Microsoft Storage Spaces) with Bitlocker enabled. This worked perfectly fine and I had no performance issues.

A project I was working on required me to add some redundancy to my backup storage so I purchased the 3rd disk and re-configured the Microsoft Storage Space as a RAID5 and re-enabled Bitlocker. Since then I had nothing but performance issues. When two backup jobs ran at the same time the server became nearly unresponsive. The jobs still ran and completed but it was very difficult to use the server and jobs took longer with the RAID5 configuration than the RAID0 configuration. A performance difference makes sense but this amount seemed abnormal.

I ended up with a spare LSI MegaRAID 9270-8i I couldn’t sell so I decided to throw it into my backup server and try running the above configuration with hardware RAID but before I did that, I ran some benchmarks.

As you can see BitLocker has a huge negative impact on this configuration even though the server is running a Intel Core i3-8100 which has hardware acceleration built in for encryption.

You can probably guess how this is all going to end now.

First a 3 disk RAID0:

Last but not least, a 3 disk RAID5:

It’ still a ~70% hit on sequential writes but the server is completely usable and backup jobs run at the speeds I would expect over 1GBe.

How to remove the NetApp Host Utilities hotfix check

 

Update 2018-10-18Check out this comment. Looks like a much easier method than my MSI editing one.

 

Sick of the hotfix requirements for NetApp Host Utilities? Me to.

On a test/dev server I have I’ve been experimenting with Veeam, iSCSI, ReFS and Server 2016. I wanted to run the NetApp Host Utilities against my Server 2016 box and was told I was missing Q3197954:

Problem is this 2016 server is fully patched and if I go manually download Q3197954 (which appears to be a previous rollup) I can’t install it on my fully patch server.

On a hunch I figured I might be able to crack open the MSI and find a way around the hotfix check and I was right. Here is how you do it:

  1. Download and installĀ Orca MSI Editor (Alternate Download)
  2. Download the NetApp Host Utilities package from https://now.netapp.com/ (v7.1 as of this writing)
  3. Launch Orca and open the MSI file with it
  4. Select ‘InstallUISequence’ on the left and then click the ‘Sequence’ column on the top right to sort by lowest value first
  5. Locate ‘CheckMSHotfixes’, right click on it and choose ‘Drop row’
  6. Click ‘Save’
  7. Close Orca
  8. Copy the edited MSI to your server and run the installation as per normal

Digging around the MSI a bit I found references to “IGNORE_HOTFIX_CHECK” listed. I suspect it’s some kind of flag or environment variable that could be set to accomplish the same thing but I was unable to find any documentation on how and a few guesses around using environment variables didn’t pan out so I decided to stick with the above solution.

A note of caution. This is likely 100% unsupported by NetApp and if they found out you did this to get the Host Utilities on your server you might run into support issues.