SolarWinds Web Helpdesk 12.7.2 post-upgrade problems

Recently after upgrading our WHD from 12.7.1 to 12.7.2 we started experiencing two issues that turned out to be related.

Adding comments to tickets could generate an error stating: “Something went wrong. Please contact SolarWinds support for assitance or review the Web Hep Desk logs.”

and when trying to upload attachments to tickets: “Your upload failed: Connection reset by peer: Amount read didn’t match content-length”

After some back and forth with SolarWinds support they said the issue is related to the version of Tomcat they bundled with WHD 12.7.2. The bundled version is 9.0.31 and SolarWinds recommends manually upgrading it to 9.0.34 to fix the issue.

The procedure I followed was:

# Login to your WHD server and become root

# Make a temp directory, go into it and download Tomcat 9.0.34
mkdir temp
cd temp
wget -c https://muug.ca/mirror/apache-dist/tomcat/tomcat-9/v9.0.34/bin/apache-tomcat-9.0.34.tar.gz

# Stop WHD
service stop webhelpdesk

# Backup your existing WHD Tomcat directory just in case
cd /usr/local/webhelpdesk/bin
tar -cf tomcatBackup-20200423.tar tomcat
gzip -9 tomcatBackup-20200423.tar

# Copy Tomcat 9.0.34 overtop of Tomcat 9.0.31 while preserving WHD related content
cd ~/temp
cd apache-tomcat-9.0.34
cp -Raf bin conf lib NOTICE RELEASE-NOTES /usr/local/webhelpdesk/bin/tomcat/

# Reboot your server or manually start WHD back up
shutdown -r now
# OR
service webhelpdesk start

Once we did this the errors above went away.

How to use CIRA Canadian Shield with a Pi-Hole and DoH

CIRA (Canadian Internet Registration Authority) has recently launched a new DNS service called the “Canadian Shield” which is basically a DNS service similar to OpenDNS or Cloudflares 1.1.1.1 for Canadians, by Canadians.

CIRA offers three levels of protection depending on how safe you want to be:

  • Private: DNS resolution service that keeps your DNS data private from third-parties.
  • Protected: Includes Private features and adds malware and phishing blocking.
  • Family: Includes Protected and Private features and blocks pornographic content.

We use the Enterprise version of this service at my place of work and based on how we use it I’d say we’re using the equivalent of their “Protected” offering. We’ve had zero issues with the service and defiantly feels like it adds an extra layer or protection to our users.

Alright, enough free advertising (I am not receiving compensation from CIRA for this post).

When this was first announced I was eager to try it at home. CIRA’s instructions are how to either configure DNS over HTTPS (DoH) on a per-browser basis (not ideal for me since I have many devices on my network and don’t only use Firefox/Chrome) or configure your outbound DNS to use their servers over traditional, un-encrypted, DNS queries.

What I really want is to configure my Pi-hole which is my DNS endpoint for anything on my network to use the new CIRA service. This would capture ALL outbound DNS traffic and send it to CIRA making it so I only have to configure things in one place.

My current setup is: Clients -> 1 of 2 Active Directory DNS Servers -> Pi-hole -> Cloudflare via DoH (1.1.1.1 / Cloudfared)

This will easily work on a more traditional deployment of: Clients -> Pi-hole -> Cloudflare via DoH (1.1.1.1 / Cloudfared)

My Pi-hole is a basic CentOS 8 VM with the Pi-hole software installed on it with cloudflared so I can take advantage of DoH for all of my outbound DNS traffic. This is the minimum you need to get this working, a functional Pi-hole that is already sending it’s outbound DNS queries to Cloudflare (or another DoH provider) via cloudflared.

The first thing I did was re-configure my cloudflared to simply try using the CIRA DoH:

# Edit the cloudflared configuration file
vim /etc/default/cloudflared

# Commandline args for cloudflared to use 1.1.1.1
CLOUDFLARED_OPTS=--port 5053 --upstream https://1.1.1.1/dns-query --upstream https://1.0.0.1/dns-query

# Changed the above to:
CLOUDFLARED_OPTS=--port 5053 --upstream https://private.canadianshield.cira.ca/dns-query

# Save and close the file

# Restart cloudflared
systemctl restart cloudflared

# Test DNS
nslookup google.ca

# This failed

 

This ended up not working and after I tried it I realized why. To be able to use private.canadianshield.cira.ca you have to have functioning DNS. Cloudflare has skirted this issue by using 1.1.1.1 and 1.0.0.1 for it’s service, they are IPs and do not need DNS to be working to function.

Fortunately the solution was very easy:

# I did a nslookup on the specific CIRA service I wanted to use (Private) via traditional DNS

nslookup private.canadianshield.cira.ca 1.1.1.1

Server:         192.168.0.4
Address:        192.168.0.4#53

Non-authoritative answer:
Name:   private.canadianshield.cira.ca
Address: 149.112.121.10
Name:   private.canadianshield.cira.ca
Address: 149.112.122.10
Name:   private.canadianshield.cira.ca
Address: 2620:10a:80bb::10
Name:   private.canadianshield.cira.ca
Address: 2620:10a:80bc::10

# If you want to use Protected or Family instead of private 
# do a nslookup on either protected.canadianshield.cira.ca
# or family.canadianshield.cira.ca instead and use those IPs

# I then edited my /etc/hosts file on my Pi-hole
vim /etc/hosts

# and added the following (I don't use IPv6 at this time):

# CloudA DNS
149.112.121.10 private.canadianshield.cira.ca
149.112.122.10 private.canadianshield.cira.ca

# Save and close the file

# Test DNS
nslookup cira.ca
Server:         192.168.0.4
Address:        192.168.0.4#53

Non-authoritative answer:
Name:   cira.ca
Address: 52.60.203.65

 

That’s it. I now have the CIRA Canadian Shield working on my Pi-hole using the cloudflared software.

Now for some DNS benchmarks, what’s faster? CIRA or 1.1.1.1?

I flushed the DNS cache on my Active Directory DNS servers and then restarted cloudflared on my Pi-hole before running these benchmarks:

1.1.1.1.1 / 1.0.0.1

  192.168.  0.  4 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.000 | 0.000 | 0.000 | 0.000 | 100.0 |
  + Uncached Name | 0.009 | 0.052 | 0.196 | 0.043 | 100.0 |
  + DotCom Lookup | 0.009 | 0.013 | 0.027 | 0.004 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                   CAYENNE.nom.fizi.ca
                Local Network Nameserver


  192.168.  0.  5 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.000 | 0.000 | 0.000 | 0.000 | 100.0 |
  + Uncached Name | 0.009 | 0.051 | 0.195 | 0.044 | 100.0 |
  + DotCom Lookup | 0.010 | 0.014 | 0.028 | 0.003 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                 HORSERADISH.nom.fizi.ca
                Local Network Nameserver

 

and here are the three CIRA services:

CIRA - Private

  192.168.  0.  4 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.000 | 0.000 | 0.000 | 0.000 | 100.0 |
  + Uncached Name | 0.019 | 0.062 | 0.236 | 0.048 | 100.0 |
  + DotCom Lookup | 0.022 | 0.046 | 0.086 | 0.020 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                   CAYENNE.nom.fizi.ca
                Local Network Nameserver


  192.168.  0.  5 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.000 | 0.000 | 0.000 | 0.000 | 100.0 |
  + Uncached Name | 0.014 | 0.068 | 0.238 | 0.056 | 100.0 |
  + DotCom Lookup | 0.023 | 0.047 | 0.075 | 0.019 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                 HORSERADISH.nom.fizi.ca
                Local Network Nameserver




CIRA - Protected

  192.168.  0.  4 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.000 | 0.000 | 0.000 | 0.000 | 100.0 |
  + Uncached Name | 0.019 | 0.062 | 0.236 | 0.048 | 100.0 |
  + DotCom Lookup | 0.022 | 0.046 | 0.086 | 0.020 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                   CAYENNE.nom.fizi.ca
                Local Network Nameserver


  192.168.  0.  5 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.000 | 0.000 | 0.000 | 0.000 | 100.0 |
  + Uncached Name | 0.014 | 0.068 | 0.238 | 0.056 | 100.0 |
  + DotCom Lookup | 0.023 | 0.047 | 0.075 | 0.019 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                 HORSERADISH.nom.fizi.ca
                Local Network Nameserver



CIRA - Family

  192.168.  0.  4 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.000 | 0.000 | 0.000 | 0.000 | 100.0 |
  + Uncached Name | 0.014 | 0.064 | 0.246 | 0.053 | 100.0 |
  + DotCom Lookup | 0.022 | 0.039 | 0.080 | 0.018 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                   CAYENNE.nom.fizi.ca
                Local Network Nameserver


  192.168.  0.  5 |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  + Cached Name   | 0.000 | 0.000 | 0.000 | 0.000 | 100.0 |
  + Uncached Name | 0.014 | 0.073 | 0.248 | 0.062 | 100.0 |
  + DotCom Lookup | 0.024 | 0.049 | 0.082 | 0.020 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+
                 HORSERADISH.nom.fizi.ca
                Local Network Nameserver

 

If I am interpreting this data correctly, CIRA is 0.010s-0.030s slower on average compared to 1.1.1.1. Hardly worth mentioning.

I’m happily switching over to a Canadian based DoH service (full disclosure, I’m Canadian). No offence Cloudflare, you still get to hold my DNS until CIRA starts offering their DNS Anycast Service for home users (hint hint).

Oh and just in case your curious, if you choose their ‘Family’ service and try and hit up Pornhub, you’re greeted with this:

Adding a Quadro P620 to my Plex VM

I currently run Plex in a CentOS 7 VM (on top of vSphere 6.7) with two 2vCPUs and 2GB of vRAM.

When I needed to transcode video to sync it to a mobile device for a trip the process takes a while and consumes a lot of CPU on the VM. I could just add more vCPUs to the VM but I have a limit on how much CPU I have to toss around and there are more efficient ways to transcode video.

I bought my Dell T340 specifically with a Xeon E-2176G CPU in it so I could take advantage of the on-board GPU to handle my transcoding work. After a bunch of back and forth with VMware, Dell and Intel it turns out that Dell did not build the T340 in a way that it can actually use the on-board GPU on my CPU. Why they offer it as a choice, I don’t know but here we are.

My next option was to purchase a video card to do the work. I did some research and came up with the Quadro P620 (specifically the PYN version) being the most affordable with the features I wanted, specifically NVENC. Added bonus, it supports HEVC (H.265) which should future-proof me for a while and allow me to eventually take advantage of this card for transcoding my Blurays to H.265, but that’s another post.

The card arrived, I installed it, enabled it for passthruough in vSphere, attached it to my Plex VM and booted it up.

I downloaded the latest nVidia driver to my VM and ran the installer (as root):

[[email protected] ~]# chmod a+x NVIDIA-Linux-x86_64-430.50.run
[[email protected] ~]# ./NVIDIA-Linux-x86_64-430.50.run

The installation was straight forward, it in fact took care of everything I needed. It automatically blacklisted the default video device for me, asked me to reboot and re-run the installer, which I did and everything almost worked.

After the drive was successfully installed I ran the nvidia tool provided with the drivers to verify things and was greeted with:

[[email protected] ~]# nvidia-smi

Unable to determine the device handle for GPU 0000:03:00.0: Unknown Error

Fortunately this issue is well documented on the internet and the quick fix was to shut down the VM and make a tweak to it’s configuration. Since I have vCenter I used the GUI to solve this problem instead of downloading the VMX file, editing it and re-uploading the VMX file for the VM:

  1. Login to vCenter
  2. Right click and choose ‘Edit Settings’ on the VM
  3. Go to ‘VM Options’ and expand ‘Advanced’
  4. Click ‘Edit Configuration’
  5. Click ‘Add Configuration Params’
  6. Enter the following without quotes:
    Name: “hypervisor.cpuid.v0”
    Value: “FALSE”
  7. Click ‘Ok’
  8. Click ‘Ok’
  9. Boot up the VM

Once the VM came back up I got the output I was expecting from nvidia-smi

[[email protected] ~]# nvidia-smi

Thu Oct 24 18:36:20 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:13:00.0 Off |                  N/A |
| 40%   54C    P0    N/A /  N/A |     10MiB /  2000MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

The last thing to do before testing was to make sure Plex was configured to use hardware transcoding:

  1. Login to your Plex’s WebUI
  2. Under ‘Settings’ click ‘Transcoder’
  3. Checkmark ‘Use hardware acceleration when avalible’
  4. Click ‘Save Changes’

I then gave things a quick test by trying to sync a TV show to my iPhone and then re-ran nvidia-smi:

[[email protected] ~]# nvidia-smi

Thu Oct 24 18:38:59 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro P620         Off  | 00000000:13:00.0 Off |                  N/A |
| 41%   57C    P0    N/A /  N/A |    177MiB /  2000MiB |     20%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     22510      C   /usr/lib/plexmediaserver/Plex Transcoder     167MiB |
+-----------------------------------------------------------------------------+

Bingo, that was it. Now. How much faster was the Quadro P620 over my Xeon E-2176G, roughly 4.5x faster.

My Plex transcoding settings are:

  • Transcoder quality: Prefer higher quality encoding
  • Background transcoding x264 preset: Medium
  • Maximum simultaneous video transcode: 4

But wait you might say, why set “Maximum simultaneous video transcode” to “4”? A Quadro P620 can only do 2?

This is why, only took a few seconds as root:

# git clone https://github.com/keylase/nvidia-patch.git
# cd nvidia-patch/
# bash patch.sh

 

 

Accessing a Pi-Hole behind an Apache reverse proxy

Update 2019-09-15: Finally got around to looking into this and it turns out all I had to change was “ProxyPreserveHost Off” to “ProxyPreserveHost¬† On” to get things working. I’ve updated the original post to reflect the changes. I also didn’t note in my original host that I purposely restricted access to the apache virtual host to 10.0.0.0/24 and 192.168.0.0/24 (my internal networks). You’ll want to update the “Allow from” lines to reflect your internal networks OR remove the “<Location /></Location>” all together to make it accessible from anywhere (not recommended).

Update 2019-08-19: I just recently found out that this proxy configuration only allows read-only access to the Pi-Hole UI. I was attempting to white-list a domain and it was failing when accessing my Pi-Hole via the proxy. I had to go directly to the box’s FQDN to white-list a domain. I will leave this post for reference and update it when I figure out a fix to this problem.

Update 2019-09-29: My first Lets Encrypt certificate came due for auto-renewal and failed because of my original configuration. I’ve updated the apache configuration below so Lets Encrypt can access the non-SSL /.well-known directory to automatically renew certificates.

Original Post

Today I got tired of accessing my Pi-Hole over HTTP, having to remember to put /admin/ in the URL and having to load up a browse that wasn’t Vivaldi or Firefox because they don’t have an easy way to ignore Strict-Transport-Security for my domain.

I checked out some documentation about adding SSL to the Pi-Hole directly but have concerns that future updates will wipe out all the custom configuration to lighttpd. According to this you also have to be careful when enabling SSL on your Pi-Hole as it could interfere with blocking.

I already have an Apache webserver running so configuring it to reverse-proxy seemed like an easier task, plus if for some reason I wanted to access my Pi-Hole from the general internet (without VPN) it would be simple to enable that.

Here is the reverse proxy configuration I used with a restriction to my two internal networks and a redirect from HTTP to HTTPS:

<VirtualHost *:80>
       ServerName pihole.mydomain.com
       DocumentRoot /var/www/html
       CustomLog logs/pihole.mydomain.com.log combined
       ErrorLog logs/pihole.mydomain.com-error.log

	<Location /.well-known>
		Order allow,deny
		Allow from all
	</Location>

	<Location />
		Order deny,allow
		Deny from all
		Allow from 127.0.0.1
		Allow from 192.168.0.0/24
		Allow from 10.0.0.0/24
	</Location>

       RewriteEngine On
       RewriteRule ^(.well-known)($|/) - [L]
       RewriteCond %{SERVER_PORT} 80
       RewriteRule ^(.*)$ https://pihole.mydomain.com/ [R,L]
</VirtualHost>

<VirtualHost *:443>
        ServerName pihole.mydomain.com
        DocumentRoot /var/www/html
        CustomLog logs/pihole.mydomain.com.log combined
        ErrorLog logs/pihole.mydomain.com-error.log

	<Location />
		Order deny,allow
		Deny from all
		Allow from 127.0.0.1
		Allow from 192.168.0.0/24
		Allow from 10.0.0.0/24
	</Location>

        RewriteEngine On
        RewriteRule ^/$ /admin [R]

        # The below line is not required so I have commented it out but left it in place
        # in case I am wrong. I no longer use Apache so I can't test. If this config doesn't
        # work uncomment the below line.
        # ProxyRequests On
        ProxyPass /  http://pihole-internal-hostname.mydomain.com/
        ProxyPassReverse / http://pihole-internal-hostname.mydomain.com/
        ProxyPreserveHost On

        SSLEngine on
        SSLHonorCipherOrder off
        Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"

        SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
        SSLCipherSuite ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

        SSLCertificateFile /etc/letsencrypt/live/pihole.mydomain.com/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/pihole.mydomain.com/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/pihole.mydomain.com/chain.pem
</VirtualHost>

I am aware that my SSL configuration is not the best. I’m waiting for CentOS 8 to come out before migrating off my existing CentOS 6 server.

To find the best SSL configuration for your OS and Web Server I recommend checking out Mozilla’s SSL Configuration Generator: https://ssl-config.mozilla.org/

How to install pfSense in DigitalOcean

Inspired by this post, I’m basically re-creating it with copy/paste commands instead of images of the commands and updating the partitioning portion as I found some steps the original author took are no longer required.

Create your droplet

  1. Login to your DigitalOcean Dashboard and create a new droplet
  2. Select ‘FreeBSD 11.1 x64’ as your droplet image
  3. Select the data center region of your choice
  4. Check mark ‘Private Networking’ and ‘IPv6’ if you want it
  5. Add your SSH key
  6. Enter a hostname
  7. Click ‘Create’

Once the droplet has been created boot it up, grab the public IP and SSH into it as root.

Note: If you don’t SSH in as root put “sudo” in front of all of the commands after step 7

  1. Go to https://www.pfsense.org/download/
  2. Select ‘AMD64 (64-bit)’ as the architecture
  3. Select ‘USB Memstick Installer’ as the installer
  4. Select ‘VGA’ for the console
  5. Pick which ever mirror you want
  6. Right click the ‘Download’ button and choose ‘Copy Link Location’
  7. On your SSH connection to your droplet run the following command:
    cd /tmp
    curl -O <URL FROM STEP 6>
    
    # Example
    curl -O https://nyifiles.pfsense.org/mirror/downloads/pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64.img.gz
  8. Disable SWAP
    swapoff -a
  9. Enable debug mode for GEOM, more info on why here
    sysctl kern.geom.debugflags=0x10
  10. Write the ISO of pfSense to /dev/vtbd0
    gunzip <PFSENSE DOWNLOAD> | dd of=/dev/vtbd0 bs=512k
    
    # Example:
    gunzip -c pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64.img.gz | dd of=/dev/vtbd0 bs=512k

     

  11. You can now reboot the droplet and the the pfSense installer will start
    reboot

Go back to the DigitalOcean interface, select your droplet and open the console window

  1. Once the installer starts hit <ENTER> to accept the copy right notice
  2. Choose ‘Install’
  3. Choose ‘>>> Continue with default keymap’
  4. Choose ‘Manual’
  5. Delete everything listed EXCEPT for vtbd0, vtbd0s2 and vtbd0s2a
  6. Highlight vtbd0 and press ‘C’ and choose ‘OK’
  7. Select vtbd0s1 and press ‘C’
  8. Change the mount point to “/” and choose ‘OK’
  9. Choose ‘Finish’
  10. Choose ‘Commit’
  11. The installation will now progress, once complete choose ‘No’ and ‘Reboot’

Once the droplet reboots you’ll be at the initial configuration wizard for setting up pfSense. Since this is deployment specific I will leave it to you to configure.