Accessing a Pi-Hole behind an Apache reverse proxy

Today I got tired of accessing my Pi-Hole over HTTP, having to remember to put /admin/ in the URL and having to load up a browse that wasn’t Vivaldi or Firefox because they don’t have an easy way to ignore Strict-Transport-Security for my domain.

I checked out some documentation about adding SSL to the Pi-Hole directly but have concerns that future updates will wipe out all the custom configuration to lighttpd. According to this you also have to be careful when enabling SSL on your Pi-Hole as it could interfere with blocking.

I already have an Apache webserver running so configuring it to reverse-proxy seemed like an easier task, plus if for some reason I wanted to access my Pi-Hole from the general internet (without VPN) it would be simple to enable that.

Here is the reverse proxy configuration I used with a restriction to my two internal networks and a redirect from HTTP to HTTPS:

I am aware that my SSL configuration is not the best. I’m waiting for CentOS 8 to come out before migrating off my existing CentOS 6 server.

To find the best SSL configuration for your OS and Web Server I recommend checking out Mozilla’s SSL Configuration Generator: https://ssl-config.mozilla.org/

Some Microsoft Storage Spaces Benchmarks

My backup server has a ASRock H370M-ITX/AC motherboard in it and at the time of these benchmarks 3x6TB Seagate Ironwolf SATA disks.

I run Veeam and used the SATA disks as my backup repository.

My original configuration was 2x6TB Ironwolfs in a RAID0 (using Microsoft Storage Spaces) with Bitlocker enabled. This worked perfectly fine and I had no performance issues.

A project I was working on required me to add some redundancy to my backup storage so I purchased the 3rd disk and re-configured the Microsoft Storage Space as a RAID5 and re-enabled Bitlocker. Since then I had nothing but performance issues. When two backup jobs ran at the same time the server became nearly unresponsive. The jobs still ran and completed but it was very difficult to use the server and jobs took longer with the RAID5 configuration than the RAID0 configuration. A performance difference makes sense but this amount seemed abnormal.

I ended up with a spare LSI MegaRAID 9270-8i I couldn’t sell so I decided to throw it into my backup server and try running the above configuration with hardware RAID but before I did that, I ran some benchmarks.

As you can see BitLocker has a huge negative impact on this configuration even though the server is running a Intel Core i3-8100 which has hardware acceleration built in for encryption.

You can probably guess how this is all going to end now.

First a 3 disk RAID0:

Last but not least, a 3 disk RAID5:

It’ still a ~70% hit on sequential writes but the server is completely usable and backup jobs run at the speeds I would expect over 1GBe.

How to install pfSense in DigitalOcean

Inspired by this post, I’m basically re-creating it with copy/paste commands instead of images of the commands and updating the partitioning portion as I found some steps the original author took are no longer required.

Create your droplet

  1. Login to your DigitalOcean Dashboard and create a new droplet
  2. Select ‘FreeBSD 11.1 x64’ as your droplet image
  3. Select the data center region of your choice
  4. Check mark ‘Private Networking’ and ‘IPv6’ if you want it
  5. Add your SSH key
  6. Enter a hostname
  7. Click ‘Create’

Once the droplet has been created boot it up, grab the public IP and SSH into it as root.

Note: If you don’t SSH in as root put “sudo” in front of all of the commands after step 7

  1. Go to https://www.pfsense.org/download/
  2. Select ‘AMD64 (64-bit)’ as the architecture
  3. Select ‘USB Memstick Installer’ as the installer
  4. Select ‘VGA’ for the console
  5. Pick which ever mirror you want
  6. Right click the ‘Download’ button and choose ‘Copy Link Location’
  7. On your SSH connection to your droplet run the following command:
  8. Disable SWAP
  9. Enable debug mode for GEOM, more info on why here
  10. Write the ISO of pfSense to /dev/vtbd0

     
  11. You can now reboot the droplet and the the pfSense installer will start

Go back to the DigitalOcean interface, select your droplet and open the console window

  1. Once the installer starts hit <ENTER> to accept the copy right notice
  2. Choose ‘Install’
  3. Choose ‘>>> Continue with default keymap’
  4. Choose ‘Manual’
  5. Delete everything listed EXCEPT for vtbd0, vtbd0s2 and vtbd0s2a
  6. Highlight vtbd0 and press ‘C’ and choose ‘OK’
  7. Select vtbd0s1 and press ‘C’
  8. Change the mount point to “/” and choose ‘OK’
  9. Choose ‘Finish’
  10. Choose ‘Commit’
  11. The installation will now progress, once complete choose ‘No’ and ‘Reboot’

Once the droplet reboots you’ll be at the initial configuration wizard for setting up pfSense. Since this is deployment specific I will leave it to you to configure.

Detecting duplicate accounts in Web Help Desk

We run a copy of Web Help Desk and from time to time we end up with duplicate client entries. Our WHD is connected to AD so you’d think this wouldn’t be possible but….. surpriiiiiiise.

Our monitoring platform, PRTG, allows us to run SQL queries and alert based on the response we get. I’ve created a monitoring rule for this SQL query (with this excellent help):

If it ever returns more than 0 rows it means there is a duplicate account we need to find, merge and purge using SolarWinds procedure: https://support.solarwinds.com/Success_Center/Web_Help_Desk_(WHD)/Knowledgebase_Articles/Merge_duplicate_WHD_clients_in_bulk

Our Web Help Desk is backed by a MSSQL Database instead of the built-in PostgreSQL. You may need to tweak the query if you are using PostgreSQL and can even get into the embedded database.

Silencing my Dell T340

I recently upgraded my Homelab (the thing that hosts this very blog) from a custom built server to a Dell T340. I have experience with the tower line of Dell servers from work and they’ve always been fairly quiet I’ve found once they’ve finished their power-on self-tests.

Not the case with the new T340. I did some testing with my iPhone and the dB Meter & Spectrum Analyzer App, holding it about 1 foot away from the front of the case, and got a average reading of ~52db-62db while the server wasn’t working very hard, in fact it was nearly idle. The ramping up and down of the fan is fairly annoying to, fair hairdryer like.

The T340 is cooled by one single 120mm fan at the back of the case. There is a plastic shroud that Dell installed that basically creates a air channel from the back of the HHD backplane, through the heatsink on the CPU and out the back of the case:

After looking up the fans model number I found out why it was so loud.

Model: Sunon PSD1212PMB1-A
Dell P/N: 9X5J5-A00
Airflow: 226.5
CFM Speed: 6000 RPM
Noise: 65.5 dBA

Since this server sits in my office at home I wanted to find a solution to this problem with out removing the OEM CPU heatsink.

After taking a few measurements I determined that a 96mm fan should fit nicely on the OEM CPU heatsink and any old 120mm fan will work to replace the case fan. I ended up purchasing the following:

One other problem with the T340. There is only a single on-motherboard connector for a fan and it’s a proprietary 5-pin connection and there are only two SATA power connections available so in addition to the fans I purchases:

With the above adapters I can connect the 120mm fan directly to the motherboard with PWM support which should keep the iDRAC (BMC) happy and adapt one of my SATA power connectors into two fan connectors in case I decided to install a 3rd fan (more on that later).

Replacing the 120mm is trivial. Release the bracket holding the current fan, disconnect the power cable, remove the OEM fan, install your replacement fan, connect the 4-pin -> 5-pin adapter, slide the fan back in place and connect it to the motherboard.

Next up is the CPU fan, this one is a bit more tricky since there are no screw mounts for it. I found this trick online:

You’ll notice in the heat sink photo I ran 4 zap straps through the heat sink. In the end I removed the bottom two and went with an alternative solution. That small heat sink you see is just high enough that it pushed the fan I was going to put on the CPU heat sink up 2-3mm which would have caused the top of the fan to rest above the top of the CPU heat sink. This would have likely caused problems re-installing the shroud and could have caused the fan blades to impact the heat sink causing a failure or annoying ticking noise.

Instead I did the following for the bottom two fan mounting holes:

Finally I took two additional zap straps and fed their locking heads on to the top two straps on the fan and tightened everything up:

After verifying everything was snug and the fan fired up I snipped off the excess zap strap ends and re-installed the shroud over the CPU and powered everything back up.

My T340 is now nearly silent. I think the hard drives might be louder than the fans inside the case at this point.

Shortly after booting the system back up the iDRAC started complaining about the fan’s speed. The OEM fan is 6000RPM and the replacement fan is only 1700RPM. To address this I did the following:

  1. Login to the iDRAC
  2. Click ‘System’ and ‘Overview’
  3. Click ‘Cooling’
  4. To the right of ‘Fan Overview’ click ‘Configure Fans’
  5. Change the ‘Minimum Fan Speed in PWM (% of Max)’ to “50” and click ‘Apply’

Since making this change I haven’t received further fan speed warnings and I still can’t hear the fans. 

Unfortunately I was not doing any kind of temperature logging with the OEM fan installed so I can’t comment on whether or not this has made the CPU temperatures higher or lower. If I were to guess I’d say the overall operating temperature has increased with the custom fans over the OEM fans.

I ran my server with the custom fans for about 24 hours before I got some logging configured.

Right after I configured temperature logging I shut down my server and installed one additional 140mm fan in the 5.25″ cage to suck fresh air into the case. I also removed the shroud thinking that cooling would be better with out it.

To get this 140mm fan to stay in place I used two small pieces of double sided duct tape.

Finally, here is the CPU temperature over the last 3 days:

Looks like my average temperature is 40c with spikes up to 55-60c. These numbers are within the Intel recommended maximum of 100c but I’m not liking these numbers.

Before I removed the shroud and installed the 140mm fan I did have some temperature readings:

The area to the left of the red line was with the shroud still installed. It appears with the shroud installed the average temperature is still the same but the spikes are less severe.

I’m out of town right now but when I return I am going to re-install the shroud and run my server for a few days to see if the above remains true. I will update this post accordingly.

Update 2019-05-25

Here is 7 days of standard load on my server with the shroud installed. Doesn’t seem to make much of a difference. That being said, these temps are a little high for my comfort. I’m going to get a Noctua CPU heat sink the next time I have $150 lying around and will create a new post with it’s installation and operating temperatures.