How to install pfSense in DigitalOcean

Inspired by this post, I’m basically re-creating it with copy/paste commands instead of images of the commands and updating the partitioning portion as I found some steps the original author took are no longer required.

Create your droplet

  1. Login to your DigitalOcean Dashboard and create a new droplet
  2. Select ‘FreeBSD 11.1 x64’ as your droplet image
  3. Select the data center region of your choice
  4. Check mark ‘Private Networking’ and ‘IPv6’ if you want it
  5. Add your SSH key
  6. Enter a hostname
  7. Click ‘Create’

Once the droplet has been created boot it up, grab the public IP and SSH into it as root.

Note: If you don’t SSH in as root put “sudo” in front of all of the commands after step 7

  1. Go to https://www.pfsense.org/download/
  2. Select ‘AMD64 (64-bit)’ as the architecture
  3. Select ‘USB Memstick Installer’ as the installer
  4. Select ‘VGA’ for the console
  5. Pick which ever mirror you want
  6. Right click the ‘Download’ button and choose ‘Copy Link Location’
  7. On your SSH connection to your droplet run the following command:
  8. Disable SWAP
  9. Enable debug mode for GEOM, more info on why here
  10. Write the ISO of pfSense to /dev/vtbd0

     
  11. You can now reboot the droplet and the the pfSense installer will start

Go back to the DigitalOcean interface, select your droplet and open the console window

  1. Once the installer starts hit <ENTER> to accept the copy right notice
  2. Choose ‘Install’
  3. Choose ‘>>> Continue with default keymap’
  4. Choose ‘Manual’
  5. Delete everything listed EXCEPT for vtbd0, vtbd0s2 and vtbd0s2a
  6. Highlight vtbd0 and press ‘C’ and choose ‘OK’
  7. Select vtbd0s1 and press ‘C’
  8. Change the mount point to “/” and choose ‘OK’
  9. Choose ‘Finish’
  10. Choose ‘Commit’
  11. The installation will now progress, once complete choose ‘No’ and ‘Reboot’

Once the droplet reboots you’ll be at the initial configuration wizard for setting up pfSense. Since this is deployment specific I will leave it to you to configure.

Detecting duplicate accounts in Web Help Desk

We run a copy of Web Help Desk and from time to time we end up with duplicate client entries. Our WHD is connected to AD so you’d think this wouldn’t be possible but….. surpriiiiiiise.

Our monitoring platform, PRTG, allows us to run SQL queries and alert based on the response we get. I’ve created a monitoring rule for this SQL query (with this excellent help):

If it ever returns more than 0 rows it means there is a duplicate account we need to find, merge and purge using SolarWinds procedure: https://support.solarwinds.com/Success_Center/Web_Help_Desk_(WHD)/Knowledgebase_Articles/Merge_duplicate_WHD_clients_in_bulk

Our Web Help Desk is backed by a MSSQL Database instead of the built-in PostgreSQL. You may need to tweak the query if you are using PostgreSQL and can even get into the embedded database.

Silencing my Dell T340

I recently upgraded my Homelab (the thing that hosts this very blog) from a custom built server to a Dell T340. I have experience with the tower line of Dell servers from work and they’ve always been fairly quiet I’ve found once they’ve finished their power-on self-tests.

Not the case with the new T340. I did some testing with my iPhone and the dB Meter & Spectrum Analyzer App, holding it about 1 foot away from the front of the case, and got a average reading of ~52db-62db while the server wasn’t working very hard, in fact it was nearly idle. The ramping up and down of the fan is fairly annoying to, fair hairdryer like.

The T340 is cooled by one single 120mm fan at the back of the case. There is a plastic shroud that Dell installed that basically creates a air channel from the back of the HHD backplane, through the heatsink on the CPU and out the back of the case:

After looking up the fans model number I found out why it was so loud.

Model: Sunon PSD1212PMB1-A
Dell P/N: 9X5J5-A00
Airflow: 226.5
CFM Speed: 6000 RPM
Noise: 65.5 dBA

Since this server sits in my office at home I wanted to find a solution to this problem with out removing the OEM CPU heatsink.

After taking a few measurements I determined that a 96mm fan should fit nicely on the OEM CPU heatsink and any old 120mm fan will work to replace the case fan. I ended up purchasing the following:

One other problem with the T340. There is only a single on-motherboard connector for a fan and it’s a proprietary 5-pin connection and there are only two SATA power connections available so in addition to the fans I purchases:

With the above adapters I can connect the 120mm fan directly to the motherboard with PWM support which should keep the iDRAC (BMC) happy and adapt one of my SATA power connectors into two fan connectors in case I decided to install a 3rd fan (more on that later).

Replacing the 120mm is trivial. Release the bracket holding the current fan, disconnect the power cable, remove the OEM fan, install your replacement fan, connect the 4-pin -> 5-pin adapter, slide the fan back in place and connect it to the motherboard.

Next up is the CPU fan, this one is a bit more tricky since there are no screw mounts for it. I found this trick online:

You’ll notice in the heat sink photo I ran 4 zap straps through the heat sink. In the end I removed the bottom two and went with an alternative solution. That small heat sink you see is just high enough that it pushed the fan I was going to put on the CPU heat sink up 2-3mm which would have caused the top of the fan to rest above the top of the CPU heat sink. This would have likely caused problems re-installing the shroud and could have caused the fan blades to impact the heat sink causing a failure or annoying ticking noise.

Instead I did the following for the bottom two fan mounting holes:

Finally I took two additional zap straps and fed their locking heads on to the top two straps on the fan and tightened everything up:

After verifying everything was snug and the fan fired up I snipped off the excess zap strap ends and re-installed the shroud over the CPU and powered everything back up.

My T340 is now nearly silent. I think the hard drives might be louder than the fans inside the case at this point.

Shortly after booting the system back up the iDRAC started complaining about the fan’s speed. The OEM fan is 6000RPM and the replacement fan is only 1700RPM. To address this I did the following:

  1. Login to the iDRAC
  2. Click ‘System’ and ‘Overview’
  3. Click ‘Cooling’
  4. To the right of ‘Fan Overview’ click ‘Configure Fans’
  5. Change the ‘Minimum Fan Speed in PWM (% of Max)’ to “50” and click ‘Apply’

Since making this change I haven’t received further fan speed warnings and I still can’t hear the fans. 

Unfortunately I was not doing any kind of temperature logging with the OEM fan installed so I can’t comment on whether or not this has made the CPU temperatures higher or lower. If I were to guess I’d say the overall operating temperature has increased with the custom fans over the OEM fans.

I ran my server with the custom fans for about 24 hours before I got some logging configured.

Right after I configured temperature logging I shut down my server and installed one additional 140mm fan in the 5.25″ cage to suck fresh air into the case. I also removed the shroud thinking that cooling would be better with out it.

To get this 140mm fan to stay in place I used two small pieces of double sided duct tape.

Finally, here is the CPU temperature over the last 3 days:

Looks like my average temperature is 40c with spikes up to 55-60c. These numbers are within the Intel recommended maximum of 100c but I’m not liking these numbers.

Before I removed the shroud and installed the 140mm fan I did have some temperature readings:

The area to the left of the red line was with the shroud still installed. It appears with the shroud installed the average temperature is still the same but the spikes are less severe.

I’m out of town right now but when I return I am going to re-install the shroud and run my server for a few days to see if the above remains true. I will update this post accordingly.

Update 2019-05-25

Here is 7 days of standard load on my server with the shroud installed. Doesn’t seem to make much of a difference. That being said, these temps are a little high for my comfort. I’m going to get a Noctua CPU heat sink the next time I have $150 lying around and will create a new post with it’s installation and operating temperatures.

Event ID 20292 from DHCP-Server

Checking over our DHCP server we were seeing quite a few of these errors appearing in the ‘Microsoft-Windows-DHCP Server Events/Admin’ event log:

Researching this error I came across this forum post: https://social.technet.microsoft.com/Forums/en-US/15d00412-3dfc-4520-a74e-1f32fe1329ef/windows-server-2012-dhcp-event-id-20291?forum=winserveripamdhcpdns

Which lead me to this KB article: https://support.microsoft.com/en-ca/help/2955135/event-id-20291-is-logged-in-the-system-log-when-a-client-computer-is-m

The hotfix that Microsoft mentions is from November 2014 and has been installed on our server for a very long time. We never noticed this error back in 2014 when the hotfix was installed so we were not able to “first remove the failover relationship, install the update to both DHCP nodes and restart them, and then reestablish the failover relationship” per Microsoft’s article.

The article leads me to believe you have to deconfigure failover on all subnets, destroy the failover relationship, re-create the failover relationship and then re-configure failover on each subnet.

Turns out you can just right click ‘Deconfigure failover’ and then right click ‘Configure failover’ on the specific subnets having the issue and re-use the existing failover relationship to resolve this issue assuming you’ve installed the November 2014 hotfix.

Microsoft RAS VPN and VXLAN not quite working

I’m not overly knowledgeable about advanced networking but I figured I’d share this since I couldn’t find anything online about it at the time.

We run a Microsoft Remote Access Server (RAS) for our VPN server. We provide L2TP primarily for users.

Due to a limitation in the Windows VPN client our RAS server has two network interfaces, one directly on the internet with a public IP (VLAN1) and one internally with a private IP (VLAN2).

The private IP relays VPN users DHCP/DNS requests to our internal DHCP and DNS servers.

RAS handles the authentication instead of RADIUS and we have our internal routes published via RIP to the RAS server so they can be provided to VPN clients when they connect.

I believe this is a fairly common design.

On the network end, our original design involved spanning VLAN1 and 2 all the way from our edge into our data center so the VM could pretty much sit directly on them. This worked fine.

As part of a major network redesign we performed we changed the VLAN spanning design over to using a VXLAN from our edge into our data center.

After making this change we ran into the strangest VPN issues. Users could connect and ping anything they wanted, do DNS lookups and browse most HTTP websites. HTTPS websites would partially load or fail to load and network share (SMB) access would partially work (you could get to the DFS root but not down to an actual file server).

After many hours of troubleshooting we determined our problem.

The MTU of most devices is configured to default to 1500 bytes. When we started tunneling the traffic through a VXLAN the tunneling added 52 bytes to the packet size making the total packet size 1552 bytes which is just over what most network cards are expecting. This caused large packets to drop (loading a HTTPS website, connecting to a share) but small packets (pings, some HTTP websites) to work fine.

I believe the final solution from our network team was to enable Jumbo packets from end to end of the VXLAN tunnel so it could transmit slightly larger than normal packets.

If you have any specific questions I can relay them to our Network Team and try to get you an answer. No promises :)