MikroTik CSS326 fan installation

It was time to upgrade my networking equipment in my homelab. I needed two 24p 1GB switches with 10GB uplinks to facilitate moving my homelab into my crawl space and out of my office. As things are configured right now I’d easily saturate a 1GB uplink between the switches and since I don’t have any 10GB in my homelab yet the CSS326 fits the bill. I am replacing a single first generation Ubiquiti 24p switch with two CSS-326-254G-2S+RM’s.

After receiving my CSS-326-254G-2S+RM’s I plugged them in to test them and verify my 10GB SFP+ transceivers were working properly. While I was doing that I also hooked them into PRTG using SNMP so see what kind of monitoring data I could pull from them. I had been optimistic that these new switches would run cooler than my Ubuiqiti but that does not seem to be the case. My Ubiquiti (which has a fan) runs at about 76c. The MikroTik ran at about 70c without a fan. An improvement but not a fantastic one.

Below is 1 hour of monitoring data from the MikroTik with a single 10GB SFP+ installed in it running idle but connected to my Ubiquiti via 1GBe and the other CSS-326 via fiber.

CSS-326-254G-2S+RM – 1 hour – CPU Temperature – Average 70c
CSS-326-254G-2S+RM – 1 hour – SFP+ Temperature – Average 43c

The CSS-326-254G-2S+RM has a mount for a 40mm fan and figured I’d just buy a fan and slap it in there. Unfortunately once I popped it open I saw there was no header to connect a fan to. I did some digging and came across this video which shows a significant temperature improvement once a fan is installed and the creator helpfully pointed out where you could tap into the switches board to get power. I also found this helpful forum post that also showed the J2 header and mentioned which connector was positive (+) and which was negative (-).

“YOU DON’T NEED TO DO THIS. Mine runs in a warm rack enclosure and has for 3 years now”

– Someone I know

Using my multimeter I checked the J2 connector and it outputs 24v a few seconds after the switch boots up. From what I have read, the J2 only outputs power if you connect the included power cable to your CSS-326. If you use PoE to power the CSS-326 the J2 connector does not output any power. I did not test this but since I’m using the supplied power cables this isn’t a problem for me.

Noctua is my preferred fan manufacturer but they do not make a 24v 40mm fan which means I had to use a buck converter to drop 24v down to 12v for the Noctua NF-A4x20 I wanted to use. I will put a full parts list at the bottom of this blog post.

I did not solder to the J2 connector as my first step. I just want to show where it is before continuing. Please excuse my poor soldering skills. This is only my 3rd or 4th time seriously soldering something and my first time soldering to a PCB. Using the information I gathered, I am labelling which connectors I treated as positive (+) and negative (-). I might be wrong but it all worked in the end.

J2 before I soldered my wires
J2 after I soldered my wires

Easy step first, I mounted the fan into the CSS-326 using some M3x12mm screws, nuts and a little patience.

I then 3D printed a baffle to block off the dead space to the right of the fan if you’re looking at the switch from the front. The electrical tape is just to hold the baffle in place while I’m fiddling inside the switch. Once you put the top back on the baffle is firmly pinched in place and won’t move. As designed the baffle is overkill. It could be 3mm thinner but I don’t care about saving 6g of filament so I didn’t change it for the second switch. The STL is linked at the bottom of this post in the parts list.

Using the cables and adapters that came with the Noctua fan I was able to piece things together in a way that I would never need to touch the J2 connector again if something failed. The buck converter can be detached from the main feed connected to the J2 and the fan can be disconnected from the buck converter. If either piece ever dies it should be very simple to replace them. I specifically used the extension cable and the Y-splitter. You can toss the one labelled “Low-Noise Adapter” into your spare parts bin, we won’t be needing it.

I have labelled each connector with a number so you can see how I piece things together

I removed all of the sheathing from the cables, removed any blue/green wires because I only need the yellow and black ones and then cut off some of the connectors.

Piecing them all together they will look like this:

Numbers in brackets mean a cut

(1) – Is the small connector on the extension cable that gets removed and soldered to J2

2 – Is the large connector on the extension cable that does NOT get removed. The IN on the buck connector will plug into this.

(3) – Is the large connecter on the upper leg of the Y-splitter that gets removed and soldered to the IN on the buck converter

4 – Is the large connector on the lower leg of the Y-splitter that does NOT get removed. The fan will plug into this which is attached to the OUT on the buck connector.

(5) – Is the small connector at the base of the Y-splitter. You want to cut this so that the smaller connector remains attached to the upper leg of the Y-splitter that you removed the large connector (3) from. This then gets soldered to the OUT on the buck converter

6 – Is the fans connector, leave it alone. You will plug it into 4 when everything is done

Before soldering (1) to the J2 connector feed it through the gap in baffle and under the mainboard so you can keep it all tucked out of the way. If you don’t do this first you’ll have to remove the mainboard and baffle to do it later. There should be just enough wire to make it to the J2.

Solder it all together based on the diagram above and plug everything in except for the fan. We need to adjust the buck converter before we can plugin the fan. Odds are its default setting is too high (more than 12v) for our Noctua.

Set your multimeter for DCV at whatever setting can read higher than 20v, connect alligator clips to the OUT side of the buck converter and then connect those to your multimeter. Plug the power into the switch and check your multimeter reading. Using a flathead screw driver carefully turn the small knob on top of the blue box on the buck converter until your multimeter reads 12v.

Initial buck converter setting
Buck converter reading after a few turns

Disconnect the power from the switch, remove your multimeter and alligator clips, screw together the buck converter case and put the top back on the CSS-326. You’re done!

My final results were that the switch ended up running about 30c cooler and the SFP was 11c cooler.

CSS-326-254G-2S+RM – 1 hour – CPU Temperature – Average 40c
CSS-326-254G-2S+RM – 1 hour – SFP+ Temperature – Average 32c

I get MikroTik saving cost by not including a fan in the switch but I really wish they would have at least installed a connector on the J2 to make adding a fan an easy option.

Update – 2022-08-20

I moved my entire homelab off my old Ubiquiti switch yesterday and have some real word temperatures with actual load on the switch:

The first low section on the left was the switch idling with no load while I configured VLANs and LAGs. The gap is me unplugging it, sliding it under my Ubiquiti switch and powering it back on. The initial high temperature (45.8c) was from the Ubiquiti smothering it while I did cable swaps. I eventually removed the Ubiquiti switch and the temperatures dropped a bit.

Parts List

Buck Converter – I used a “LM2596 DC-DC Buck Converter Step Down Module Power Supply DIP Output 1.25V-30V 3A”. There are a ton of these on Amazon. Here is a non-referral link to a 10pack I bought.

Noctua NF-A4x20 – Since there is plenty of room in the case I went with the 40mm * 20mm version of this fan to get the most air movement possible.

Buck Converter Case – I printed one of these to insulate the buck converter from the chassis of the switch.

Baffle – Completely optional but I designed and printed one of these to block off the section of the case to the right of the fan mount. Seemed pointless to circulate that air since there are no electronics in there except the buck converter.

How to install pfSense in DigitalOcean

Inspired by this post, I’m basically re-creating it with copy/paste commands instead of images of the commands and updating the partitioning portion as I found some steps the original author took are no longer required.

Create your droplet

  1. Login to your DigitalOcean Dashboard and create a new droplet
  2. Select ‘FreeBSD 11.1 x64’ as your droplet image
  3. Select the data center region of your choice
  4. Check mark ‘Private Networking’ and ‘IPv6’ if you want it
  5. Add your SSH key
  6. Enter a hostname
  7. Click ‘Create’

Once the droplet has been created boot it up, grab the public IP and SSH into it as root.

Note: If you don’t SSH in as root put “sudo” in front of all of the commands after step 7

  1. Go to https://www.pfsense.org/download/
  2. Select ‘AMD64 (64-bit)’ as the architecture
  3. Select ‘USB Memstick Installer’ as the installer
  4. Select ‘VGA’ for the console
  5. Pick which ever mirror you want
  6. Right click the ‘Download’ button and choose ‘Copy Link Location’
  7. On your SSH connection to your droplet run the following command:
    cd /tmp
    curl -O <URL FROM STEP 6>
    
    # Example
    curl -O https://nyifiles.pfsense.org/mirror/downloads/pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64.img.gz
  8. Disable SWAP
    swapoff -a
  9. Enable debug mode for GEOM, more info on why here
    sysctl kern.geom.debugflags=0x10
  10. Write the ISO of pfSense to /dev/vtbd0
    gunzip <PFSENSE DOWNLOAD> | dd of=/dev/vtbd0 bs=512k
    
    # Example:
    gunzip -c pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64.img.gz | dd of=/dev/vtbd0 bs=512k

     

  11. You can now reboot the droplet and the the pfSense installer will start
    reboot

Go back to the DigitalOcean interface, select your droplet and open the console window

  1. Once the installer starts hit <ENTER> to accept the copy right notice
  2. Choose ‘Install’
  3. Choose ‘>>> Continue with default keymap’
  4. Choose ‘Manual’
  5. Delete everything listed EXCEPT for vtbd0, vtbd0s2 and vtbd0s2a
  6. Highlight vtbd0 and press ‘C’ and choose ‘OK’
  7. Select vtbd0s1 and press ‘C’
  8. Change the mount point to “/” and choose ‘OK’
  9. Choose ‘Finish’
  10. Choose ‘Commit’
  11. The installation will now progress, once complete choose ‘No’ and ‘Reboot’

Once the droplet reboots you’ll be at the initial configuration wizard for setting up pfSense. Since this is deployment specific I will leave it to you to configure.

Event ID 20292 from DHCP-Server

Checking over our DHCP server we were seeing quite a few of these errors appearing in the ‘Microsoft-Windows-DHCP Server Events/Admin’ event log:

Log Name:      DhcpAdminEvents
Source:        Microsoft-Windows-DHCP-Server
Date:          1/28/2019 10:23:49 PM
Event ID:      20292
Task Category: DHCP Failover
Level:         Error
Keywords:      
User:          NETWORK SERVICE
Computer:      dc2.mydomain.com
Description:
A BINDING-ACK message with transaction id: 943568 was received for IP address: 10.253.166.162 with reject reason:  (Reject Reason Unknown ) from partner server: dc1.mydomain.com for failover relationship: dc1.mydomain.com-dc2.mydomain.com.

Researching this error I came across this forum post: https://social.technet.microsoft.com/Forums/en-US/15d00412-3dfc-4520-a74e-1f32fe1329ef/windows-server-2012-dhcp-event-id-20291?forum=winserveripamdhcpdns

Which lead me to this KB article: https://support.microsoft.com/en-ca/help/2955135/event-id-20291-is-logged-in-the-system-log-when-a-client-computer-is-m

The hotfix that Microsoft mentions is from November 2014 and has been installed on our server for a very long time. We never noticed this error back in 2014 when the hotfix was installed so we were not able to “first remove the failover relationship, install the update to both DHCP nodes and restart them, and then reestablish the failover relationship” per Microsoft’s article.

The article leads me to believe you have to deconfigure failover on all subnets, destroy the failover relationship, re-create the failover relationship and then re-configure failover on each subnet.

Turns out you can just right click ‘Deconfigure failover’ and then right click ‘Configure failover’ on the specific subnets having the issue and re-use the existing failover relationship to resolve this issue assuming you’ve installed the November 2014 hotfix.

Microsoft RAS VPN and VXLAN not quite working

I’m not overly knowledgeable about advanced networking but I figured I’d share this since I couldn’t find anything online about it at the time.

We run a Microsoft Remote Access Server (RAS) for our VPN server. We provide L2TP primarily for users.

Due to a limitation in the Windows VPN client our RAS server has two network interfaces, one directly on the internet with a public IP (VLAN1) and one internally with a private IP (VLAN2).

The private IP relays VPN users DHCP/DNS requests to our internal DHCP and DNS servers.

RAS handles the authentication instead of RADIUS and we have our internal routes published via RIP to the RAS server so they can be provided to VPN clients when they connect.

I believe this is a fairly common design.

On the network end, our original design involved spanning VLAN1 and 2 all the way from our edge into our data center so the VM could pretty much sit directly on them. This worked fine.

As part of a major network redesign we performed we changed the VLAN spanning design over to using a VXLAN from our edge into our data center.

After making this change we ran into the strangest VPN issues. Users could connect and ping anything they wanted, do DNS lookups and browse most HTTP websites. HTTPS websites would partially load or fail to load and network share (SMB) access would partially work (you could get to the DFS root but not down to an actual file server).

After many hours of troubleshooting we determined our problem.

The MTU of most devices is configured to default to 1500 bytes. When we started tunneling the traffic through a VXLAN the tunneling added 52 bytes to the packet size making the total packet size 1552 bytes which is just over what most network cards are expecting. This caused large packets to drop (loading a HTTPS website, connecting to a share) but small packets (pings, some HTTP websites) to work fine.

I believe the final solution from our network team was to enable Jumbo packets from end to end of the VXLAN tunnel so it could transmit slightly larger than normal packets.

If you have any specific questions I can relay them to our Network Team and try to get you an answer. No promises :)

DHCP stops serving IPs when audit log is full

We run two DHCP servers in a HA configuration. The HA is configured to split the scopes in half. Depending on how high up the scope your IP is will determine which DHCP server you get your IP from. We have DHCP audit logging enabled.

DHCP1 handles 0-127 and DHCP2 handles 128-254 (we mostly use /24’s right now).

We started getting reports of random devices on the network not being able to connect or login to the domain. By the time a technician got to the PC to check it the issue was resolved magically.

We dug into the DHCP servers and found the DHCP audit log on DHCP1 was full (36MB in size). The log on DHCP2 was not full (yet, only 34MB in size).

Stopping DHCP on DHCP1, renaming the audit log and then starting DHCP on DHCP1 again appeared to resolve the issue.

The thing that had us scratching our heads is we’ve had this problem before and we had re-configured DHCP on these servers to allow the log files to grow to 250MB but things had stopped at 36MB.

We used this PowerShell to make the change a long while ago and restarted the DHCP service: https://docs.microsoft.com/en-us/powershell/module/dhcpserver/set-dhcpserverauditlog?view=win10-ps

Set-DhcpServerAuditLog -MaxMBFileSize 250

Per the above link it states “-MaxMBFileSize Specifies the maximum size of the audit log, in megabytes (MB).”

It turns out this PowerShell command simply changes the registry value for HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DHCPServer\Parameters\DhcpLogFilesMaxSize which you can just do manually if you’d prefer.

I have no idea how I found it but after some digging I found this article for Server 2008 (we’re using 2012R2): https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc726869(v=ws.10)

It states:

 

Dynamic Host Configuration Protocol (DHCP) servers include several logging features and server parameters that provide enhanced auditing capabilities. You can specify the following features:

  • The file path in which the DHCP server stores audit log files. DHCP audit logs are located by default at %windir%\System32\Dhcp.
  • A maximum size restriction (in megabytes) for the total amount of disk space available for all audit log files created and stored by the DHCP service.
  • An interval for disk checking that is used to determine how many times the DHCP server writes audit log events to the log file before checking for available disk space on the server.
  • A minimum size requirement (in megabytes) for server disk space that is used during disk checking to determine if sufficient space exists for the server to continue audit logging.

 

I’ve bolded and italicized the relevant line. The article also specifically references the registry key the PowerShell command changes.

This leads me to believe the PowerShell documentation is incorrect and “-MaxMBFileSize” specifies the maximum size of all audit logs added together. Not a maximum size per individual audit log.

I checked the directory size of “%windir%\system32\dhcp” on both servers and they were very close to 250MB.

We’ve since made the following change:

Set-DhcpServerAuditLog -MaxMBFileSize 4096

I will update this article if this does not resolve the issue for us.

 

Update 2019-01-10: I can confirm this resolved the issue for us. The log file for the following day reached 54MB with no issue.