Script for detecting potentially vulnerable Log4j jars [CVE-2021-44228] on Windows Server

Update 2021-12-18 – This looks like a much more competent script for detecting this vulnerability and there is a python version for Linux: https://github.com/CERTCC/CVE-2021-44228_scanner

Updated 2021-12-17 – Script is v1.4 and looks for .war files now too

Original post below

Inspired by the one-liner here: https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b#find-vulnerable-software-windows

gci 'C:\' -rec -force -include *.jar -ea 0 | foreach {select-string "JndiLookup.class" $_} | select -exp Path

I wrote a script to expand on the command, support Windows Server 2008 onward and to be more automated.

This script is basically the one liner with a bit of logic to get all the local fixed disks on a server and iterate through them all looking for Log4j jar file:

<# 
.Synopsis 
    Checks the local system for Log4Shell Vulnerability [CVE-2021-44228]
.DESCRIPTION 
    Gets a list of all volumes on the server, loops through searching each disk for Log4j stuff
    Using base search from https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b#find-vulnerable-software-windows

    Version History
        1.0 - Initial release
        1.1 - Changed ErrorAction to "Continue" instead of stopping the script
        1.2 - Went back to SilentlyContinue, so much noise
        1.3 - Borrowed some improvements from @cedric2bx (https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b#gistcomment-3995092)
                Replace attribute -Include by -Filter (prevent unauthorized access exception stopping scan)
                Remove duplicate path with Get-Unique cmdlet
        1.4 - Added .war support thanks to @djblazkowicz (https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b#gistcomment-3998189)
.EXAMPLE 
    .\check_CVE-2021-44228.ps1
.NOTES 
    Created by Eric Schewe 2021-12-13
    Modified by Cedric BARBOTIN 2021-12-14
#> 

# Get Windows Version string
$windowsVersion = (Get-WmiObject -class Win32_OperatingSystem).Caption

# Server 2008 (R2)
if ($windowsVersion -like "*2008*") {

    $disks = [System.IO.DriveInfo]::getdrives() | Where-Object {$_.DriveType -eq "Fixed"}

}
# Everything else
else {

    $disks = Get-Volume | Where-Object {$_.DriveType -eq "Fixed"}

}

# I have no idea why I had to write it this way and why .Count didn't just work
$diskCount = $disks | Measure-Object | Select-Object Count -ExpandProperty Count

Write-Host -ForegroundColor Green "$(Get-Date -Format "yyyy-MM-dd H:mm:ss") - Starting the search of $($diskCount) disks"

foreach ($disk in $disks) {

    # One liner from https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b#find-vulnerable-software-windows
    # gci 'C:\' -rec -force -include *.jar -ea 0 | foreach {select-string "JndiLookup.class" $_} | select -exp Path

    # Server 2008 (R2)
    if ($windowsVersion -like "*2008*") {

        Write-Host -ForegroundColor Yellow "  $(Get-Date -Format "yyyy-MM-dd H:mm:ss") - Checking $($disk.Name): - $($disk.VolumeLabel)"
        Get-ChildItem "$($disk.Name)" -Recurse -Force -Include @("*.jar","*.war") -ErrorAction SilentlyContinue | ForEach-Object { Select-String "JndiLookup.class" $_ } | Select-Object -ExpandProperty Path | Get-Unique

    }
    # Everything else
    else {

        Write-Host -ForegroundColor Yellow "  $(Get-Date -Format "yyyy-MM-dd H:mm:ss") - Checking $($disk.DriveLetter): - $($disk.VolumeLabel)"
        Get-ChildItem "$($disk.DriveLetter):\" -Recurse -Force -Include @("*.jar","*.war") -ErrorAction SilentlyContinue | ForEach-Object { Select-String "JndiLookup.class" $_ } | Select-Object -ExpandProperty Path | Get-Unique

    }

}

Write-Host -ForegroundColor Green "$(Get-Date -Format "yyyy-MM-dd H:mm:ss") - Done checking all drives"

Sample output with nothing found:

check_CVE-2021-44228.ps1 sample output

Sample output with something found:

check_CVE-2021-44228.ps1 sample output 2

Good luck everyone.

How to perform an offline audit of your Active Directory NTLM hashes

It’s read-only Friday so I decided to perform a offline audit of our Active Directory passwords.

I found this great tool: https://gitlab.com/chelmzy/five-minute-password-audit which in turn is a fork of this tool: https://github.com/DGG-IT/Match-ADHashes

What I’m going to write here is mostly a repeat of these two Gitrepos with a few tweaks and corrections.

To perform this procedure you will need to be able to login to a Domain Controller. You’re also going to want a secure location to perform all of this work so the dumped list of usernames and hashes doesn’t escape your control.

The secure location should be a workstation or server running the same or a newer version of Windows than your Domain Controller. For example if you’re running AD 2012R2 you can’t complete this on a 2008R2 box. You’re secure workstation or server will need to be running PowerShell 5.0 or newer.

Step 1 – Export NTDS.dit and the SYSTEM hive

  1. Login to a domain controller
  2. Open a Command Prompt window
  3. Type “ntdsutil”
  4. Click ‘Yes’ if the UAC prompts you
  5. Run the following commands:
    activate instance ntds
    ifm
    
    # Replace <DOMAINNAME> with your domains name
    create full c:\temp\<DOMAINNAME>-audit
    
    # Wait for command to complete
    quit
    quit
  6. Transfer “C:\Temp\<DOMAINNAME>-audit” to the secure location you’ll work on it. I do not recommend performing the rest of these steps on your Domain Controllers

Step 2 – Download the latest Have I Been Pwned Offline NTLM password list

  1. Go to https://haveibeenpwned.com/Passwords
  2. Scroll to the bottom and download the “ordered by prevalence” NTLM link
  3. Once downloaded, transfer the password list to your secure location in the audit directory and extract it

Step 3 – Covert the hashes in the NTDS.dit file to Hashcat formatting

  1. On your secure workstation/server launch PowerShell as an administrator (right click, run as administrator on the PowerShell shortcut)
  2. Install the DSInternals tools by running
    Install-Module -Name DSInternals -Force
  3. Go into the audit directory
    cd c:\temp\<DOMAINNAME>-audit
  4. Convert the hashes
    $key = Get-BootKey -SystemHivePath .\registry\SYSTEM
    
    # Change <DOMAINNAME> to your domains name
    Get-ADDBAccount -All -DBPath '.\Active Directory\ntds.dit' -BootKey $key | Format-Custom -View HashcatNT | Out-File <DOMAINNAME>-hashes.txt -Encoding ASCII

Step 4 – Compare your hashes to HIBP

The code in the Git Repos I linked at the beginning of the article are written as functions. For myself I just wanted a script I could execute with the appropriate parameters instead of futzing around with importing the function.

I also tweaked the original script for formatting (I like a bit more white space personally), added CSV headers, removed the spaces between commas, had the script append it’s execution time to the end of the CSV file and allowed for relative filenames as parameters instead of requiring absolute paths.

Here is my version of the script:

<#
This is a slightly altered version of https://gitlab.com/chelmzy/five-minute-password-audit/blob/master/Match-ADHashes.ps1 which is a slightly alter version of https://github.com/DGG-IT/Match-ADHashes/ for no nonsense output. All credit to them.
.NAME
    Match-ADHashes
.SYNOPSIS
    Matches AD NTLM Hashes against other list of hashes
.DESCRIPTION
    Builds a hashmap of AD NTLM hashes/usernames and iterates through a second list of hashes checking for the existence of each entry in the AD NTLM hashmap
        -Outputs results as object including username, hash, and frequency in database
        -Frequency is included in output to provide additional context on the password. A high frequency (> 5) may indicate password is commonly used and not necessarily linked to specific user's password re-use.
.PARAMETER ADNTHashes
    File Path to 'Hashcat' formatted .txt file (username:hash)
.PARAMETER HashDictionary
    File Path to 'Troy Hunt Pwned Passwords' formatted .txt file (HASH:frequencycount)
.PARAMETER Verbose
    Provide run-time of function in Verbose output
.EXAMPLE
    $results = Match-ADHashes -ADNTHashes C:\temp\adnthashes.txt -HashDictionary -C:\temp\Hashlist.txt 
.OUTPUTS
    Array of HashTables with properties "User", "Frequency", "Hash"
    User                            Frequency Hash                            
    ----                            --------- ----                            
    {TestUser2, TestUser3}             20129     H1H1H1H1H1H1H1H1H1H1H1H1H1H1H1H1
    {TestUser1}                     1         H2H2H2H2H2H2H2H2H2H2H2H2H2H2H2H2
.NOTES
    If you are seeing results for User truncated as {user1, user2, user3...} consider modifying the Preference variable $FormatEnumerationLimit (set to -1 for unlimited)
    
    =INSPIRATION / SOURCES / RELATED WORK
        -DSInternal Project https://www.dsinternals.com
        -Checkpot Project https://github.com/ryhanson/checkpot/
    =FUTURE WORK
        -Performance Testing, optimization
        -Other Languages (golang?)
.LINK
    https://github.com/DGG-IT/Match-ADHashes/
#>

param(
    [Parameter(Mandatory = $true)]
    [System.IO.FileInfo] $ADNTHashes,
    [Parameter(Mandatory = $true)]
    [System.IO.FileInfo] $HashDictionary
)

process {
    $stopwatch = [System.Diagnostics.Stopwatch]::StartNew()

    # Set the current location so .NET will be nice and accept relative paths
    [Environment]::CurrentDirectory = Get-Location

    # Declare and fill new hashtable with ADNThashes. Converts to upper case to 
    $htADNTHashes = @{}
    Import-Csv -Delimiter ":" -Path $ADNTHashes -Header "User","Hash" | % {$htADNTHashes[$_.Hash.toUpper()] += @($_.User)}
       
    # Create Filestream reader
    $fsHashDictionary = New-Object IO.Filestream $HashDictionary,'Open','Read','Read'
    $frHashDictionary = New-Object System.IO.StreamReader($fsHashDictionary)

    # Output CSV headers
    Write-Output "Username,Frequency,Hash"

    #Iterate through HashDictionary checking each hash against ADNTHashes
    while ($null -ne ($lineHashDictionary = $frHashDictionary.ReadLine())) {
        if($htADNTHashes.ContainsKey($lineHashDictionary.Split(":")[0].ToUpper())) {
                $user = $htADNTHashes[$lineHashDictionary.Split(":")[0].ToUpper()]
                $frequency = $lineHashDictionary.Split(":")[1]
                $hash = $linehashDictionary.Split(":")[0].ToUpper()
                Write-Output "$user,$frequency,$hash"
            }
        }

    $stopwatch.Stop()

    Write-Output "Function Match-ADHashes completed in $($stopwatch.Elapsed.TotalSeconds) Seconds"
}
    
end {
}

 

To execute it, copy/paste it into notepad and save it as ‘myAudit.ps1’ or what ever file name you’d like.

Now perform your audit:

# Replace <DOMAINNAME> with your domain name
.\myAudit.ps1 -ADNTHashes <DOMAINNAME>-hashes.txt -HashDictionary <HIBP TEXT FILE> | Out-File <DOMAINNAME>-PasswordAudit.csv

# Example
.\myAudit.ps1 -ADNTHashes myDomain-hashes.txt -HashDictionary pwned-passwords-ntlm-ordered-by-count-v5.txt | Out-File myDomain-PasswordAudit.csv

The final result will be a CSV file you can dig through.

Step 6 – Clean it all up

The output may or may not surprise you but what ever the outcome, when you’re done you want to get rid of the <DOMAINNAME>-hashes.txt and the NTDIR.dis file as soon as possible. If someone snags a copy of that you’ll likely get in some serious trouble.

Head on over to SysInternals and grab SDelete

.\sdelete.exe -p 7 -r -s <DIRECTORY OR FILE>

 

Event ID 20292 from DHCP-Server

Checking over our DHCP server we were seeing quite a few of these errors appearing in the ‘Microsoft-Windows-DHCP Server Events/Admin’ event log:

Log Name:      DhcpAdminEvents
Source:        Microsoft-Windows-DHCP-Server
Date:          1/28/2019 10:23:49 PM
Event ID:      20292
Task Category: DHCP Failover
Level:         Error
Keywords:      
User:          NETWORK SERVICE
Computer:      dc2.mydomain.com
Description:
A BINDING-ACK message with transaction id: 943568 was received for IP address: 10.253.166.162 with reject reason:  (Reject Reason Unknown ) from partner server: dc1.mydomain.com for failover relationship: dc1.mydomain.com-dc2.mydomain.com.

Researching this error I came across this forum post: https://social.technet.microsoft.com/Forums/en-US/15d00412-3dfc-4520-a74e-1f32fe1329ef/windows-server-2012-dhcp-event-id-20291?forum=winserveripamdhcpdns

Which lead me to this KB article: https://support.microsoft.com/en-ca/help/2955135/event-id-20291-is-logged-in-the-system-log-when-a-client-computer-is-m

The hotfix that Microsoft mentions is from November 2014 and has been installed on our server for a very long time. We never noticed this error back in 2014 when the hotfix was installed so we were not able to “first remove the failover relationship, install the update to both DHCP nodes and restart them, and then reestablish the failover relationship” per Microsoft’s article.

The article leads me to believe you have to deconfigure failover on all subnets, destroy the failover relationship, re-create the failover relationship and then re-configure failover on each subnet.

Turns out you can just right click ‘Deconfigure failover’ and then right click ‘Configure failover’ on the specific subnets having the issue and re-use the existing failover relationship to resolve this issue assuming you’ve installed the November 2014 hotfix.

Microsoft RAS VPN and VXLAN not quite working

I’m not overly knowledgeable about advanced networking but I figured I’d share this since I couldn’t find anything online about it at the time.

We run a Microsoft Remote Access Server (RAS) for our VPN server. We provide L2TP primarily for users.

Due to a limitation in the Windows VPN client our RAS server has two network interfaces, one directly on the internet with a public IP (VLAN1) and one internally with a private IP (VLAN2).

The private IP relays VPN users DHCP/DNS requests to our internal DHCP and DNS servers.

RAS handles the authentication instead of RADIUS and we have our internal routes published via RIP to the RAS server so they can be provided to VPN clients when they connect.

I believe this is a fairly common design.

On the network end, our original design involved spanning VLAN1 and 2 all the way from our edge into our data center so the VM could pretty much sit directly on them. This worked fine.

As part of a major network redesign we performed we changed the VLAN spanning design over to using a VXLAN from our edge into our data center.

After making this change we ran into the strangest VPN issues. Users could connect and ping anything they wanted, do DNS lookups and browse most HTTP websites. HTTPS websites would partially load or fail to load and network share (SMB) access would partially work (you could get to the DFS root but not down to an actual file server).

After many hours of troubleshooting we determined our problem.

The MTU of most devices is configured to default to 1500 bytes. When we started tunneling the traffic through a VXLAN the tunneling added 52 bytes to the packet size making the total packet size 1552 bytes which is just over what most network cards are expecting. This caused large packets to drop (loading a HTTPS website, connecting to a share) but small packets (pings, some HTTP websites) to work fine.

I believe the final solution from our network team was to enable Jumbo packets from end to end of the VXLAN tunnel so it could transmit slightly larger than normal packets.

If you have any specific questions I can relay them to our Network Team and try to get you an answer. No promises :)

DHCP stops serving IPs when audit log is full

We run two DHCP servers in a HA configuration. The HA is configured to split the scopes in half. Depending on how high up the scope your IP is will determine which DHCP server you get your IP from. We have DHCP audit logging enabled.

DHCP1 handles 0-127 and DHCP2 handles 128-254 (we mostly use /24’s right now).

We started getting reports of random devices on the network not being able to connect or login to the domain. By the time a technician got to the PC to check it the issue was resolved magically.

We dug into the DHCP servers and found the DHCP audit log on DHCP1 was full (36MB in size). The log on DHCP2 was not full (yet, only 34MB in size).

Stopping DHCP on DHCP1, renaming the audit log and then starting DHCP on DHCP1 again appeared to resolve the issue.

The thing that had us scratching our heads is we’ve had this problem before and we had re-configured DHCP on these servers to allow the log files to grow to 250MB but things had stopped at 36MB.

We used this PowerShell to make the change a long while ago and restarted the DHCP service: https://docs.microsoft.com/en-us/powershell/module/dhcpserver/set-dhcpserverauditlog?view=win10-ps

Set-DhcpServerAuditLog -MaxMBFileSize 250

Per the above link it states “-MaxMBFileSize Specifies the maximum size of the audit log, in megabytes (MB).”

It turns out this PowerShell command simply changes the registry value for HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DHCPServer\Parameters\DhcpLogFilesMaxSize which you can just do manually if you’d prefer.

I have no idea how I found it but after some digging I found this article for Server 2008 (we’re using 2012R2): https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc726869(v=ws.10)

It states:

 

Dynamic Host Configuration Protocol (DHCP) servers include several logging features and server parameters that provide enhanced auditing capabilities. You can specify the following features:

  • The file path in which the DHCP server stores audit log files. DHCP audit logs are located by default at %windir%\System32\Dhcp.
  • A maximum size restriction (in megabytes) for the total amount of disk space available for all audit log files created and stored by the DHCP service.
  • An interval for disk checking that is used to determine how many times the DHCP server writes audit log events to the log file before checking for available disk space on the server.
  • A minimum size requirement (in megabytes) for server disk space that is used during disk checking to determine if sufficient space exists for the server to continue audit logging.

 

I’ve bolded and italicized the relevant line. The article also specifically references the registry key the PowerShell command changes.

This leads me to believe the PowerShell documentation is incorrect and “-MaxMBFileSize” specifies the maximum size of all audit logs added together. Not a maximum size per individual audit log.

I checked the directory size of “%windir%\system32\dhcp” on both servers and they were very close to 250MB.

We’ve since made the following change:

Set-DhcpServerAuditLog -MaxMBFileSize 4096

I will update this article if this does not resolve the issue for us.

 

Update 2019-01-10: I can confirm this resolved the issue for us. The log file for the following day reached 54MB with no issue.