Migrating DHCP from 2003 to 2012 R2

This post will likely fall on deaf ears since no one out there is still running Windows Server 2003 right?

Well we are for our oldest DHCP server. Better yet it’s 2003 Standard (non R2) which means I can’t installed Active Directory Management Gateway Service which would allow me to remotely access DHCP on the 2003 server via PowerShell 5.1 running on a different box. Newer versions of PowerShell include support for working with DHCP. PowerShell 2.0 (latest available for Server 2003) does not support these commands.

Googling around found me the standard recommended way of migrating DHCP subnets.

On the source machine run:

netsh dhcp server export 192.168.0.0.txt 192.168.0.0

This will export all of the current leases and reservations in the 192.168.0.0 scope into a text file, you can then transfer the text file over to your new DHCP server and run the following to import it:

netsh dhcp server import 192.168.0.0.txt 192.168.0.0

The downside to this method is that it causes a temporary outage of your DHCP server during the import/export. I just migrated 80 odd subnets during the day and the outages were so short no one noticed.

Alright so that was the easy part.

In addition to 80 standard DHCP scopes we have 4 superscopes that also need to be migrated.

Attempting to migrate the superscopes using the above method failed. When I attempted to import the scope into the destination DHCP server I got the error “TLS supported but not configured”.

The first post I found for this error in Google links to a Technet discussion where someone states “No, you cannot direct migrate windows server 2003 DHCP/DNS to windows server 2012 DHCP/DNS.”. Clearly not an accurate statement since I’d just migrated 80 standard scopes.

So here I am. I cannot move 4 super scopes using the netsh method, I cannot use PowerShell because the 2003 server is to old and I do not want to upgrade it to 2003 R2 for multiple reasons (did I mention it’s a Domain Controller to?).

What I ended up doing was building a new Windows 2012 R2 Standard box, joined it to our domain so I could remotely access it and have the benefit of domain logins for accessing resources across our network and then installed DHCP on it. Immediately after the DHCP Server installation completed I went into the Windows Firewall and blocked DHCP just in case.

Then on the old DHCP server I ran this command:

netsh dhcp server export all.txt all

I then transferred the “`all.txt“` file over to the DHCP server I just built and ran this:

netsh dhcp server import all.txt all

and ended up with a complete copy of my old DHCP server on my temporary DHCP server including my superscopes, no errors.

Now I can use PowerShell to finish this up. On the temporary DHCP server I ran this:

Export-DhcpServer -ComputerName "localhost" -File "C:\temp\SUPERSCOPE1.xml" -ScopeId 192.168.48.0,192.168.49.0 -Leases

transferred “`SUPERSCOPE1.xml“` to the new DHCP server and ran:

Import-DhcpServer -ComputerName "localhost" -File "C:\temp\SUPERSCOPE1.xml" -BackupPath "C:\temp\" -ScopeId 192.168.48.0,192.168.49.0 -Leases

And bingo. Superscope successfully migrated from 2003 to 2012R2 with a small middle step.

If you screw up or need to do this in batches over time you can quickly and easily wipe everything out on the temporary DHCP server by doing the following:

  1. Stop the DHCP service
  2. Delete the contents of C:\Windows\system32\dhcp
  3. Start DHCP servoce

You’ll end up with a blank DHCP server that you can re-import a fresh copy of your old DHCP server into.

Previous version of the Active Directory Replication Status Tool

Who liked using the Active Directory Replication Status Tool? I did.

Who thought it was a great, simple, straight forward tool that was far easier than interpreting the output of some command line tools and didn’t feel it needed to become a cloud service with a less intitive interface? I do.

Digging through a few of my servers I found the old installer for the Active Directory Replication Status Tool. The version you can install on your own servers and doesn’t appear to give an error about the installer being expired.

Download it here: https://dl.dropboxusercontent.com/u/22772464/adreplstatusInstaller.zip

I believe this is version 1.1 even though the splash page when launching it says it’s 1.0. It was the latest version you could download before Microsoft expired it here and told everyone to start using the cloud based SCOM solution they offer.

Please share it/mirror it for all to enjoy.

 

Update – 2016-04-05

Looks like Microsoft is trying to kill this tool. When you try the use the version I’ve linked above now it says it has also expired.

I’ve figured out a workaround. Download this handy tool and then do this:

runasdate settings

If anyone knows assembler and wants to try dissecting repl.exe to remove the date check I’ll happily link/host to their modified repl.exe.

I’ve seen a few comments on other sites about the saftey of getting this tool from a non-Microsoft source and this is the best I can provide to prove I have not altered the MSI file. If you check the hashes against the two download links below you’ll see they match files uploaded well before this blog post.

MSI File
MD5: d63ceaa4131f8dc64800d33ac3b242c7
SHA1: 1a117510e42d284199743c53722dd51690a93d59

http://www.herdprotect.com/adreplstatusinstaller.msi-1a117510e42d284199743c53722dd51690a93d59.aspx
https://malwr.com/analysis/Y2JlNDQ0YmM0MjM3NGY4MWJlOGJhOTkyMDNkZGQxZGI/

You can try the workaround I’ve mentioned above on the official download as well: https://www.microsoft.com/en-ca/download/details.aspx?id=30005

 

Update 2016-04-20

Good news everyone! Microsoft has brought back the stand-alone tool thanks to everyone who provided them feedback demanding it.

See: https://feedback.azure.com/forums/267889-operational-insights/suggestions/12293631-bring-back-the-on-prem-ad-replication-status-tool

The new, non-expiring download, can be found here: https://www.microsoft.com/en-us/download/details.aspx?id=30005

 

Update 2017-07-12

Looks like the tool is expiring again.

I was able to re-download, re-install from here: https://www.microsoft.com/en-us/download/details.aspx?id=30005

The tool worked again but now gives me a 24 day count down.

 

Update 2020-01-13

Microsoft might have finally killed this app. The latest version says the license is expired and the RunAsDate trick isn’t working for me anymore, at least not on Server 2019.

Looks like Microsoft posted a new version of the tool here: https://www.microsoft.com/en-us/download/details.aspx?id=30005 sadly it’s license is still expired.

KB3002657 breaks everything!

Update: 2015-03-20 09:24 PST – Thank you Didi for informing us that Microsoft has released two updated hotfixes KB3002657-v2 and KB3033395-v2 that shouldn’t cause this problem. I have not had a chance to try these patches yet.

Update: 2015-03-12 09:45 PST – Now that I’ve had a chance to sleep I’ve updated this post to include some of the specific errors we saw and gather some additional information from around the web. Hopefully it will help others diagnose this problem.

 

We installed our Windows Updates this week and this little gem, KB3002657, came with them.

After this patch was installed on our Windows 2003 Domain Controllers and we rebooted them and all of the other servers in our organization we randomly started having authentication issues with certain services.

Outlook Anywhere and Mac’s running Outlook could no longer authenticate to Exchange 2010 but everyone could login to webmail. To add to this once I had patched and rebooted our Exchange 2010 servers Windows Outlook clients could no longer authenticate and no e-mail was being delivered it was just backing up in the queues. I dug into the Exchange Event Viewer and looked under ‘Security’ and found thousands of these errors:

An account failed to log on.

Subject:
	Security ID:		NULL SID
	Account Name:		-
	Account Domain:		-
	Logon ID:		0x0

Logon Type:			3

Account For Which Logon Failed:
	Security ID:		NULL SID
	Account Name:		<EXCHANGE CAS SERVER HOSTNAME>$
	Account Domain:		<OUR DOMAIN>

Failure Information:
	Failure Reason:		An Error occured during Logon.
	Status:			0xc000006d
	Sub Status:		0x0

Process Information:
	Caller Process ID:	0x0
	Caller Process Name:	-

Network Information:
	Workstation Name:	<EXCHANGE CAS SERVER HOSTNAME>
	Source Network Address:	<EXCHANGE CAS SERVER IP>
	Source Port:		54244

Detailed Authentication Information:
	Logon Process:		NtLmSsp 
	Authentication Package:	NTLM
	Transited Services:	-
	Package Name (NTLM only):	-
	Key Length:		0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
	- Transited services indicate which intermediate services have participated in this logon request.
	- Package name indicates which sub-protocol was used among the NTLM protocols.
	- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

A departments NAS would no longer authenticate users. The couldn’t access any of their files. I didn’t have access to this device so I couldn’t review and logs.

A few of our websites that use ‘Integrated Authentication’ (Kerberos) wouldn’t authenticate users but if we switched the site to ‘Basic Authentication’ it would work fine. When ‘Integrated Authentication’ was enabled the IIS logs would show error “401.1” for every failed login.

Another department couldn’t login to a MSSQL 2005 Database using their AD accounts but applications that used MSSQL accounts worked fine. On our SQL server we saw the following errors:

Date		11/03/2015 7:55:26 AM
Log		SQL Server (Current - 12/03/2015 9:53:00 AM)

Source		Logon

Message
Login failed for user ''. The user is not associated with a trusted SQL Server connection. [CLIENT: XXX.XXX.XXX.XXX]


Date		11/03/2015 7:55:26 AM
Log		SQL Server (Current - 12/03/2015 9:53:00 AM)

Source		Logon

Message
Error: 18452, Severity: 14, State: 1.




Date		11/03/2015 7:55:26 AM
Log		SQL Server (Current - 12/03/2015 9:53:00 AM)

Source		Logon

Message
SSPI handshake failed with error code 0x8009030c while establishing a connection with integrated security; the connection has been closed. [CLIENT: XXX.XXX.XXX.XXX]

Date		11/03/2015 7:55:26 AM
Log		SQL Server (Current - 12/03/2015 9:53:00 AM)

Source		Logon

Message
Error: 17806, Severity: 20, State: 2.

These errors lead us to believe there was a trust issue going on but on. We checked our DCs and they were all happy and replicating BUT there was a red herring. We logged into one of our DCs and ran “nltest.exe /server:<DC HOSTNAME> /sc_verify:<DOMAIN FQDN>”.

All of our DCs returned the expected results:

C:\>nltest.exe /server:<DC HOSTNAME> /sc_verify:<DOMAIN FQDN>

Flags: b0 HAS_IP  HAS_TIMESERV
Trusted DC Name \\<FQDN OF A DC>
Trusted DC Connection Status Status = 0 0x0 NERR_Success
Trust Verification Status = 0 0x0 NERR_Success
The command completed successfully

Except one. That one DC returned this:

C:\>nltest.exe /server:<DC HOSTNAME> /sc_verify:<DOMAIN FQDN>

I_NetLogonControl failed: Status = 1355 0x54b ERROR_NO_SUCH_DOMAIN

That lead us down the wrong path of thinking one of our DCs had some how fallen off the domain even though users were authenticating against it successfully and replication was working properly.

After spending an few hours troubleshooting what we thought was a DC problem it all came down to this bloody hotfix.

Removing KB3002657 (and per Microsofts suggestion KB3046049) from our Domain Controllers and another reboot the problem was solved.

After some reflection I don’t think removing KB3046049 was necessary. I’d recommend starting with KB3002657 first.

As of this morning our users are reporting that everything is back to normal. Didn’t have to reboot my Exchange, SQL or Webservers. Once the DCs had the patch removed and were rebooted everything just magically started working again.

Finally some credit. Thank you to Eds for his post here and to those who responded: http://serverfault.com/questions/674541/has-march-2015-patch-tuesday-broken-2003-shares they helped me narrow down and solve the problem for our organization before Microsoft called me back and confirmed this was the solution.

Also give this a read: http://www.infoworld.com/article/2895900/security/microsoft-netlogon-patch-kb-3002657-woes-continue-kb-3032359-cisco-anyconnect-fix-confirmed.html

They suggest re-configuring local policies so that Windows LAN Manager Authentication is set to “Send LM & NTLM responses”. According to gpedit.msc on our Domain Controllers the default setting should be “Send LM & NTLM responses”. For some odd reason on ours we’ve manually changed it to “Send NTLM responses only”. I can’t comment if this would work or not for all of the above cases I came across or if you need to push this setting change out to all of your clients and/or servers.

Workaround for 0x80248015 when trying to do Windows Updates on Server 2003

If you’re like us you probably noticed this patch Tuesday that a bunch, if not all, of your Windows 2003 servers are not able to load the Microsoft Update page. When trying to load the page the error 0x80248015 is displayed.

After some Googling I’ve pieced together a workaround that will get you some updates this patch Tuesday. It seems switching back to Windows Update does the trick.

To switch back to Windows Update you just need to do the following:

  1. Login to the server
  2. Run ‘services.msc’
  3. Stop the ‘Automatic Update’ service
  4. Delete C:\Windows\SoftwareDistrabution
  5. Start the ‘Automatic Update’ service
  6. Launch ‘Internet Explorer’ and click ‘Tools’ and ‘Windows Update’
  7. Use Windows Updates. DO NOT upgrade to Microsoft Update

OR if you’re like me and don’t want to have to do that for 50+ servers you can do it from a batch file:

net stop wuauserv
rd /s /q %SystemRoot%\SoftwareDistribution
net start wuauserv
%SystemRoot%\system32\wupdmgr.exe

 

Proxy Settings pushed by GPO not applying to Windows 7 and Internet Explorer 9

I support a Windows 2003 Active Directory Domain, until today only Windows XP clients and Internet Explorer 8.

We use a proxy server to filter and monitor the internet and we push the configuration for that proxy server via Group Policy. Our current GPO works perfectly with Windows XP and Internet Explorer 8.

Today we got our first Windows 7 client and I noticed the proxy settings from our Windows 2003 Active Directory Domain were not applying top the Windows 7 machine running Internet Explorer 9.

It turned out there were two reasons the settings were not applying. The first is a no brainer. I hadn’t updated my Internet Explorer Group Policy Administrative Template to support Internet Explorer 9. That was easy enough to fix by going here and updating my Administrative Template.

After updating the Administrative Template I figured I’d be done. Turns out I was wrong. I ran ‘rsop.msc’ and ‘gpresult /r’ and both showed the proxy settings were being applied but Internet Explorer wasn’t using our proxy server.

Turns out there was a second problem and it was FAR more difficult to figure out. I did some research and found others complaining about this problem. Removing proxy settings for ‘GOPHER’ from their Internet Explorer GPO resolved the problem for the majority of these users. To remove GOPHER settings open up your group policy and make the following tweak:

User Configuration
  Windows Settings
    Internet Explorer Maintenance
      Connections
        Proxy Settings
          Uncheck 'Use the same proxy server for all addresses'
          Delete any settings for Gopher and SOCKS

This didn’t solve the problem for me but it may have had a hand in it.

The second thing I had to do in my case was to disable caching of auto-proxy scripts and disable changing automatic configuration settings. My understanding from the post I read was that the cache can become corrupt and cause a bunch of problems. To make this change open up your Internet Explorer GPO again and make the following changes:

User Configuration
  Administrative Templates
    Windows Components
      Internet Explorer
        Disable caching of Auto-Proxy scripts: ENABLED
        Disable changing Automatic Configuration settings: ENABLED

Once I’d made both of those tweaks I ran a quick ‘gpupdate /force’ and that was it! Things started working properly.

There seems to be a bug in Internet Explorer 9, or at least the version I’m running, where if you go into the proxy settings under ‘Internet Options’ you won’t actually see your proxy servers name in the ‘Address:’ box. I’ll see if that clears up once I get Windows Updates working on these Windows 7 machines or when we get to Internet Explorer 10.

References