Error 1603 when upgrading vCenter 6.0u1 to 6.0u2

Recently ran across this one:

 

The upgrade would get to the VCSServiceManager step, fail and back out. It then left our existing vCenter 6.0u1 installation unable to start.

I did all the standard things you’ll find on VMwares Support site (and recommended by the support rep I got a hold of):

2119768 Error code 1603 when upgrading to vCenter Server 6.0
2127519 Installing the VMware vCenter Server 6.0 fails with the vminst.log error: MSI result of install of “C:\vCenter-Server\Packages\vcsservicemanager.msi” may have failed : 1603
2137365 Upgrade of vCenter from 5.x to 6.0 fails with “Installation of component VCSServiceManager failed with error code ‘1603’. Check the logs for more details.”
2113068 Upgrading or installing VMware vCenter Server 6.0 fails with the vminst.log error: Error in accessing registry entry for DSN
2119169 Installing VMware vCenter Server 6.0 using a Microsoft SQL database fails with the error: An error occurred while starting service ‘invsvc’

None helped.

While waiting for my VMware Support rep to dig through the log files, on a hunch, I made the following changes:

  1. Checkmarked ‘IPv6’ in the network stack for the servers network card
  2. Re-ran the vCenter installer separately by right clicking it and ‘Running As Administrator’ (\VMware-VIMSetup-all-6.0.0-3634788\vCenter-Server\VMware-vCenter-Server.exe)

The installation then succeeded and we have a functioning vCenter again.

Two fun facts:

  1. The UAC is disabled on our server
  2. IPv6 was (and still is) disabled via the registry using these utilities even though I’ve now re-checked IPv6 in the network cards network stack

Our server is in a fairly unique configuration I suspect but hopefully this will help someone else.

Exchange permission issues for a single user on a generic mailbox

We created a new generic mailbox in Exchange 2010. Created a group (Global) and added the users to it. We then created a Universal group, added the Global group to it and then added the Universal group to the mailbox with full mailbox permissions.

After the obligatory wait 24 hours for Exchange to update it’s permissions 4 of the 5 users in the Global group had access to the generic mailbox. The 5th user could not access the mailbox via their Outlook or Webmail. Permission denied errors.

We tried removing/re-adding them from the group with no success. We tried migrating their mailbox back and forth with no success.

Finally after some digging we thing we figured out the problem.

The user in question had originally been migrated from Exchange 2003 to 2010. Then at some point while on Exchange 2010 we had to purge/re-create their Mailbox.

The process of purging re-creating their mailbox changed their LegacyDN to the new, correct, format for Exchange 2010 and dumped their old Exchange 2003 LegacyDN.

To view a users legacy DN run the following Powershell command:

What we ended up doing is re-creating their LegacyDN from Exchange 2003 as a new X500 record and then they were instantly able to access the generic mailbox.

It could always be coincidence…..

References:

Networking randomly dies on a 2012 R2 vSphere VM

Strange issue. Simple solution.

We had a Windows Server 2012 R2 Domain Controller sitting on vSphere 5.5 (2068190) which would randomly lose it’s network connection.

When you logged into the system locally the network interface appeared to be up but you could not connect to anything outside of the VM.

If I rebooted the VM it would work for a few hours or less and then the network would drop out again.

Digging through the event viewer I came across these:

The VM had a E1000 NIC attached to it. I figured the issue was the VM NIC model and got some backup to my theory from here: https://community.spiceworks.com/topic/504405-windows-server-2012-r2-guest-os-on-vmware-keeps-losing-gateway-connection

The solution appears to have been removing the E1000 NIC and adding either a E1000E NIC or in my case a VMXNET 3 NIC.

Previous version of the Active Directory Replication Status Tool

Who liked using the Active Directory Replication Status Tool? I did.

Who thought it was a great, simple, straight forward tool that was far easier than interpreting the output of some command line tools and didn’t feel it needed to become a cloud service with a less intitive interface? I do.

Digging through a few of my servers I found the old installer for the Active Directory Replication Status Tool. The version you can install on your own servers and doesn’t appear to give an error about the installer being expired.

Download it here: https://dl.dropboxusercontent.com/u/22772464/adreplstatusInstaller.zip

I believe this is version 1.1 even though the splash page when launching it says it’s 1.0. It was the latest version you could download before Microsoft expired it here and told everyone to start using the cloud based SCOM solution they offer.

Please share it/mirror it for all to enjoy.

 

Update – 2016-04-05

Looks like Microsoft is trying to kill this tool. When you try the use the version I’ve linked above now it says it has also expired.

I’ve figured out a workaround. Download this handy tool and then do this:

runasdate settings

If anyone knows assembler and wants to try dissecting repl.exe to remove the date check I’ll happily link/host to their modified repl.exe.

I’ve seen a few comments on other sites about the saftey of getting this tool from a non-Microsoft source and this is the best I can provide to prove I have not altered the MSI file. If you check the hashes against the two download links below you’ll see they match files uploaded well before this blog post.

MSI File
MD5: d63ceaa4131f8dc64800d33ac3b242c7
SHA1: 1a117510e42d284199743c53722dd51690a93d59

http://www.herdprotect.com/adreplstatusinstaller.msi-1a117510e42d284199743c53722dd51690a93d59.aspx
https://malwr.com/analysis/Y2JlNDQ0YmM0MjM3NGY4MWJlOGJhOTkyMDNkZGQxZGI/

You can try the workaround I’ve mentioned above on the official download as well: https://www.microsoft.com/en-ca/download/details.aspx?id=30005

 

Update 2016-04-20

Good news everyone! Microsoft has brought back the stand-alone tool thanks to everyone who provided them feedback demanding it.

See: https://feedback.azure.com/forums/267889-operational-insights/suggestions/12293631-bring-back-the-on-prem-ad-replication-status-tool

The new, non-expiring download, can be found here: https://www.microsoft.com/en-us/download/details.aspx?id=30005

Powershell script to monitor mount points in Windows and e-mail an alert

Simple script inspired by this blog post.

This script will read a servers.txt file located in the same directory as the script, check each server listed (one per line, use the servers FQDN) for mount points, see how much free space is available, compare it to a minimum amount you set and then e-mail an alert if below the minimum amount.

Should work on Windows Server 2008 and newer. Might work on Server 2003 but I haven’t tested it.