Error 1603 while upgrading vCenter 5.5 to 6.0u1

Upgrading vCenter recently from vCenter 5.5 to 60.u1 and the upgrade would consistently fail displaying two error messages.

First it would display “An error occurred while invoking external command: ‘Database instance not defined for mssql provider'”

Then the installation would appear to proceed until it got to installing the VCSServiceManager. Then I would get Error 1603 saying that it couldn’t talk to the database server.

We run our vCenter Server database off a separate MSSQL 2012 Standard server.

I found plenty of resources on VMware’s site for this error:

None of these solved the issue for us.

Some how I ended up on this article: Installing or Upgrading to vCenter Server 6.0 fails with the error: Unable to get port number for mssql provider (2125492)

That isn’t the error we were getting but the solution ended up fixing the problem for us.

This issue is caused due to the use of certain ASCII characters in the Microsoft SQL Server user's password used for the DSN on the vCenter Server. 

To resolve this issue, ensure your Microsoft SQL Server user password used in the DSN does not contain the following:

    - (dash) 
    ? (question mark) 
    _ (underscore)
    ( (left parentheses)
    = (equal sign)
    ! (exclamation mark)
    , (comma)

Once the password has been updated removing any of the above characters:

    If you are performing a fresh installation, attempt the fresh install again.
    If you are performing an upgrade, roll back your vSphere environment to the pre-upgraded state, upgrade the vCenter Server database password stored and re-run the upgrade. For more information on updating the vCenter Server database password, see Changing the vCenter Server database user ID and password (1006482).

We had been using most of those special characters in the password for the vCenter user accessing our MSSQL Server.

I changed the password to something that didn’t have those special characters on the MSSQL server and then did the following to update the password in vCenter:

  1. Change the MSSQL users password
  2. Updating the database user password stored in vCenter Server (2000493)
  3. Updated the password in the ODBC connector
  4. Restarted the VMware vCenter Server service

The upgrade was successful after that.

Note: My upgrades were getting far enough to completely remove the existing installation of vCenter 5.5 but not far enough to alter the database. I had to revert my vCenter VM to my pre-upgrade Snapshot so vCenter 5.5 was back up and running before I could change the password.

NetApp SnapManager for Exchange – Error Code: 0xc004146f

We use SnapManager for Exchange and NetApp Single Mailbox Recovery to backup our Exchange 2010 environment.

We’ve noticed that randomly our backups will fail with a verification error. The backups are still good and the failure usually happens 2-3 times in a row and then doesn’t happen again for a month.

NetApp has a KB article (login required) for this which basically tells you to verify that a set of applicable Exchange DLL and EXE files are all the same version on all of our Exchange servers that SME runs on.

We did this, they are and in our case we actually only involve a single server when it comes to SME so the versions on our other Exchange servers are irrelevant (I think).

Here is a snippet of what our log looks like:

[02:47:01.595]  ErrCheckLogs: failed with ERROR -1811

[02:47:01.595]  Operation terminated with error -1811 ChecUnknown, in 0.0 seconds.

[02:47:01.595]  WARNING: Database/log consistency checking returned with error code 0xFFFFF8ED.

[02:47:01.611]  Error Code: 0xc004146f
Transaction log verification failed.

[02:47:01.611]  Error Code: 0xc004146f
Transaction log verification failed.

[02:47:01.611]  Error Code: 0xc004146f
Transaction log verification failed.

[02:47:01.611]  Updating SnapInfo file at C:\Exchange\Logs\lun_Exchange_Logs_AE\sme_snapinfo\EXCH__MYSERVER\SG__AE\08-31-2015_22.00.12__Daily\SnapInfo__08-31-2015_22.00.12.sme with LogConsistentCheck=Failure

[02:47:02.126]  Verification failed. Error code: 0xc004146f, LUN: C:\Exchange\Databases\lun_Exchange_AE\
[02:47:02.126]  An autosupport message is sent on failure to the storage system of LUN [C:\Exchange\Databases\lun_Exchange_AE\].

[02:47:02.126]  ***LOG VERIFICATION STATUS: AE

[02:47:02.126]  Failed to verify physical integrity of the transaction logs.

[02:47:02.126]  ***LOG CONSISTENT VERIFICATION: Public Folders (MYSERVER)

[02:47:02.126]  Start to verify the logs in SnapInfo directory...

I’ve opened a case with NetApp and it appears they have an internal bug for this issue which you can learn nothing about here:


Update 2015-09-18

Heard back from NetApp and they recommend the following:

  1. Disable circular logging (this is enabled in our case)
  2. There are log files with bad generation numbers and you can fix the issue by running the SnapManager for Exchange backup without verification which will commit and truncate the transaction logs, removing the log files with the bad generation number then you can Re-run the job again with verification enabled.


Update 2015-09-22

After finally digging into the circular logging configuration on our Exchange system it turns out we only had it enabled for 2 of our 5 mail stores. I then went back through all of our failure logs and the only databases that ever generated a verification error were the 2 configured for circular logging.

I’ve disabled circular logging on both of those mail stores and we’ll see if that solves this problem.

vSphere VMs won’t live migrate to new host because of AES-NI

I was building out our vSphere 5.5u2 Cluster today by repurposing one of our hold Hyper-V Nodes. The system in question is a Dell R820 with the latest BIOS (2.3.2) with quad Xeon E5-4607’s.

After building up the node, joining it to the cluster and doing some sanity checks I was ready to re-distribute our VMs.

I selected a batch of VMs and told them to migrate and during the validation process some of them failed and some of them didn’t. The VMs that failed disabled this:

vSphere AES Warning

VMware has a KB article for this: EVC incompatibility issues due to AES/PCLMULQDQ (1034926)

Strange thing is when I checked the BIOS I saw this:

Dell R820 BIOS

After a bit more digging I found the problem.

We have a whack of VMs who’s hardware compatibility is still sitting at version 8.

I took one of our problematic VMs, scheduled an upgrade to hardware compatibility version 10 and rebooted it. When the VM came back up it now passed validation and could live migrate.

References: Schedule a Compatibility Upgrade for Virtual Machines