NetApp SnapManager for Exchange failing with VSS_E_WRITERERROR_RETRYABLE

Running into a problem with Exchange 2010, a FAS2240-4 and SnapManager for Exchange 7.1 where backups would randomly fail every now and then started failing consistently.

Our DFM server would send us an e-mail when the failure occurred that looked like this:

The backup failure would also knock the databases offline and require us to re-sync them the next day.

Digging into the SME logs we found the following:

NetApp provides this page for what they call “Common VSS errors”: https://kb.netapp.com/support/index?page=content&id=1010785&locale=en_US

None of the suggestions there helped us.

In the end I found this forum post for a different product: https://community.emc.com/thread/168678?tstart=0 and applied the registry edits they suggested here: https://community.emc.com/message/705346#705346

and then rebooted our Exchange server running SME.

Since making the change roughly 20 days ago we haven’t had a single failed backup.

VeraCrypt CLI Benchmark Script

I was setting up VeraCrypt on a Raspberry Pi 2 the other day so I could use it as a backup target for my main server and was curious how fast, hahaha just kidding, I mean how slow VeraCrypt would be.

To my disappointment VeraCrypt does not provide a method for running the benchmark built into the GUI via the CLI.

This is the nice benchmark you can run from the GUI:

veracryptBenchmark

So I took some time this weekend and wrote a simple BASH script you can use to benchmark the CLI version of VeraCrypt.

I only tested it with VeraCrupt 1.18a. Chances are if you run it with a previous version you’ll get some really fast times for the new Encryption/Hashes they added in 1.18 because the test won’t actually run.

The benchmark I wrote simply outputs how long it takes to create and encrypt a container of a specific size. It’s not quite as good as the GUI version which outputs the actual speed but it’s at least something. I think this will work on any version of Linux. I tried to use only build-in system utilities and since I wrote it on CentOS 6 means I probably used some of the oldest GNU utilities still commonly used.

During my testing I found that having a container size to small would result in all times being nearly the same with the exception of ripemd160 and streebog. To get better results I recommend using at least a 1GB test file size on modern hardware. Even at 1GB you can see the sample from my main server has each encryption and hash type only varying by 2-45 seconds.

Here is a sample of what the script outputs:

I will admit I don’t fully understand these results. I would have expected much more variety in timing between the different types of encryption under a single hash type. Especially on the Raspberry Pi 2.

The script does a simply container creation benchmark in it’s default state. However if you add  <USERNAME RUNNING BENCHMARK> ALL=NOPASSWD: <PATH TO bin/veracrypt>  to your sudo file and change  FILLCONTAINER=0  to  FILLCONTAINER=1  it will perform file write speed benchmark.

and here is the script itself:

I’m not super confident in the output. Some of the numbers leave me scratching my head. It’s completely possible I’ve got something totally wrong with this script. Please feel free to post comments/revisions.

Error 1603 when upgrading vCenter 6.0u1 to 6.0u2

Recently ran across this one:

 

The upgrade would get to the VCSServiceManager step, fail and back out. It then left our existing vCenter 6.0u1 installation unable to start.

I did all the standard things you’ll find on VMwares Support site (and recommended by the support rep I got a hold of):

2119768 Error code 1603 when upgrading to vCenter Server 6.0
2127519 Installing the VMware vCenter Server 6.0 fails with the vminst.log error: MSI result of install of “C:\vCenter-Server\Packages\vcsservicemanager.msi” may have failed : 1603
2137365 Upgrade of vCenter from 5.x to 6.0 fails with “Installation of component VCSServiceManager failed with error code ‘1603’. Check the logs for more details.”
2113068 Upgrading or installing VMware vCenter Server 6.0 fails with the vminst.log error: Error in accessing registry entry for DSN
2119169 Installing VMware vCenter Server 6.0 using a Microsoft SQL database fails with the error: An error occurred while starting service ‘invsvc’

None helped.

While waiting for my VMware Support rep to dig through the log files, on a hunch, I made the following changes:

  1. Checkmarked ‘IPv6’ in the network stack for the servers network card
  2. Re-ran the vCenter installer separately by right clicking it and ‘Running As Administrator’ (\VMware-VIMSetup-all-6.0.0-3634788\vCenter-Server\VMware-vCenter-Server.exe)

The installation then succeeded and we have a functioning vCenter again.

Two fun facts:

  1. The UAC is disabled on our server
  2. IPv6 was (and still is) disabled via the registry using these utilities even though I’ve now re-checked IPv6 in the network cards network stack

Our server is in a fairly unique configuration I suspect but hopefully this will help someone else.

Exchange permission issues for a single user on a generic mailbox

We created a new generic mailbox in Exchange 2010. Created a group (Global) and added the users to it. We then created a Universal group, added the Global group to it and then added the Universal group to the mailbox with full mailbox permissions.

After the obligatory wait 24 hours for Exchange to update it’s permissions 4 of the 5 users in the Global group had access to the generic mailbox. The 5th user could not access the mailbox via their Outlook or Webmail. Permission denied errors.

We tried removing/re-adding them from the group with no success. We tried migrating their mailbox back and forth with no success.

Finally after some digging we thing we figured out the problem.

The user in question had originally been migrated from Exchange 2003 to 2010. Then at some point while on Exchange 2010 we had to purge/re-create their Mailbox.

The process of purging re-creating their mailbox changed their LegacyDN to the new, correct, format for Exchange 2010 and dumped their old Exchange 2003 LegacyDN.

To view a users legacy DN run the following Powershell command:

What we ended up doing is re-creating their LegacyDN from Exchange 2003 as a new X500 record and then they were instantly able to access the generic mailbox.

It could always be coincidence…..

References:

Networking randomly dies on a 2012 R2 vSphere VM

Strange issue. Simple solution.

We had a Windows Server 2012 R2 Domain Controller sitting on vSphere 5.5 (2068190) which would randomly lose it’s network connection.

When you logged into the system locally the network interface appeared to be up but you could not connect to anything outside of the VM.

If I rebooted the VM it would work for a few hours or less and then the network would drop out again.

Digging through the event viewer I came across these:

The VM had a E1000 NIC attached to it. I figured the issue was the VM NIC model and got some backup to my theory from here: https://community.spiceworks.com/topic/504405-windows-server-2012-r2-guest-os-on-vmware-keeps-losing-gateway-connection

The solution appears to have been removing the E1000 NIC and adding either a E1000E NIC or in my case a VMXNET 3 NIC.