Monthly Archives: September 2012

NetApp SnapManager for Hyper-V won’t install because KB2263829 isn’t installed

by

0

I’m deploying a new Hyper-V cluster on Windows Server 2008 R2 with SP1 and trying to install NetApp SnapManager for Hyper-V 1.1.

As part of the installation pre-requisites NetApp wants you to install the following KBs:

  • KB2406705
  • KB2531907
  • KB2263829
  • KB2494016
  • KB2637197
  • KB2517329

Some of these KBs have been replaced with newer ones if you’ve already run Windows Updates on your system before getting to this list of pre-reqs. For the most part that seems fine with one exception.

KB2263829 is now obsolete (or at least in my deployment it was) and would not install telling me “The update is not applicable to your computer.”

A uninstall and re-install of the Hyper-V role still would not allow me to install KB2263829 and because it wasn’t installed SnapManager for Hyper-V won’t install complaining the hotfix is missing.

I called up NetApp and was told to call Microsoft to find out why the hotfix won’t install. They didn’t seem to like my explanation that the hotfix seemed to be obsolete now. A bit of Googling turned up this discussion thread: https://communities.netapp.com/message/91200

The short answer is that the problem should be resolved in SnapManager for Hyper-V 1.2. In the mean time you can install KB2586470 on your Hyper-V server and then SnapManager for Hyper-V will install… or I guess you could use non-SP1 media for your Server 2008 R2 deployment and make sure you get KB2263829 installed before running Windows Updates.

Symantec Backup Exec NDMP backups fail the verify stage when backing up from a NetApp

by

17

We are using Symantec Backup Exec 2010 R3 to perform NDMP backups of our NetApp FAS2020 running Data OnTap 7.3.7.

All was working well until we upgraded from Data OnTap 7.3.6 to 7.3.7. Since then Backup Exec 2010 R3 is reporting every NDMP backup as a failure with the below error during the verify stage of the backup:

Job ended: Wednesday, September 12, 2012 at 3:52:55 AM
Completed status: Failed
Final error: 0xe000fe0d - A device-specific error occurred.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-65037

After contacting NetApp they told us Symantec was at fault.

Symantec did some digging into job logs and came across this error in the NDMP jobs log file taken using Backup Execs built in logging tools:

BENGINE: [07/09/12 10:58:57] [8864] [ndmp\ndmpcomm] - ERROR: 7 Error: I/O error
BENGINE: [07/09/12 10:58:57] [8864] [loops] - NDMP Log Message: Storing of nlist entries failed.
BENGINE: [07/09/12 10:58:57] [8864] [loops] - NDMP Notify Data Halted: Aborted
BENGINE: [07/09/12 10:58:57] [8864] [loops] - NDMP Log Message: Aborted by client

Symantec then recommended contacting NetApp again after reviewing their ticketing system and seeing other NetApp customers had this exact same problem and were told to contact NetApp.

After getting a hold of NetApp again with the above information they have now told me this is a known issue and there is an internal bug report at NetApp for it. There is supposedly a known fix but it is not yet available for any shipped versions of Data OnTap. The internal bug report lists the following work arounds for NetBackup (which I assume will work on Backup Exec):

  1. Restore the directory to another location and extract the file after the restore completes.
  2. To perform a single file restore without using DAR, set the value of the environment variable EXTRACT to e or E. However, the single file restore reads the whole backup stream on the tape and this restore operation might be slow.
  3. Set the NDMP version on the storage system to version 3 and then perform the restore.

I’ve tested option 3 by running the following commands on our FAS2020

filer> ndmpd off
filer> ndmpd version 3
filer> ndmpd on

and backups are now failing with a different error:

Job ended: Wednesday, September 12, 2012 at 2:49:51 PM
Completed status: Failed
Final error: 0xe000feb9 - The NDMP subsystem reports that a request cannot be processed because it is in the wrong state to service the request.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-65209

I reverted the NDMP version back to 4 and will now wait for a conference call with NetApp and Symantec to get to the bottom of this.

Despite the reported failures the backups do appear to still be good.

Update – September 14th, 2012

After a conference call with Symantec and NetApp the final conclusion is that this is a bug that only exists in Data OnTap 7.3.7 and it will be fixed in Data OnTap 7.3.7p1 which should be released sometime in the near future. No exact dates were provided.

The public bug report for this on NetApps site is 613414. You can subscribe to that bug with your NetApp account and when it’s resolved (the release of 7.3.7p1) you will receive an e-mail. The NetApp rep wasn’t certain if general e-mails go out to NetApp custers on ‘p’ releases of Data OnTap and subscribing to the bug should guarantee notification when the new version is released.

That public bug report lists the problem only effects Data OnTap 7.3.8. That is incorrect and it should read 7.3.7.

In the mean time the workarounds remain almost the same:

  1. Create CIFS shares for the volumes you backup via NDMP and change your backups to use the CIFS share instead of NDMP
  2. Disable backup verification in Backup Exec for your NDMP jobs
  3. Downgrade to Data OnTap 7.3.6
  4. Wait for Data OnTap 7.3.7p1

 

Update – October 4th, 2012

Sam in the comments got this update from NetApp

This fix will be included in 7.3.7P1. We are expecting 7.3.7P1 currently has a target release date of Oct 29th.

Here’s hoping.

 

Update – October 30th, 2012

Data OnTap 7.3.7P1 is out! We have one confirmation that this patch has fixed the verify problem.

Release Notes and Download: http://support.netapp.com/NOW/download/software/ontap/7.3.7P1/

 

Update – November 27th, 2012

I can confirm that the 7.3.7P1 patch has corrected this problem for us.