We are using Symantec Backup Exec 2010 R3 to perform NDMP backups of our NetApp FAS2020 running Data OnTap 7.3.7.
All was working well until we upgraded from Data OnTap 7.3.6 to 7.3.7. Since then Backup Exec 2010 R3 is reporting every NDMP backup as a failure with the below error during the verify stage of the backup:
Job ended: Wednesday, September 12, 2012 at 3:52:55 AM Completed status: Failed Final error: 0xe000fe0d - A device-specific error occurred. Final error category: Resource Errors For additional information regarding this error refer to link V-79-57344-65037
After contacting NetApp they told us Symantec was at fault.
Symantec did some digging into job logs and came across this error in the NDMP jobs log file taken using Backup Execs built in logging tools:
BENGINE: [07/09/12 10:58:57]  [ndmp\ndmpcomm] - ERROR: 7 Error: I/O error BENGINE: [07/09/12 10:58:57]  [loops] - NDMP Log Message: Storing of nlist entries failed. BENGINE: [07/09/12 10:58:57]  [loops] - NDMP Notify Data Halted: Aborted BENGINE: [07/09/12 10:58:57]  [loops] - NDMP Log Message: Aborted by client
Symantec then recommended contacting NetApp again after reviewing their ticketing system and seeing other NetApp customers had this exact same problem and were told to contact NetApp.
After getting a hold of NetApp again with the above information they have now told me this is a known issue and there is an internal bug report at NetApp for it. There is supposedly a known fix but it is not yet available for any shipped versions of Data OnTap. The internal bug report lists the following work arounds for NetBackup (which I assume will work on Backup Exec):
- Restore the directory to another location and extract the file after the restore completes.
- To perform a single file restore without using DAR, set the value of the environment variable EXTRACT to e or E. However, the single file restore reads the whole backup stream on the tape and this restore operation might be slow.
- Set the NDMP version on the storage system to version 3 and then perform the restore.
I’ve tested option 3 by running the following commands on our FAS2020
filer> ndmpd off filer> ndmpd version 3 filer> ndmpd on
and backups are now failing with a different error:
Job ended: Wednesday, September 12, 2012 at 2:49:51 PM Completed status: Failed Final error: 0xe000feb9 - The NDMP subsystem reports that a request cannot be processed because it is in the wrong state to service the request. Final error category: Resource Errors For additional information regarding this error refer to link V-79-57344-65209
I reverted the NDMP version back to 4 and will now wait for a conference call with NetApp and Symantec to get to the bottom of this.
Despite the reported failures the backups do appear to still be good.
Update – September 14th, 2012
After a conference call with Symantec and NetApp the final conclusion is that this is a bug that only exists in Data OnTap 7.3.7 and it will be fixed in Data OnTap 7.3.7p1 which should be released sometime in the near future. No exact dates were provided.
The public bug report for this on NetApps site is 613414. You can subscribe to that bug with your NetApp account and when it’s resolved (the release of 7.3.7p1) you will receive an e-mail. The NetApp rep wasn’t certain if general e-mails go out to NetApp custers on ‘p’ releases of Data OnTap and subscribing to the bug should guarantee notification when the new version is released.
That public bug report lists the problem only effects Data OnTap 7.3.8. That is incorrect and it should read 7.3.7.
In the mean time the workarounds remain almost the same:
- Create CIFS shares for the volumes you backup via NDMP and change your backups to use the CIFS share instead of NDMP
- Disable backup verification in Backup Exec for your NDMP jobs
- Downgrade to Data OnTap 7.3.6
- Wait for Data OnTap 7.3.7p1
Update – October 4th, 2012
Sam in the comments got this update from NetApp
This fix will be included in 7.3.7P1. We are expecting 7.3.7P1 currently has a target release date of Oct 29th.
Update – October 30th, 2012
Data OnTap 7.3.7P1 is out! We have one confirmation that this patch has fixed the verify problem.
Release Notes and Download: http://support.netapp.com/NOW/download/software/ontap/7.3.7P1/
Update – November 27th, 2012
I can confirm that the 7.3.7P1 patch has corrected this problem for us.