We’ve just finished migrating all of our users from a legacy AD forest with Exchange 2010 into a whole new AD forest with Exchange 2016.
During the initial deployment of Exchange 2016 a new mailbox store was created, the existing system mailboxes were migrated (mailbox move) to the new store and the default Exchange 2016 store was deleted. We think our OAB broke at this point.
The behavior we were seeing was Outlook clients perpetually saying “Updating Address Book”, slow performance in Outlook where previously there wasn’t, delayed e-mail coming into mailboxes.
If users went into “C:\Users\%username%\AppData\Local\Microsoft\Outlook\Offline Address Books” they either had nothing or a old copy from our previous Exchange 2010 deployment. If they closed Outlook, purged the contents of “Offline Address Books” they would never be re-created because the OAB would never download from Exchange.
I’m assuming you’ve got everything else configured correctly on your end at this point. Things like Internal/External URLs, SSL certificates, yadda.
First thing first was to check if the OAB was on Exchange (I’ve redacted some information with *’s):
Get-OfflineAddressBook |fl
Server :
GeneratingMailbox :
AddressLists : {\Default Global Address List}
Versions : {Version4}
IsDefault : True
PublicFolderDatabase :
PublicFolderDistributionEnabled : False
GlobalWebDistributionEnabled : False
WebDistributionEnabled : True
ShadowMailboxDistributionEnabled : False
UseE14SortOrder : False
UseE14SortOrderOrdinal : False
UseOrdinalSortedMultivaluedProperties : True
LastTouchedTime :
LastRequestedTime :
LastFailedTime :
LastNumberOfRecords : 2911
HttpHomeMdbLastProcessedBucket : 1000
LastGeneratingData : MailboxGuid: ee2ae703-5c35-4892-a43b-e2120618cc4a; DatabaseGuid: dcfde51e-44ff-4bb9-a44f-465998ffa3c8; Server: ********************
MaxBinaryPropertySize : 32768
MaxMultivaluedBinaryPropertySize : 65536
MaxStringPropertySize : 3400
MaxMultivaluedStringPropertySize : 65536
ConfiguredAttributes : {OfficeLocation, ANR, ProxyAddresses, ANR, PhoneticGivenName, ANR, GivenName, ANR...}
DiffRetentionPeriod : 30
Schedule : {Sun.5:00 AM-Sun.5:15 AM, Mon.5:00 AM-Mon.5:15 AM, Tue.5:00 AM-Tue.5:15 AM, Wed.5:00 AM-Wed.5:15 AM...}
VirtualDirectories : {}
AdminDisplayName :
Identity : \Default Offline Address Book
IsValid : True
ExchangeVersion : 0.20 (15.0.0.0)
Name : Default Offline Address Book
DistinguishedName : CN=Default Offline Address Book,CN=Offline Address Lists,CN=Address Lists Container,CN=******,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=*****,DC=*****,DC=*****
Guid : 28a57cb5-b2c8-442a-81f3-a48a05f9ab7b
ObjectCategory : *********/Configuration/Schema/ms-Exch-OAB
ObjectClass : {top, msExchOAB}
WhenChanged : 7/20/2017 9:47:15 AM
WhenCreated : 2/2/2017 10:22:06 AM
WhenChangedUTC : 7/20/2017 4:47:15 PM
WhenCreatedUTC : 2/2/2017 6:22:06 PM
OrganizationId :
Id : \Default Offline Address Book
OriginatingServer : ****************************
ObjectState : Unchanged
The lines that mattered here to me were:
As you can see this OAB has never been touched which means to me it’s never been updated or created.
Using the GUID I checked to see what IIS had to say about the OAB:
- https://mail.mydomain.com/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml
Loading this URL prompted me for a login, good so far, entered my domain credentials and got this error:
XML Parsing Error: no root element found
Location: https://************/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/OAB.xml
Line Number 1, Column 1:
^
After this I manually checked each of our Exchange servers (we have 3) to see if they all gave the exact same Error 500. They did:
- https://server-01.mydomain.com/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml
- https://server-02.mydomain.com/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml
- https://server-03.mydomain.com/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml
During my Googling for this I found this blog post: http://unified.swiatelski.com/2011/02/exchange-2010-cannot-download-offline.html
The permissions were incorrect on our web.config file so I tried the suggested fix in the blog post. It didn’t work.
I then went through this document: https://technet.microsoft.com/en-us/library/gg247612%28v=exchg.150%29.aspx and made all of the tweaks needed to match those recommendations. I couldn’t find a Exchange 2016 specific document but there weren’t many tweaks to be made and they all made sense to me.
This did not resolve the issue either.
Next up was a Outlook Web Services health check using this PowerShell command:
Test-OutlookWebServices -Identity ******@******* -MailboxCredential (Get-Credential) |fl
Source : ********************
ServiceEndpoint : ********************
Scenario : OfflineAddressBook
ScenarioDescription : Offline Address Book
Result : Failure
Latency : 57
Error : System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.InternalInvoke()
at Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.Invoke()
Verbose : [2017-07-19 20:51:57Z] Offline Address Book connecting to 'https://*****************/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml'.
[2017-07-19 20:51:57Z] Test account: ****************** Password: ******
[2017-07-19 20:51:57Z] Offline Address Book request:
User-Agent: *****************/Test-OutlookWebServices/*************@**************
Authorization: Negotiate ********************
Host: *******************
[2017-07-19 20:51:57Z] Offline Address Book request:
[2017-07-19 20:51:57Z] Offline Address Book response:
request-id: 9322f001-038d-447b-9825-127d6a53e2b4
X-CasErrorCode: OrganizationMailboxNotFound
Persistent-Auth: true
X-FEServer: ******************
Content-Length: 0
Cache-Control: private
Date: Wed, 19 Jul 2017 20:51:56 GMT
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
[2017-07-19 20:51:57Z] Offline Address Book response:
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.InternalInvoke()
at Microsoft.Exchange.Management.SystemConfigurationTasks.ServiceValidatorBase.Invoke()
MonitoringEventId : 6004
Ok. Error 500. We know this already.
On my local system I ran this command in Command Prompt:
C:\> bitsadmin /list /allusers /verbose
GUID: {60706BD3-5582-47E1-9F25-B9E9FEC6E5C0} DISPLAY: 'Microsoft Outlook Offline Address Book c31853e2bbf1be4583586e98075f0831'
TYPE: DOWNLOAD STATE: ERROR OWNER: ******\*********
PRIORITY: HIGH FILES: 0 / 1 BYTES: 0 / UNKNOWN
CREATION TIME: 2017-07-19 10:01:46 AM MODIFICATION TIME: 2017-07-19 11:01:48 AM
COMPLETION TIME: UNKNOWN ACL FLAGS:
NOTIFY INTERFACE: UNREGISTERED NOTIFICATION FLAGS: 3
RETRY DELAY: 300 NO PROGRESS TIMEOUT: 3600 ERROR COUNT: 13
PROXY USAGE: PRECONFIG PROXY LIST: NULL PROXY BYPASS LIST: NULL
ERROR FILE: https://****************/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml -> C:\Users\************\AppData\Local\Microsoft\Outlook\oab4.xml
ERROR CODE: 0x801901f4 - HTTP status 500: An unexpected condition prevented the server from fulfilling the request.
ERROR CONTEXT: 0x00000005 - The error occurred while the remote file was being processed.
DESCRIPTION: Microsoft Outlook Offline Address Book Directory
JOB FILES:
0 / UNKNOWN WORKING https://*************/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml -> C:\Users\**************\AppData\Local\Microsoft\Outlook\oab4.xml
NOTIFICATION COMMAND LINE: none
owner MIC integrity level: MEDIUM
owner elevated ? false
Peercaching flags
Enable download from peers :false
Enable serving to peers :false
CUSTOM HEADERS: NULL
After digging around the file system of our Exchange servers I was unable to find a folder called 28a57cb5-b2c8-442a-81f3-a48a05f9ab7b in the following locations:
- C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\OAB
- C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\oab
This confirmed for me that the OAB wasn’t even being generated.
We then tried manually updating the OAB. The command succeeded but no OAB was generated.
Update-OfflineAddressBook -Identity "Default Offline Address Book"
We think what happened is the moving of the System Mailboxes way back at the beginning of our Exchange 2016 deployment broke the association of which system mailbox generates the OAB.
Running this command confirmed that there was no mailbox assigned to generate the OAB:
Get-Mailbox -Arbitration | where {$_.PersistedCapabilities -like “*OAB*”} | ft Name,Servername,Database
To fix the problem we did the following:
Enabled Global Distribution of the OAB (since we have 3 Exchange Servers why let one of them have all the fun?):
Set-OfflineAddressBook -Identity "Default Offline Address Book" -VirtualDirectories $null -DomainController DC01.users.int.mydom.com -Verbose
Set-OfflineAddressBook -Identity "Default Offline Address Book" -GlobalWebDistributionEnabled $true -DomainController DC01.users.int.mydom.com -Verbose
Then we created a new system mailbox dedicated to the purpose of generating the OAB and manually regenerated the OAB. We put the account in the same Domain as Exchange instead of where the users are:
New-Mailbox -Arbitration -Name "OAB Master" -UserPrincipalName [email protected]******* -DomainController DC01.it.mydom.com
Set-Mailbox -Identity "OAB Master" -Arbitration -OABGen $true -MaxSendSize 1GB -DomainController DC01.it.mydom.com
Update-OfflineAddressBook -Identity "Default Offline Address Book" -Verbose
It took all of 30 seconds and then the OAB URLs started working again and displaying a properly formatted XML file. Outlook clients automatically started downloading the OAB with no issues.
Checking our OAB:
Get-OfflineAddressBook -Identity "Default Offline Address Book" |fl
Server :
GeneratingMailbox :
AddressLists : {\Default Global Address List}
Versions : {Version4}
IsDefault : True
PublicFolderDatabase :
PublicFolderDistributionEnabled : False
GlobalWebDistributionEnabled : True
WebDistributionEnabled : True
ShadowMailboxDistributionEnabled : False
UseE14SortOrder : False
UseE14SortOrderOrdinal : False
UseOrdinalSortedMultivaluedProperties : True
LastTouchedTime : 7/20/2017 9:46:54 AM
LastRequestedTime :
LastFailedTime :
LastNumberOfRecords : 2911
HttpHomeMdbLastProcessedBucket : 1000
LastGeneratingData : MailboxGuid: ee2ae703-5c35-4892-a43b-e2120618cc4a; DatabaseGuid: dcfde51e-44ff-4bb9-a44f-465998ffa3c8; Server: BROCKTON-03.it.int.viu.ca
MaxBinaryPropertySize : 32768
MaxMultivaluedBinaryPropertySize : 65536
MaxStringPropertySize : 3400
MaxMultivaluedStringPropertySize : 65536
ConfiguredAttributes : {OfficeLocation, ANR, ProxyAddresses, ANR, PhoneticGivenName, ANR, GivenName, ANR...}
DiffRetentionPeriod : 30
Schedule : {Sun.5:00 AM-Sun.5:15 AM, Mon.5:00 AM-Mon.5:15 AM, Tue.5:00 AM-Tue.5:15 AM, Wed.5:00 AM-Wed.5:15 AM...}
VirtualDirectories : {}
AdminDisplayName :
Identity : \Default Offline Address Book
IsValid : True
ExchangeVersion : 0.20 (15.0.0.0)
Name : Default Offline Address Book
DistinguishedName : CN=Default Offline Address Book,CN=Offline Address Lists,CN=Address Lists Container,CN=*****,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=*****,DC=*****,DC=****
Guid : 28a57cb5-b2c8-442a-81f3-a48a05f9ab7b
ObjectCategory : ************/Configuration/Schema/ms-Exch-OAB
ObjectClass : {top, msExchOAB}
WhenChanged : 7/20/2017 9:47:15 AM
WhenCreated : 2/2/2017 10:22:06 AM
WhenChangedUTC : 7/20/2017 4:47:15 PM
WhenCreatedUTC : 2/2/2017 6:22:06 PM
OrganizationId :
Id : \Default Offline Address Book
OriginatingServer : *****************************
ObjectState : Unchanged
The LastTouchedTime was now set. I’ll be checking again tomorrow to make sure the LastTouchedTime changes based on the Schedule.
Re-running this command confirmed there was now an account set to generate the OAB:
Get-Mailbox -Arbitration | where {$_.PersistedCapabilities -like “*OAB*”} | ft Name,Servername,Database
Re-running the OutlookWebServices test shows a successful OAB check:
Test-OutlookWebServices -Identity *********@********** -MailboxCredential (Get-Credential) |fl
Source : ******************
ServiceEndpoint : ******************
Scenario : OfflineAddressBook
ScenarioDescription : Offline Address Book
Result : Success
Latency : 46
Error :
Verbose : [2017-07-20 17:52:34Z] Offline Address Book connecting to 'https://******************/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml'.
[2017-07-20 17:52:34Z] Test account: ****************** Password: ******
[2017-07-20 17:52:34Z] Offline Address Book request:
User-Agent: ******************/Test-OutlookWebServices/******************
Authorization: Negotiate ******************
Host: ******************
[2017-07-20 17:52:34Z] Offline Address Book request:
[2017-07-20 17:52:34Z] Offline Address Book response:
request-id: 3dc2a268-33a2-428f-8caa-fff069600cb9
X-CalculatedBETarget: ******************
X-DiagInfo: ******************
X-BEServer: ******************
Persistent-Auth: true
X-FEServer: ******************
Accept-Ranges: bytes
Content-Length: 20831
Cache-Control: private
Content-Type: text/xml
Date: Thu, 20 Jul 2017 17:52:34 GMT
ETag: W/"993c1dcb771d31:0"
Last-Modified: Thu, 20 Jul 2017 16:46:54 GMT
Set-Cookie: X-BackEndCookie=S-1-5-21-1542403177-3275365000-3172300963-32179=rJqNiZqNgb2tsLy0q7Cx0s/M0ZaL0ZaRi9GJlorRnJ6BzsbLzc/JzsjNyoHNz87I0s/I0s3Pq87Hxc/NxczK;
expires=Thu, 20-Jul-2017 18:02:35 GMT; path=/OAB; secure; HttpOnly
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
MonitoringEventId : 5004
Finally the event viewer on our Exchange server should also show Event ID 17001 and 17002 for the MSExchange Mailbox Assistants Provider showing the start and completion of the OAB generation:
Log Name: Application
Source: MSExchange Mailbox Assistants Provider
Date: 7/20/2017 9:46:44 AM
Event ID: 17001
Task Category: OAB Generator Assistant
Level: Information
Keywords: Classic
User: N/A
Computer: ****************
Description:
Generation of OAB "\Default Offline Address Book" started.
Dn: CN=Default Offline Address Book,CN=Offline Address Lists,CN=Address Lists Container,CN=****************,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=****,DC=****,DC=****
ObjectGuid: 28a57cb5-b2c8-442a-81f3-a48a05f9ab7b
Log Name: Application
Source: MSExchange Mailbox Assistants Provider
Date: 7/20/2017 9:46:54 AM
Event ID: 17002
Task Category: OAB Generator Assistant
Level: Information
Keywords: Classic
User: N/A
Computer: ****************
Description:
Generation of OAB "\Default Offline Address Book" completed.
Dn: CN=Default Offline Address Book,CN=Offline Address Lists,CN=Address Lists Container,CN=****************,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=****,DC=****,DC=****
ObjectGuid: 28a57cb5-b2c8-442a-81f3-a48a05f9ab7b
GenerationTimeMilliseconds: S:OAB='\Default Offline Address Book';I64:Status=0;Dt:StartTime=2017-07-20T16:46:45.2430374Z;Dt:EndTime=2017-07-20T16:46:54.4459486Z;S:DC=****;I32:Total.Records=2911;I32:Total.TempFiles=3;Ti:TimeWritingFiles=00:00:03.4627334;S:Org==****,;S:Wasted=True;S:HABEnabled=False;S:DroppedFilesOnly=False;I32:Total.RecordsAddedChurn=0;I32:Total.RecordsDeletedChurn=0;I32:Total.RecordsModifiedChurn=0;I32:UnCompressedFileSize=1882118;I32:CompressedFileSize=401288;I32:DiffFileSize=0;I32:ManifestSize=20831;S:ProductVersion=15.01.0669.032;S:GenerationType=None;Ti:PrepareFilesForOABGeneration.DownloadFilesFromMailbox.StoreRpcLatency=00:00:00.0470000;I32:PrepareFilesForOABGeneration.DownloadFilesFromMailbox.StoreRpcCount=17;Ti:PrepareFilesForOABGeneration.DownloadFilesFromMailbox.CpuTime=00:00:00.0156250;Ti:PrepareFilesForOABGeneration.DownloadFilesFromMailbox.ElapsedTime=00:00:00.0770176;Ti:PrepareFilesForOABGeneration.CpuTime=00:00:00.0156250;Ti:PrepareFilesForOABGeneration.ElapsedTime=00:00:00.0783983;Ti:Total.CpuTime=00:00:07.4218750;Ti:Total.ElapsedTime=00:00:09.1764103;I32:GenerateOrLinkTemplateFiles.GenerateTemplateFiles.FS.BytesRead=1495956;I32:GenerateOrLinkTemplateFiles.GenerateTemplateFiles.FS.BytesWritten=2208646;Ti:GenerateOrLinkTemplateFiles.GenerateTemplateFiles.FS.Reading.ElapsedTime=00:00:00.0059147;Ti:GenerateOrLinkTemplateFiles.GenerateTemplateFiles.FS.Writing.ElapsedTime=00:00:00.0177426;Ti:GenerateOrLinkTemplateFiles.GenerateTemplateFiles.CpuTime=00:00:02.6875000;Ti:GenerateOrLinkTemplateFiles.GenerateTemplateFiles.ElapsedTime=00:00:03.4445248;Ti:GenerateOrLinkTemplateFiles.CpuTime=00:00:02.6875000;Ti:GenerateOrLinkTemplateFiles.ElapsedTime=00:00:03.4445926;Ti:BeginGeneratingAddressListFiles.CpuTime=00:00:00;Ti:BeginGeneratingAddressListFiles.ElapsedTime=00:00:00.0017451;Ti:ProcessOnePageOfADResults.ADQuery.LdapLatency=00:00:01.3780000;I32:ProcessOnePageOfADResults.ADQuery.LdapCount=3;Ti:ProcessOnePageOfADResults.ADQuery.CpuTime=00:00:01.2968750;Ti:ProcessOnePageOfADResults.ADQuery.ElapsedTime=00:00:02.2469338;Ti:ProcessOnePageOfADResults.SortADResults.CpuTime=00:00:00.0468750;Ti:ProcessOnePageOfADResults.SortADResults.ElapsedTime=00:00:00.0264904;I32:ProcessOnePageOfADResults.ResolveLinks.ActiveManager.CalculatePreferredHomeServer.Count=5;Ti:ProcessOnePageOfADResults.ResolveLinks.ActiveManager.CalculatePreferredHomeServer.Latency=00:00:00.0046106;Ti:ProcessOnePageOfADResults.ResolveLinks.LdapLatency=00:00:00;I32:ProcessOnePageOfADResults.ResolveLinks.LdapCount=0;Ti:ProcessOnePageOfADResults.ResolveLinks.CpuTime=00:00:00;Ti:ProcessOnePageOfADResults.ResolveLinks.ElapsedTime=00:00:00.0216625;I32:ProcessOnePageOfADResults.WriteTempFiles.FS.BytesRead=0;I32:ProcessOnePageOfADResults.WriteTempFiles.FS.BytesWritten=1939744;Ti:ProcessOnePageOfADResults.WriteTempFiles.FS.Reading.ElapsedTime=00:00:00;Ti:ProcessOnePageOfADResults.WriteTempFiles.FS.Writing.ElapsedTime=00:00:00.0144446;Ti:ProcessOnePageOfADResults.WriteTempFiles.CpuTime=00:00:00.3125000;Ti:ProcessOnePageOfADResults.WriteTempFiles.ElapsedTime=00:00:00.3055291;Ti:ProcessOnePageOfADResults.CpuTime=00:00:01.6562500;Ti:ProcessOnePageOfADResults.ElapsedTime=00:00:02.6047902;I32:ProduceSortedFlatFile.FS.BytesRead=1928100;I32:ProduceSortedFlatFile.FS.BytesWritten=1882130;Ti:ProduceSortedFlatFile.FS.Reading.ElapsedTime=00:00:00.0049700;Ti:ProduceSortedFlatFile.FS.Writing.ElapsedTime=00:00:00.0059073;Ti:ProduceSortedFlatFile.CpuTime=00:00:00.0468750;Ti:ProduceSortedFlatFile.ElapsedTime=00:00:00.0408414;I32:FinishGeneratingAddressListFiles.CompressGeneratedFiles.FS.BytesRead=1882118;I32:FinishGeneratingAddressListFiles.CompressGeneratedFiles.FS.BytesWritten=1882118;Ti:FinishGeneratingAddressListFiles.CompressGeneratedFiles.FS.Reading.ElapsedTime=00:00:00.0025546;Ti:FinishGeneratingAddressListFiles.CompressGeneratedFiles.FS.Writing.ElapsedTime=00:00:01.3073740;Ti:FinishGeneratingAddressListFiles.CompressGeneratedFiles.CpuTime=00:00:01.3593750;Ti:FinishGeneratingAddressListFiles.CompressGeneratedFiles.ElapsedTime=00:00:01.3557792;I32:FinishGeneratingAddressListFiles.GenerateDiffFiles.FS.BytesRead=3764236;I32:FinishGeneratingAddressListFiles.GenerateDiffFiles.FS.BytesWritten=932;Ti:FinishGeneratingAddressListFiles.GenerateDiffFiles.FS.Reading.ElapsedTime=00:00:00.0116925;Ti:FinishGeneratingAddressListFiles.GenerateDiffFiles.FS.Writing.ElapsedTime=00:00:00.0000426;Ti:FinishGeneratingAddressListFiles.GenerateDiffFiles.CpuTime=00:00:01.6250000;Ti:FinishGeneratingAddressListFiles.GenerateDiffFiles.ElapsedTime=00:00:01.6246996;Ti:FinishGeneratingAddressListFiles.CpuTime=00:00:02.9843750;Ti:FinishGeneratingAddressListFiles.ElapsedTime=00:00:02.9806045;Ti:Publish.CpuTime=00:00:00.0312500;Ti:Publish.ElapsedTime=00:00:00.0251272;;
TotalNumberofRecords: %5
TypeofGeneration: %6
UncompressedSizeBytes: %7
CompressedFilesSizeBytes: %8
ManifestSizeBytes: %9
BuildVersion %10
%11
References: