DFS not working properly over VPN for personal computers

We recently switched to a new VPN server after Mac OS dropped support for PPTP and because we were way overdue to do it anyway. Since then personal computers were unable to access network shares via DFS.

They could go directly to the file server and that would work.

Users who connected to VPN with a organization owned laptops were able to use DFS.

After some digging it turned out the issue was our old VPN allowed for WINS (yes yes I know) and our new VPN has WINS disabled (by design, see… we’re trying)

The proper solution to this problem is to re-configure DFS to use DNS only: https://support.microsoft.com/en-us/help/244380/how-to-configure-dfs-to-use-fully-qualified-domain-names-in-referrals

Unfortunately I didn’t have the time to implement this.

What we ended up doing is re-configured the DHCP scope to set VPN users DNS Suffix to ‘vpn.mydomain.com’

I then added aliases for all of our file servers and DFS servers under ‘vpn.mydomain.com’. Example:

  • fileserver1.vpn.mydomain.com, CNAME, fileserver1.it.mydomain.com
  • dfsserver1.vpn.mydomain.com, CNAME, dfsserver1.it.mydomain.com

This is a crappy hacky work around that isn’t really sustainable but will work for now until we can sit down and plan changing our DFS over to use DNS only.

Domain Computers worked fine because we use group policy to push out multiple DNS search suffixes. DHCP doesn’t allow you to do this with Windows PCs so when they try to lookup ‘fileserver1’ they would try to hit WINS if implemented and then append their DNS suffix (vpn.mydomain.com) and then fail to find the file server resulting in a “Network Path Not Found” errors.

Error 500 when downloading/accessing OAB in Exchange 2016

We’ve just finished migrating all of our users from a legacy AD forest with Exchange 2010 into a whole new AD forest with Exchange 2016.

During the initial deployment of Exchange 2016 a new mailbox store was created, the existing system mailboxes were migrated (mailbox move) to the new store and the default Exchange 2016 store was deleted. We think our OAB broke at this point.

The behavior we were seeing was Outlook clients perpetually saying “Updating Address Book”, slow performance in Outlook where previously there wasn’t, delayed e-mail coming into mailboxes.

If users went into “C:\Users\%username%\AppData\Local\Microsoft\Outlook\Offline Address Books” they either had nothing or a old copy from our previous Exchange 2010 deployment. If they closed Outlook, purged the contents of “Offline Address Books” they would never be re-created because the OAB would never download from Exchange.

I’m assuming you’ve got everything else configured correctly on your end at this point. Things like Internal/External URLs, SSL certificates, yadda.

First thing first was to check if the OAB was on Exchange (I’ve redacted some information with *’s):

The lines that mattered here to me were:

  • Guid
  • LastTouchedTime

As you can see this OAB has never been touched which means to me it’s never been updated or created.

Using the GUID I checked to see what IIS had to say about the OAB:

  • https://mail.mydomain.com/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml

Loading this URL prompted me for a login, good so far, entered my domain credentials and got this error:

After this I manually checked each of our Exchange servers (we have 3) to see if they all gave the exact same Error 500. They did:

  • https://server-01.mydomain.com/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml
  • https://server-02.mydomain.com/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml
  • https://server-03.mydomain.com/OAB/28a57cb5-b2c8-442a-81f3-a48a05f9ab7b/oab.xml

During my Googling for this I found this blog post: http://unified.swiatelski.com/2011/02/exchange-2010-cannot-download-offline.html

The permissions were incorrect on our web.config file so I tried the suggested fix in the blog post. It didn’t work.

I then went through this document: https://technet.microsoft.com/en-us/library/gg247612%28v=exchg.150%29.aspx and made all of the tweaks needed to match those recommendations. I couldn’t find a Exchange 2016 specific document but there weren’t many tweaks to be made and they all made sense to me.

This did not resolve the issue either.

Next up was a Outlook Web Services health check using this PowerShell command:

Ok. Error 500. We know this already.

On my local system I ran this command in Command Prompt:

After digging around the file system of our Exchange servers I was unable to find a folder called 28a57cb5-b2c8-442a-81f3-a48a05f9ab7b in the following locations:

  • C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\OAB
  • C:\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\oab

This confirmed for me that the OAB wasn’t even being generated.

We then tried manually updating the OAB. The command succeeded but no OAB was generated.

We think what happened is the moving of the System Mailboxes way back at the beginning of our Exchange 2016 deployment broke the association of which system mailbox generates the OAB.

Running this command confirmed that there was no mailbox assigned to generate the OAB:

To fix the problem we did the following:

Enabled Global Distribution of the OAB (since we have 3 Exchange Servers why let one of them have all the fun?):

Then we created a new system mailbox dedicated to the purpose of generating the OAB and manually regenerated the OAB. We put the account in the same Domain as Exchange instead of where the users are:

It took all of 30 seconds and then the OAB URLs started working again and displaying a properly formatted XML file. Outlook clients automatically started downloading the OAB with no issues.

Checking our OAB:

The LastTouchedTime was now set. I’ll be checking again tomorrow to make sure the LastTouchedTime changes based on the Schedule.

Re-running this command confirmed there was now an account set to generate the OAB:

Re-running the OutlookWebServices test shows a successful OAB check:

Finally the event viewer on our Exchange server should also show Event ID 17001 and 17002 for the MSExchange Mailbox Assistants Provider showing the start and completion of the OAB generation:

 

References:

Single Mailbox Recovery can’t connect to Exchange

Our AD structure contains three domains, one root domain (int.mydom.com) and two sub-domains, one for servers/services (it.int.mydom.com) and one for user accounts (users.int.mydom.com).

  • int.mydom.com
    • users.int.mydom.com (Users live here)
    • it.int.mydom.com (Privileged IT accounts and Exchange live here)

We are running into problems with NetApp Single Mailbox Recovery where it is unable to connect to the target mailbox for drag/drop restores. In the mean time we’re just exporting PST files and letting the user do the work.

We started with SMR 7.1 where you could specify connecting to all Exchange mailboxes instead of an individual users mailbox. We found we had to change the domain controller SMR was using to find the mailboxes to one of the users.int.mydom.com DCs. By default SMR would select a DC in it.int.mydom.com where Exchange and our privileged IT accounts are.

We’re running Exchange 2016 so there were other issues when trying to do a drag/drop restore so we had to upgrade to SMR 7.2 which added support for Exchange 2016.

SMR 7.2 removed the ability to open all mailboxes and specify a domain controller so we’re stuck again.

After opening a case with NetApp they confirmed this is a know issue and SMR 7.2.1 should resolve it:

Please note that after checking with additional resources from SnapManager for Exchange we confirm there is a known issue with this function, there will be changes to this in the next version of SMBR, the Next SMBR release and version 7.2.1 should come soon but I am afraid at this point there is no a workaround we can follow.

So now we wait.

If you’re really desperate for drag/drop and have an environment like ours you might be able to create privileged accounts in users.int.mydom.com and use them for the restores. That might work.

Exchange users unable to share calendars post AD/Exchange migration

We just recently went through an AD forest migration AND an Exchange 2010 -> 2016 migration across forests at the same time. Good times.

One of the many issues that came up after the migration was the majority of our users being unable to share their calender’s with other users.

When trying to share via manually editing the calendar permissions users would get the error “One or more users cannot be added to the folder access list. Non-local users cannot be given rights on this server.”

 

If users tried to go the invite route by right clicking their calendar, choosing ‘Share’ and ‘Share Calendar’ they would get “Calendar sharing is not available with the following entries because of permission settings on your network:”

 

If you took a look at our GAL you’d see all of the users you couldn’t share with had a circle with a line through their entry:

 

I ended up stumbling across a solution by accident when trying to fix this on my own account. It turned out my account was a ‘Shared’ mailbox and not a ‘User’ mailbox. I converted it with the below PowerShell and then my account started working again:

This worked great for me but my situation was unique. Other users with the issue were already ‘User’ mailboxes. I took another problematic account and ran the above command on it and got this:

Despite that warning message this users mailbox was now fixed after the user closed/re-opened Outlook.

I re-ran the command against their mailbox and the output was this:

Why didn’t I get that second warning about not making any changes the first time I ran it? Simple. It’s because something was changed and Microsoft doesn’t think I need to know that.

Digging into the account attributes I figured out what changed. It’s called ‘msExchRecipientDisplayType’ and was introduced in Exchange 2007. This attribute determines what kind of recipient the mailbox is in the Address Book.

Pre-AD Migration msExchRecipientDisplayType was set to 1073741824 which is a “ACL able Mailbox User”.

Post-AD Migration msExchRecipientDisplayType was set to 0 which is a “Mailbox User”.

Makes sense now why you can’t apply permissions (ACL) on a “Mailbox User” when a “ACL able Mailbox User” user type exists.

We used Microsoft’s own tools (ADMT, Exchange 2016) to migrate our users from one forest and Exchange to another. Some where in that migration the attribute was wiped out and not transferred on 2941 out of 3123 mailboxes.

Here is how you can identity all users in your environment with this attribute set to “0”

Our environment is a mix of Shared, User, Resource and Equipment Mailboxes. There were affected accounts in all four categories. If we did a simple script that looked for “msExchRecipientDisplayType=0” and changed it to “1073741824” we might end up with the wrong value for a mailbox depending on what type it’s supposed to be. Based on my reading msExchRecipientDisplayType should be 1073741824 for Shared and User mailboxes, 7 for a Room Mailbox and 8 for a Equipment Mailbox.

We decided the best way to fix this was simply re-applying the user type that a mailbox already was. This made the PowerShell much simpler. Here’s what we ran:

These commands together fixed 2890 of 2941 broken mailboxes.

This will generate a simple report of which mailboxes weren’t converted and what type they are:

I took one of the accounts that wasn’t fixed and ran this command:

This corrected the account. No idea why the batch command didn’t. I ran this command for all of the regular mailboxes that didn’t fix in the batch and it worked fine. That left me with a bunch of shared mailboxes that were still broken and one user account that would not fix.

Running this on the shared mailboxes did not help:

I checked one of the problematic accounts in ADSIEdit.msc and it had the correct msExchRecipientDisplayType value of 1073741824 despite PowerShell telling me it was set to 0.

Since there were only 23 accounts left that were problematic I used ADSIEdit to verify and fix the remainders.

We ran into a few users who still had problems using the e-mail invite method to share their calendar. This was fixed by having them clear their Outlook auto-complete via these steps:

  1. On the File tab, choose Options > Mail.
  2. Under Send messages, choose Empty Auto-Complete List
  3. Choose Yes to confirm you want to empty the list.
  4. Close/Re-open Outlook
  5. Try again

 

References

mod_rewrite rule to redirect root separate from everything else

Took me a bit to get this working and maybe someone else will find this useful.

In our case we have a on-premise service being moved to the cloud. To prevent an outage and not break any links for the first month or so after the migration a new domain has been chosen to.

Example URLs:

  • https://onprem.pickysysadmin.ca
    • This should forward to the root of the new service (https://cloudservice.pickysysadmin.ca/)
  • https://onprem.pickysysadmin.ca/a_bunch_of_stuff/more_stuff
    • This should redirect to the specified location on the old service (https://onprem.pickysysadmin.ca/)

Right now DNS looks like this:

  • onprem.pickysysadmin.ca -> 10.0.0.1
  • cloudservice.pickysysadmin.ca -> 10.0.0.2
  • apache_redirect_server.pickysysadmin.ca -> 10.0.0.3

Because onprem.pickysysadmin.ca is vendor maintained we didn’t want to goof with it to much or we’d just put the rewrite rules on it and save us having to use a third server (apache_redirect_server.pickysysadmin.ca).

Our solution was:

  1. Created a new DNS alias called onprem-old.pickysysadmin.ca and pointed it at 10.0.0.1 (onprem.pickysysadmin.ca)
  2. Alter the Apache vhost configuration on onprem.pickysysadmin.ca so it would respond to onprem-old.pickysysadmin.ca
  3. Configured the following re-write rules on apache_redirect_server.pickysysadmin.ca
  4. On migration day we changed onprem.pickysysadmin.ca to point to 10.0.0.3 (apache_redirect_server.pickysysadmin.ca)

Not overly complicated but it took me a while to figure out how to get the rewrite rules working properly.