Troubleshoot Exchange Hybrid Mail Flow with Office 365

There are a few consistent issues that I see commonly occur when implementing an Exchange hybrid with Office 365.  All affect mail flow to, from, or both to and from an on premise Exchange 2010 or 2013 environment and Office 365 Exchange Online.

Below are the most common issues that I would like to describe and illustrate.

Issues with mail flow to Office 365

TargetAddress

When performing a remote mailbox move via Exchange 2013, we connect to Office 365 from the hybrid server and from the migration section, we select ‘Migrate to Exchange Online’. When configuring the “move configuration” the “target delivery domain” is static–we can’t select anything other than a *.mail.onmicrosoft.com domains.

However, when performing a remote mailbox move via Exchange 2010, the experience is slightly different. During a move request via the Exchange Management Console (EMC), we have options to select a target domain which includes the default email domain as well as any *.mail.onmicrosoft.com domains may be added to Office 365.

The challenge comes when the person performing the move selects the default domain (usually denoted with “False”).  The reason this is a problem is that when we move a mailbox from on premise to Office 365 via hybrid the ‘targetAddress’ Active Directory (AD) attribute is stamped with a local email address (i.e. user@domain.com) instead of a remote email address (i.e. user@domain.mail.onmicrosoft.com).  And though the mailbox move will complete, and the user accesses the O365 mailbox and can successfully send emails to on premise and external mailboxes, it will be unable to receive emails from on premise or external senders.  This is, again, because the on premise AD attribute is stamped with a local email address.  Exchange will decipher this information and try to send the message locally but since there is no mailbox locally the sender will eventually receive an NDR, similar to the following…

#5.7.1 smtp;550 5.7.1 Unable to relay

To resolve this issue…

  1. Open Active Directory Users and Computers (ADUC).
  2. Open the properties affected user object.
  3. Select the Attribute Editor tab.
  4. Locate the ‘targetAddress’ attribute.
  5. Edit the ‘targetAddress’ attribute to something like this … “SMTP:user@domain.mail.onmicrosoft.com” (excluding quotes) and click OK.
  6. Click OK to complete and close the changes to the user object.
  7. Finally, test mail flow from an on premise or external sender.

By taking these steps, Exchange now knows how to deal with mail flow sent to it by forwarding the messages to the remote environment mailbox(es), in Office 365.  Therefore, restoring mail flow.

 

Issues with mail flow to and from Office 365

Invalid SSL Certificate

A recent experience I had occurred after creating an Exchange 2013 hybrid environment and then created a new certificate request.  After creating the new SSL cert request and applying the new certificate, mail flow both to and from Office 365 resources stopped.  The following NDRs were being received…

#4.4.7 smtp;550 4.4.7 QUEUE.Expired; message expired

In troubleshooting, we looked at the connectors in Office 365 and in Exchange but the issue was not resolved.  We also looked at the mail flow itself via logs.

In looking at the SMTP logs (C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\ProtocolLog\SmtpSend), we found TLS negotiation errors with every email we attempted to send from on premise to O365, and an associated certificate thumbprint.  The TLS error looked like this…

TLS negotiation failed with error NoCredentials

We also took the certificate thumbprint found in the SMTP log that was associated with the error and compared it to the thumbprint of the current certificates installed; the thumbprint did not match any of those installed on the Exchange 2013 servers.

We used this command to review all of the certificates and their thumbprints installed on the Exchange server…

Get-ExchangeCertificate | fl

To resolve, we used the MMC to look for the currently published “personal” certificates on the local computer.  In doing so, we found both public CA certificates present–the current one and the old/expired one.  Comparing both the error in the SMTP log and the thumbprint of the certificates, we identified that the thumbprint in the error was referring to the old (and no longer valid) certificate.  At this point, we deleted the old/expired certificate and restarted IIS (iisreset).

Almost immediately, items that were in the local Exchange server queue were delivered to the Office 365 mailboxes.  Mail flow was restored and we confirmed with additional testing to and from Office 365.  Also, we reviewed the SMTP logs to find that no other TLS errors were occurring.

 

Issues with mail flow in either director to Office 365

Opportunistic TLS

The simplest option we have for troubleshooting mail flow with a hybrid deployment is to adjust the security on the Office 365 connectors (Exchange Admin Center > Mail Flow > Connectors).

NOTE: Prior to making any changes to the connectors, it is strongly recommended to document the current settings in the event the changes need to be reversed.

IMPORTANT: For the Inbound Connector, document the entire name of the certificate.

The default security setting for the “inbound” connector is Force TLS.  Let’s change the Connector Security to Opportunistic TLS; this should change the Domain Restrictions to None. Click Save to complete the setting change to the “inbound” connector.

NOTE: Opportunistic TLS attempts a TLS connection. An SMTP connection is used if the TLS connection is unsuccessful.

With the “outbound” connector, we will make a similar change.  The default Connection Security setting is Recipient Certificate Matches Domain.  We will change the setting to Opportunistic TLS and save the change for the “outbound” connector.

At the most basic level, this simple change is what Microsoft changes first during support calls to help with mail flow between on premise Exchange and Office 365.

 

Exchange Send and Receive Connectors

Microsoft also checks the send and receive connectors in the on premise Exchange deployment.  With Exchange 2010, they confirm connectors have been created and IP addresses are set in the receive connector.

If any aspect of either connector is not set, delete both the send and receive connectors for “Office 365” and re-run the hybrid configuration wizard (HCW).

 

Additional Troubleshooting Resources

In some cases, we need or want additional resources to assist with troubleshooting mail flow before having to contact support.  If that is the case, I recommend the Mail Flow Guided Walkthrough for Office 365.  If then we are still unable to resolve the issue, it would be wise to open a service request in the Office 365 Admin Center or contact support by phone (1-800-865-9408 in the United States).

 

In conclusion, this article provides some basics options to assist with troubleshooting hybrid mail flow with Office 365 and helps you to become more successful at identifying probable issues and resolutions in the future.

Good luck and have fun!

 

Reference(s):

Advertisements

One thought on “Troubleshoot Exchange Hybrid Mail Flow with Office 365

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s