In case you were wondering, before “blocking” sign-in for users in Office 365, make 100% certain that the account being modified isn’t critical for users when accessing Office 365 resources.
Here’s a new one I experienced this week.
I received an email very early Tuesday morning. A client stated they hadn’t synced in since the previous Friday and password changes weren’t being reflected in O365. While reviewing their AAD Connect deployment, there were a few things that immediately stood out.
1) The AD user being used with AAD Connect did not have the correct permissions (membership to Enterprise Admins is required);
2) AAD Connect was installed on a domain controller. Not the end of the world because it is supported, however, strongly discouraged;
3) Running “Get-ADSyncScheduler” produced nothing but red text regarding CSLIB; and
4) Running “Start-ADSyncSyncCycle” was similar with the red text but referred to AADSTS70002 and not a single sync task ran.
Number 1 was easy to fix but didn’t resolve the issue after logging out and restarting the server. Number 2 wasn’t going to change. Number 3 and 4 baffled me. Additionally, the application event log was riddled with warnings like this…
Log Name: Application Source: Directory Synchronization Date: 6/22/2016 7:57:23 AM Event ID: 905 Task Category: None Level: Warning Keywords: Classic User: N/A Computer: dc.domain.local Description: Scheduler::SchedulerThreadMain : An error occured and scheduler run failed to perform all operation. System.Management.Automation.CmdletInvocationException: Microsoft.Online.Coexistence.ProvisionException: An error occurred. Error Code: 19. Error Description: Synchronization has been blocked. The account provided during has been disabled. Contact Technical Support. ...
The fact that the event log stated an “account has been disabled” left me wondering, “What account is disabled?” The O365 global admin isn’t disabled and neither is the AD account we logged in with to view the AAD Connect issues and settings.
I also recalled that in my lab environment recently the enabled AAD Connect automated update option caused the AAD Connect tool blow when it attempted to upgrade. Going on that train of thought, I removed AAD Connect entirely and restarted the server. Then the latest version of AAD Connect was downloaded and I began to reinstall it. After some convincing, the client conceded to create a service account in AD for AAD Connect to avoid frequent password changes or a host of other reasons. During the install, and after providing both a global admin and the AD service account info, this error popped up.
There was the error again … referring to “AADSTS70002”. The install log wasn’t much different. Frustration set in because there was no account in AD or O365 named AADSTS70002; or the like. Then I remembered that during the initial installation of AAD Connect a service account is created in O365. The account has a name like this “Sync_AADConnectServerName_xxxxxxxxxxxx@tenant.onmicrosoft.com” and is unique for every O365 tenant. I followed that instinct to find that the Directory Sync Service account showed a status of “Blocked”. WHAT?! WHY?!
Once the properties of that account was set back to the default of “Sign-in allowed”, the installation was able to complete and syncing proceeded without further issue.
In this specific case, it was essential to remove and re-install AAD Connect to resolve the CSLIB error. However, the resolution to address the sync problem was fairly simple. It is frustrating to spend 60-90 minutes troubleshooting when the issue could have been avoided completely but it was a good learning experience that I am more than happy to share.
Have fun and happy troubleshooting.