User Profile Incremental Synchronization Timer Job - Access is denied Error

I stumbled on this issue in one of our customer's SharePoint 2013 environment and I came across many forum posts and article, but did not found the complete solution, so I decided to share it with you.

The background story is as follows. We have a SharePoint Server 2013 hosted on Windows Server 2012 environment that is using User Profile Synchronization(leveraging FIM). We have one User Profile Service application.

The User Profile Synchronization service is running on the server where the Central Administration is hosted.

So far so good, we also have the User Profile Service running on two servers, in order to have a redundancy for this service, for short I will call this servers APP01 (CA,Sync Service, User Profile Service) and APP02(User Profile Service). The User Profile Synchronization service is successfully started and running on APP01.

The issue comes when you want to go and implement the principle of least privilege. This will require to remove the Farm account from the local Administrators group on the server where the User profile Synchronization is running (FIM). You need the Farm Account to be Local Admin in order to successfully start User Profile Synchronization.

You remove the the Farm account from the local Administration group.

Time goes by and you start to receive an error Event ID: 6398 on APP02 with following description:

---

The Execute method of job definition Microsoft.Office.Server.UserProfiles.UserProfileImportJob (ID 7acb171d-5d7d-4601-ba08-85f24c0969b4) threw an exception.More information is included below.

Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

---

This description tells us that the "User Profile Incremental Synchronization Timer Job" timer job is failing on APP02. To be sure you can use Get-SPTimerJob and for Identity parameter to use the GUID from the error message. This is the Timer Job responsible for the User Profile Synchronization. You check the Timer Job history for this job and it is running on APP02 and it is Failing.

It is possible to see this job to fail every minute one day it can run on APP02 and next run to be on APP01, or to see ongoing synchronization in the Central Administration, but when you open the miisclient.exe you see that there is no synchronization operations running, this however may prevent new synchronisations to start.

But User Profile Synchronization service is running on APP01 not on APP02!

Now all over the forums, folks are posting question like "The Synchronization timer job is running on the wrong server and it is failing". Actually this is not true!

Remember that we are running User Profile Service instance on both APP01 and APP02. The "User Profile Incremental Synchronization Timer Job" can run on both servers and it should run in success.

So the timer job is executed by the Timer Service and the Timer Service is running under the Farm account as the Sync Service. From the event log error it is obvious that the Farm account needs some permissions in order to execute the job successfully. When the account is in the local Administration group it "have it all".

The First thing to check is following MS Article. There is a section called "Remove unnecessary permissions". In this article it is explained how to grant Remote Enable permission over the Microsoft FIM 2010 WMI namespace MicrosoftIdentityIntegrationServer (on the server running FIM). This permission is needed for the Farm account to be able to run the job successfully from server that is not running User Profile Synchronization Service (FIM). Remember to do IISRESET if you are running the Sync. service on a server that is also hosting Central Administration.
Of course I did the procedure above, but does not worked out!
Apparently the job needs to interact with the Microsoft FIM 2010 WMI namespace. I am not sure for this but I think that the timer job is invoking Execute method on MIIS_ManagementAgent class.
Out of curiosity below you can see how the MicrosoftIdentityIntegrationServer (MIIS) WMI namespace looks like. I have based my User Profile Synchronization reporting script on MIIS WMI.

Now under the hood WMI is using DCOM, so in order a non-Administrator account to interact with WMI remotely it needs permissions over DCOM on the server where the Sync service is running at.

The Second things to check/change are the permissions over DCOM server on the machine that is running User Profile Synchronization Service. The Farm account needs to have Remote Access, Remote Launch and Remote Activation permissions. To do this use the article  Securing a Remote WMI Connection . Now this is a bit tricky and confusing for me. There is a simple way to check if the Farm account has remote access to WMI. By using below PowerShell command with farm account credentials from remote computer you should be able to retrieve the synchronization operations history.

---

Get-WmiObject -Class MIIS_RunHistory -Namespace root/MicrosoftIdentityIntegrationServer -ComputerName <YourSyncServer> -Credential <Domain>\<FarmAccount>

---

If there is an issue with the permissions you will receive some Access Denied error like this.

The DCOM permissions can be tricky after doing a change there close and open new PowerShell console and test again the remote WMI with the script above. If you still have Access Denied error locate Windows Management and Instrumentation DCOM application and give needed permissions to the Farm Account.

As you can see there are lots of moving  parts Timer Jobs, WMI, DCOM and so on. But there are countless reason for fail. Use the PowerShell line above. If there is an issue with the WMI remoting it will show it. There may be firewall issue it will show you something like "RPC service unavailable" error. If you do not receive a result from the remote WMI test you have some issue.

This is how I solved the issue. I am hoping that it is helpful for you!

Start User Profile Synchronization with PowerShell [Tip]

This is something new I learned and decided to share it in this short tip.
So I was trying to help in a TechNet Forum question. The user was asking how to start Incremental User Profile Synchronization with PowerShell because she needed it to run two times a day. Now when you manually star Full or Incremental sync. a timer job is started named "User Profile Service Application - User Profile Incremental Synchronization". If you start this job manually from the UI or in PowerShell it will trigger Incremental synchronization. If you want to set the schedule for the Incremental sync. you go to "Configure Synchronization Timer Job" in the User Profile Service Application Management page. However you cannot set the job to run twice a day. Here comes the PowerShell.
Thanks to Trevor Seward I learned this alternative method to start User Profile Synchronization in PowerShell, it will let you chose which type the sync. should be (Incremental or Full) by specifying $true or $false argument for the StartImport method. Here are the snippets for starting Full or Incremental synchronization.

---

$UPS= Get-SPServiceApplication | Where { $_.DisplayName -eq "<UPSA_DisplayName>"}
$UPS.StartImport($true)

## To Start Incremental Sync. use $false as argument for StartImport method

$UPS= Get-SPServiceApplication | Where { $_.DisplayName -eq "<UPSA_DisplayName>"}
$UPS.StartImport($false)

---

At the end of the day it is just starting the timer job I mentioned above, but it may come handy for some powershell scripts.

Get User Profile Synchronization connections in PowersShell and Advanced SharePoint functions in PowerShell

Maybe you have noticed that in SharePoint 2013 and SharePoint 2010 SP1 we have two cmdlets about Profile Sync. connections Add-SPProfileSyncConnection and Remove-SPProfileSyncConnection.
But against the PowerShell logic we have no Get-SPProfileSyncConnection cmdlet to list the current connections. In this post I am going to show you a short PowerShell function that I wrote that does this and going to explain some of the good practices when you develop advanced PowerShell functions for SharePoint. You can see the function below, it is really short.

---

function Get-SPProfileSyncConnection
{
 [CmdletBinding(DefaultParameterSetName="PSet1")]

Param(
    [parameter(Mandatory=$true,
        ParameterSetName="PSet1",
        ValueFromPipeline=$true)]
    [Microsoft.SharePoint.PowerShell.SPServiceApplicationPipeBind]$UPA,
    [parameter(Mandatory=$true,
        ParameterSetName="PSet2",
        ValueFromPipeline=$false)]
    [System.String]$UPAName
)
PROCESS{
    switch ($PsCmdlet.ParameterSetName) 
    {
        "PSet1" { $UPApp = $UPA.Read(); break} 
        "PSet2" { $UPApp = Get-SPServiceApplication | Where {$_.Name -eq $UPAName}; break} 
    }
    $context = [Microsoft.SharePoint.SPServiceContext]::GetContext($UPApp.ServiceApplicationProxyGroup,[Microsoft.SharePoint.SPSiteSubscriptionIdentifier]::Default)
    $upcMan = New-Object Microsoft.Office.Server.UserProfiles.UserProfileConfigManager($context)
    
    Write-Output $upcMan.ConnectionManager
    
}
}

---

Because of the specifics in User Profile Service Application(UPA) the function must run under the Farm account identity.
Now lets have a look on what is going on. In all cases we need to have a User Profile Service Application from where we can get the sync. connection(s) or other service application with proxy in the same proxy group as our target UPA, so we can get the Service context , this should be defined by the user as parameter. At the parameter section of the function we have defined two parameters from two parameter sets, UPA from parameter set PSet1 and UPAName from parameter set PSet2. The script can get the desired User Profile Service Application from both parameters. If the user have supplied a value for parameter UPAName, the script will decide how to assign value to the UPApp that represent our User Profile Service Application object. If the UPAName is present that means that PSet2 is used. This logic is achieved with the PowerShell switch statement. If the PSet2 is used the script will assign value of Service Application where the Name property is equal to the UPAName supplied by the user.
The interesting part is when UPA parameter is used (PSet1). This parameter is set to receive value from pipeline. This is achieved by parameter argument ValueFromPipeline=$true , it means that this parameter will take the value of object from the pipeline if it is the same type as the parameter (bind byValue). The type of the parameter is not Microsoft.Office.Server.Administration.UserProfileApplication
for example, it is Microsoft.SharePoint.PowerShell.SPServiceApplicationPipeBind .
In the native SharePoint cmdlets parameter 'Identity' can be represented by multiple ways. For examaple if you want to get SPSite with the cmdlet Get-SPSite. You can supply URL, GUID or entire SPSite object. If you need to get all sites from a web application, you will use "Get-SPSite -WebApplication"  and for the WebApplication parameter you can supply URL,ID, entire SPWebApplication object or use pipeline, because the expected type is SPWebApplicationPipeBind. In order to prevent the need for multiple parameters and multiple parameter sets Microsoft presented the PipeBind objects. After you supply valid object and valid object is object that can be converted by the PipeBind (GUID,String,..etc) you are converting it to the original object by calling Read() method. You can have a look at the available PipeBind classes in this MSDN Library .
You may have issues with the above function in advanced scenario with multiple proxies/proxy groups or with partitioned (multi-tenant) environment(haven't tested it yet), maybe it is better to take the ServiceContext from SPSite. I think it is a good example for creating a SharePoint cmdlet and it is working well in my case .

User Profile Synchronization reporting script

User Profile Synchronization Service in SharePoint have always been a sensitive topic. For the full featured User SharePoint Profile Synchronization we need to have this service instance running. In SharePoint 2013 we have lightweight "mode" called SharePoint Active Directory Import that is leveraging the System.DirectoryServices .NET classes and when we use this mode we don't need User Profile Synchronization Service running .
Under the hood SharePoint relies on a version of Forefront Identity Management to do the profile synchronization. Unfortunately sometimes the headache does not stop when you manage to properly start/restart the service instance. The profile information synchronization issues can be very visible for the users.
So we have a customer that is using the classic SharePoint Profile Synchronization with SharePoint 2013 to sync from AD.
The customer have a large number of domains that are hosted on remote sites around the globe. There isn't  something like Read-Only DCs  to be closer to the SharePoint and that can replicate from the remote domains and the SharePoint have to contact the remote DCs to do the profile synchronization.
Because of the distance between the SharePoint and the domains that should sync from, we often end up with unsuccessful profile sync. operations for some of the domains.
In this case we need to find a way to be notified for such issue on time and take some actions. It would be nice if SherePoint provide something like the crawl log  where you can see the status of the operation.
We can see the status if we go to the server where the User Profile Synchronization Service is running and open "C:\Program Files\Microsoft Office Servers\15.0\Synchronization Service\UIShell\miisclient.exe".

Some of the events can also be logged in the Event Viewer of the server.

I did some research and found that there is WMI Namespace for Microsoft Identity Integration Server (MIIS), there are some promising classes like ManagementAgent and RunHistory, also the WMI query can be executed remotely. For each Synchronization Connection in the User Profile Application we have a FIM Management Agent (MA) that link specific connected data source to FIM. Management Agent is responsible for moving the data source to FIM. For the usual setup where SharePoint is synchronizing profile information from AD DS the naming of the agent is MOSSAD-*CONNECTION_NAME*. For example, if we have synchronization connection named CONTOSO.COM, the name of the MA will be MOSSAD-CONTOSO.COM.
So I wrote a script that can do a report on the sync. operations for a given time. The report is in a form of CSV file and it can be mailed like an HTML table. Let say that you have scheduled an Incremental Synchronization  at 1:00 AM and usually entire process finishes for 1 hour and you want to have the status of the operation when it finish. You can schedule the script to run at 2:30 AM with this parameters:

---

.\Fim-SyncReport.ps1 -ServerName 'spfimsync.contoso.net' -Hours 3 `

-FromMailAdress 'spupsreporter@contoso.com' `

-MailAdress 'aaronp@contoso.com','sp-admin-team@contoso.com' `

-SMTPServer 'mailer.contoso.com' `

-ReportLoction '\\fileserver\fimnightreport'

---

This will do the WMI query on the server where the User Profile Synchronization Service instance is running, will save the report for the last 3 hours on a fileshare and will send the report via mail. The report will contain the MA name, status, synchronization profile, when the operation started, the number of sync. errors, discovery errors and retry errors. The status field is very important the script will take all the operations that are not with status "success", the status can be very helpful when you troubleshoot synchronization issue. Complete list of the RunStatus return string can be found in this MSDN Article . If all operations are in success no report will be generated and no mail will be send.
The report looks like this when it is sent via mail :

The key function from the script is :

---

function Get-FimMARuns
{
 [CmdletBinding()]
Param(
    [parameter(Mandatory=$true)]
    [string]$MaName,
    [parameter(Mandatory=$true)]
    [string]$Hours,
    [parameter(Mandatory=$true)]
    [string]$ComputerName
)
Process
{
    $timeSpan = New-TimeSpan -Hours $Hours
    $nowUTC = (Get-Date).ToUniversalTime()
    $timeToStart = $nowUTC.Add(-$timeSpan)
    $filter = ("MaName = '{0}'" -F $MaName)
    $allHistory = Get-WmiObject -ComputerName $ComputerName `

                  -Class MIIS_RunHistory `

                  -Namespace root/MicrosoftIdentityIntegrationServer `

                  -Filter $filter
    ForEach ($history in $allHistory)
    {
        $startTimeinDateTime = $history.RunStartTime | Get-Date
        if ($startTimeinDateTime -gt $timeToStart)
        {
            Write-Output $history
        }
    }
}
}

---

In this function we are utilizing the MIIS_RunHistory class to get all the runs for given MA name (we get all MAs staring with 'MOSS' in different function ). Then we are calculating a filter based on the start time so we can get the one that fit to the timeframe for our report.
When we have all operation that we need we can read the information from them. The class return objects for every run (run history entry). We can see the full information with this method:

---

[xml]$asXML = $faultyOp.RunDetails().ReturnValue

---

The history entry has method RunDetails that has property ReturnValue . When we get this property we are receiving an XML with all details from the run, then we can get whatever value we want. You can get the script from the link below and edit it (specific filtering by MA,Sync Profile, Status ...etc.) as you find useful in your case. Just remove the underscore from the file extension, this is a zip file with the script and mail body template.

Download From: TechNet Gallery