Sunday 18 January 2015

Unable to Update/Uninstall/Repair SharePoint binaries. Find out if files are missing from 'C:\Windows\Installer'

I guess that some of the things I am going to tell you in this post are well known for some of you that has been involved in Windows platform maintenance in recent years.
However this can happen, it can happen in your SharePoint environment, can happen in Production and you should be prepared for it.
So, last week I was helping a colleague of mine that has an issue in one of our customers production SharePoint 2013 servers. The farm has several language packs installed, but for one of the servers it was showing that this language packs are not installed. There was no chance this server to be missing binaries. We checked the "Product Version Job" timer job. This job by default runs every night on each SharePoint server and it should check the installed products and versions. If there is server where some binaries are not installed or they are lower version you will see some red colored alerts like the one below. However, it is possible that this job is unable to get the product data, because it runs under the Farm account identity and it needs local administrator privileges to retrieve the product data. In this case you will see event log warnings like the one described in this WiKi, you can put the account in the local admin group, restart the Timer Service and run the job, but this is not recommended. You can run the job with different identity if you log in to the server with proper account and run the command Get-SPProduct -Local and you will be able to update the data for the products installed on the server, this can help if PSConfig is blocked because of "missing products".

Missing Language Pack

Unfortunately this did not helped us in this case. We decided to run PSConfig on the server in question and oh surprise it was blocked by missing products on the server.
Logically we came up to the idea to Uninstall/Repair and reinstall the language packs.
However we were unable to do so from control panel nor by running the installation packages.
This was really strange, then we figure out that there might be some files missing from 'C:\Windows\Installer'.
This folder stores the Windows Installer Cache. The folder is hidden by default and it contains some very important files for every software that was deployed on the system using Windows Installer technology and this are all products and updates that are coming from Microsoft. The files are unique for every machine and they are not interchangeable. If you are not aware for the importance of this files it is possible to think that this are some temporary files that can be deleted. 
You will be seriously wrong! If there is missing files from this folder it will block the Update/Uninstall/Repair of the product  and your server is in "out of support" state. The guidance from Microsoft for such cases is very general. You need Machine Rebuild.
Apparently something has happened on our server. The support and maintenance of the Windows platform for this customer is delegated to a different company, so it is not possible someone from us to delete/remove/rename some of the files in that folder.
This situation is described in KB2667628, there you will find guidance for a very nice Microsoft Support Diagnostics tool called Windows Installer Cache Diagnostic. Run the tool on the server and it will generate a CAB file that contains nice reports on the installer cache. If there are no missing files you will see this:

No Windows Installer Cache Issue

If you have missing files you will see something like this for every missing files. Notice the bolded table row Action Required, it says "This .msp appears to be missing. Machine needs to be rebuilt."

Missing files from Windows Installer Cache

With this reports you have a strong weapon against the platform support teams :). 
However you will need to rebuild this server. I am not sure if restoring the files from backup will be supported. If you have not experienced such issues I think that it is good idea to run the diagnostic tool to check all your servers, it is fairly fast and simple.

6 comments:

  1. Good, I have similar issue at one of my customers and never though of checking the Installer folder... Will do that. However functionalities are working well, so I don't think we're going for the rebuild :)

    ReplyDelete
  2. Glad you find it useful :). The functionality is okay, but issues like this on single server from the farm can block updates.

    ReplyDelete
  3. You can the PowerShell script “Restore-InstallerFiles.ps1” which is capable to scan for the missing files, and to restore them from different sources such as local folder, shared folder or another machine which can be accessed from the target machine.
    http://ahmad.jempress.com/post/restore-the-missing-windows-installer-cache-files

    https://gallery.technet.microsoft.com/scriptcenter/Restore-the-Missing-d11de3a1

    ReplyDelete
    Replies
    1. Hi Ahmad,

      Nice script! But I think that this is not not supported/official way to do it.

      Delete
  4. Hi,
    Thank you very much for your reply.
    Why do you think it's not supported or official?
    It totally restores the typical missing files under the correct names on the target machine from other healthy machines instead wasting time and effort of rebuilding the server. What's wrong in this technique :)?

    ReplyDelete
    Replies
    1. Hi Ahmad,

      This is what KB2667628 is saying. As said your script is nice and clever and maybe it will save you if you have no other choice, but is it recognized as supported option by Microsoft?

      Delete