Remove Item from the Search Index in SharePoint using PowerShell

While working at a client site this week, we encountered an issue where they had duplicate entries for the same Document ID, meaning that when they tried to access that document via its DocId Url, they were presented with a search page showing the two items instead. Without going into the details as to what was actually causing this issue to surface, the easy fix for this was to go and remove one of the duplicate from the Search Index. Through this interface, this can be done as follow:

1 – Navigate to the Search Service Application Page in Central Administration.

Search Service Application page in Central Administration

Search Service Application page in Central Administration

2 – Click on Crawl Log in the left navigation.

SharePoint Search Crawl Log

SharePoint Search Crawl Log

3 – In the top navigation, click on URL View.

URL View in Search Crawl Log

URL View in Search Crawl Log

4 – Search for the URL (or pattern) you want to remove.

Search Crawl Log for URL

Search Crawl Log for URL

5 – Find the item in the list and click on it to expand the contextual menu.

Crawl Log Entry Options

Crawl Log Entry Options

6 – From the list of options, select Remove the item from the Index.

The PowerShell Equivalent

In order for my clients to automate this process for the URLs they want to remove from the Search Index, I created a quick cmdlet called Remove-SPEnterpriseSearchURLFromIndex which simply takes in a URL pattern. Upon detecting URL entries in the Crawl Log that match the provided URL, the cmdlet will prompt the user to remove the item from the index or not.

[CmdletBinding]
function Remove-SPEnterpriseSearchURLFromIndex
{
    param( 
        [System.String]
        $Url = "Default"
    )

    $ssas = Get-SPEnterpriseSearchServiceApplication

    foreach($ssa in $ssas)
    {
        $cl = New-Object Microsoft.Office.Server.Search.Administration.CrawlLog $ssa
        $logEntries = $cl.GetCrawledUrls($false,100,$Url,$true,-1,-1,-1,[System.DateTime]::MinValue, [System.DateTime]::MaxValue)

        foreach($logEntry in $logEntries.Rows)
        {
            Write-Host "You are about to remove " -NoNewline
            Write-Host $logEntry.FullUrl -ForegroundColor Green

            do{
                $deletionAnswer = Read-Host "Do you confirm the deletion (y/n)"
            }while($deletionAnswer.ToLower() -ne 'n' -and $deletionAnswer.ToLower() -ne 'y')

            switch($deletionAnswer)
            {
                'y'
                {
                    $catch = $cl.RemoveDocumentFromSearchResults($logEntry.FullUrl)
                    if($catch)
                    {
                        Write-Host "Deleted" -ForegroundColor Yellow
                    }
                    else
                    {
                        Write-Host "Could not delete the item" -ForegroundColor Red
                    }
                }
                'n'
                {
                    break
                }
            }
        }
    }
}

Easily Retrieve IIS ApplicationPool Credentials

This is going to be a very short Blog Post, not to say a brain dump, but if you ever need to retrieve the credentials used in a Specific ApplicationPool in IIS, you can use the following snippet of PowerShell code to do so:

$appPools = Get-WebConfiguration -Filter '/system.applicationHost/applicationPools/add'

foreach($appPool in $appPools)
{
    if($appPool.ProcessModel.identityType -eq "SpecificUser")
    {
        Write-Host $appPool.Name -ForegroundColor Green -NoNewline
        Write-Host " -"$appPool.ProcessModel.UserName"="$appPool.ProcessModel.Password
    }
}

This will output something similar to the following:

Now, I am sure some of you are probably freaking out by now, realizing that people with access to the server can easily retrieve the credentials from IIS app pools that are running as a specific user. Let me assure you that there is nothing magic about the PowerShell code above. When you specify credentials for an IIS application pool, after verifying against Active Directory that the provided credentials are valid, IIS will go and actually encrypt and store those credentials locally. Using the Get-WebConfiguration cmdlet allows you to retrieve and decrypt those.
So yes, the moment a user has access to run the PowerShell cmdlet on the server, he is also able to retrieved stored credentials for users running the app pool.

Patching your SharePoint Farm with PowerShell DSC

Patching a SharePoint 2013/2016 farm with the help of PowerShell Desired State Configuration (DSC) is a common ask I get from customers almost every single time I deliver a DSC engagement. As part of the SharePointDSC module, we offer two main resources to help you automate the patching process for your farm: SPProductUpdate and SPConfigWizard.

  • SPProductUpdate resource is responsible for installing the patch’s bits onto a server in the farm. It is the equivalent of manually running the installer for a Cummulative/Public update onto the given server. It is very important to note that declaring a resource block of this type in your DSC configuration ONLY installs it on the given node. You need to make sure that this resource block gets defined on every server in your farm to make sure all servers have the bits installed on them. This resource allows you to speed up the installation process on the various nodes by automatically shutting down the various Search Services that normally slow down the installation process. In order to shutdown those services during the installation, you need to specify the ShutdownServices parameter to $true
  • SPConfigWizard on the other hand, is the equivalent of running PSConfig on a given server. It is responsible for committing the installed bits into the configuration database to finalize the farm’s upgrade process. Just like the SPProductUpdate resource, this one needs to be defined against every server in the farm.

Patching Process

In this article, I will demo the process of patching a SharePoint 2016 farm, however the process is the same if you wish to patch a SharePoint 2013 farm. To properly demonstrate the patching process, I will be using a SharePoint 2016 RTM farm, and will be patching it to the September 2017 Public Update, which includes the Feature Pack 2 bits.

  1. The first step is to go an download the SharePoint 2016 – September 2017 Public Update from the web. Decide where you wish to save it. My recommendation is to put it on a Shared Network Location that all servers will be able to access. However, you need to understand the implications of running the Update installer from a Network location using DSC, because your installation process may get stuck due to the User Account Control protection. I’ve put together a short article that lists the most common gotchas for when using DSC and solutions to them. In my case, the file will be put under \\DSC-Share\SP16-Sept16PU\sts2016-kb4011127-fullfile-x64-glb.exe
  2. The second step is to add the DSC Resource blocks into your PowerShell configuration script. The recommendation here is for you to put them right after the SharePoint binaries have been installed via SPInstall, and right before your are actually attempting to have the server join the farm via SPFarm. This would also be the recommendation as far as location within the script for where to install the Language Packs. That is if you are using DSC to install your farm from the ground up.

    For this article however, I am going to demonstrate the case where you already have a SharePoint 2016 Farm built and all you are trying to do in apply a Public Update on it via DSC. The following is the complete script I will be using to achieve this:

    Configuration SP2016September2017PU
    {
        Import-DscResource -ModuleName "SharePointDSC" -ModuleVersion "1.9.0.0"
        $CredsspFarm = Get-Credential -Message "Farm Account"
    
        Node $AllNodes.NodeName
        {
            SPProductUpdate Sept2017PU
            {
                SetupFile = "\\DSC-Share\SP16-Sept16PU\sts2016-kb4011127-fullfile-x64-glb.exe"
                ShutdownServices = $true
                Ensure = "Present"
                PsDscRunAscredential = $CredsspFarm
            }
    
            SPConfigWizard PSConfig
            {
                Ensure = "Present"
                PsDscRunAscredential = $CredsspFarm
            }
        }
    }
    
    $ConfigurationData = @{
        AllNodes = @(
            @{
                NodeName = "SPWFE1"
                PSDscAllowPlainTextPassword = $true;
                PSDscAllowDomainUser = $true;
            },
            @{
                NodeName = "SPWFE2"
                PSDscAllowPlainTextPassword = $true;
                PSDscAllowDomainUser = $true;
            },
            @{
                NodeName = "SPAPP1"
                PSDscAllowPlainTextPassword = $true;
                PSDscAllowDomainUser = $true;
            }
        )
     }
    
    SP2016September2017PU -ConfigurationData $ConfigurationData
    
  3. Initiate the Start-DSCConfiguration SP2016September2017PU -Wait -Verbose -Force cmdlet to initiate the configuration of the servers in the farm.

That was easy enough wasn’t it? Now, whenever a new update comes in that you wish to apply to your farm, simply update the SetupFile parameter’s value to the new PU file. DO NOT ever include more than one SPProduct update block for a given server in your DSC configuration. Updates in SharePoint are cumulative, meaning that if your goal is to update a farm to the October 2017 PU, you don’t need to install the September 2017 PU first.

ReverseDSC for Files and Registry

As mentioned in multiple blog article I wrote in the past, ReverseDSC is technology agnostic, meaning that it can reverse engineer any component (Windows, Linux, MacOS, etc.) as long as there is a DSC module that exists for it. What we refer to as Orchestrator Scripts is what is technology specific and does specific operations for a specific component. You’ve seen me blog mainly about the SharePointDSC.Reverse Orchestrator Script lately, and show users how they can extract configuration of existing SharePoint Farms and replicate them elsewhere. There are however several other Orchestrator Scripts being developed at the moment including the SQLServer.Reverse one and the PSDesiredStateConfiguration.Reverse one that allows you to replicate file and registry structured on other machines using DSC.

In this article, I want to demo how the PSDesiredStateConfiguration.Reverse Orchestrator Script can be used to extract a DSC Configuration script. While the Orchestrator Script allows you to extract numerous components of an environment (Windows Features, Logs, Archives, Users, etc.) we will only focus on Files and Registry Entries within this article.

How to Install

Just like all other Orchestrator Scripts we are currently working on, the PSDesiredStateConfiguration.Reverse script is made available in the PowerShell Gallery. You can download and install it from any machine that has internet connectivity by executing the following lines of PowerShell (needs WMF 5.0).

Install-Script PSDesiredStateConfiguration.Reverse -Force

If you don’t have internet connectivity on your machine or if you don’t have access to a machine with WMF 5.0+ on it, then you will need to manually download the script for the GitHub Repository from a machine that has internet connectivity and copy it manually onto the machine. By default, the PowerShell code above downloads the script under c:\Program Files\WindowsPowerShell\Scripts. I would therefore recommend copying the script under that folder (create the Scripts folder if it doesn’t already exist).

ReverseDSC for Files

In this section we will cover the process of using the PSDesiredStateConfiguration.Reverse Orchestrator Script to extract a file structure from an existing environment as a DSC configuration script that we can use to recreate that exact file structure onto another environment. The challenge here is that in order for this file structure to be recreated on a new computer, we need to have access to the files from a central location. Let’s take the following example where I wish to extract the File structure in the path C:\MyCustomApplication from my machine.

MyCustomApplication file Structure

MyCustomApplication file Structure

The files are currently local to the machine. If we were to simply extract the file structure as a DSC Configuration Script, it would result in DSC Resources blocks similar to the following:

File 469e3fe0-b1c6-4b70-bc69-725dd2bc3739
{
    DestinationPath = 'C:\MyCustomApplication';
    Type = 'Directory';
    Recurse = $True;
    SourcePath = 'C:\MyCustomApplication';
    Ensure = 'Present';
}

This code tells DSC on the new machine to copy all files that exist under SourcePath (C:\MyCustomApplication) under the DestinationPath (C:\MyCustomApplication). Now, let’s forget that both paths are the same for one moment and try to understand the issue here. The new machine we are trying to configure with DSC knows nothing about the one we ran ReverseDSC on. It doesn’t have access to the original file repository where all the .dll, .chm, and .exe are located. Therefore we need to make sure these files we want to copy over our new machine are made available either on a network drive or with a Share from the machine we ran ReverseDSC on.

In order to solve this issue and to make it easy for users to extract a DSC Configuration script that can easily be leverage on other machines, the script accepts and array of HashTable values for the Folders parameter. The HashTable defines the following values:

Key Description
Source Root URL of the folder to scan on the local machine
SharedSource Root URL of the Shared folder or network location where the files to be copied are located.

To go back to our example above, I will be using the following values when calling the script:

Key Value
Source C:\MyCustomApplication
SharedSource \\NetworkShare\MyCustomApplication

I will then be calling the PSDesiredStateConfiguration.Reverse.ps1 script as follow:

.\PSDesiredStateConfiguration.Reverse.ps1 -Folders @{Source="C:\MyCustomApplication";SharedSource="\\NetworkShare\MyCustomApplication"}

Doing so will then result in the DSC Resource blocks having the DestinationPath match what was on the original server we ran ReverseDSC onto, and the SourcePath pointing to the shared location as follow:

File 617c1dc1-a7f6-48b6-bd69-24b4cefe2d21
{
    DestinationPath = 'C:\MyCustomApplication\Bin\Core.dll';
    Type = 'File';
    SourcePath = '\\NetworkShare\MyCustomApplication\Bin\Core.dll';
    Ensure = 'Present';
}

Upon compiling the DSC Configuration Script into a MOF file and initiating the DSC configuration on the new machine, the machine will automatically copy the files from the shared location to a local path matching the original machine on which we ran ReverseDSC onto.

It is also important to note that as mentioned above, the Folders parameter is actually an array of HashTable, meaning that multiple values could be passed to it. Assuming you had two folders you wanted to extract, you code do it as follow:

.\PSDesiredStateConfiguration.Reverse.ps1 -Folders @(@{Source="C:\Folder1";SharedSource="\\NetworkShare\Folder1"}, @{Source="C:\Folder2";SharedSource="\\NetworkShare\Folder2"})

ReverseDSC for Registry Keys

The process to use the PSDesiredStateConfiguration.Reverse Orchestrator Script against some Registry Keys is very similar to the Files process described above. The Script accepts a parameter named RegistryPaths that is an array of string that contains the paths to the Parent Registry key you wish to extract. The Registry Key paths can be entered either as the long or short form (e.g. HKLM:\System or HKEY_LOCAL_MACHINE\System).

Assuming I wanted to extract everything under the following Registry Paths:

  • HKLM:\System\CurrentControlSet\Control\Lsa
  • HKLM:\Software\dotnet

All I would need to do is call the script as follow:

.\PSDesiredStateConfiguration.Reverse.ps1 -RegistryPaths @("HKLM:\System\CurrentControlSet\Control\Lsa", "HKLM:\Software\dotnet")

That would then produce an output having DSC Resource Blocks similar to the following:

Registry 23e6bc1f-2f88-4b14-afbe-dddd8e3aa3e6
{
    ValueName = "NoLmHash";
    Key = "HKLM:\System\CurrentControlSet\Control\Lsa";
    Ensure = "Present";
    ValueType = "DWord";
    ValueData = @("1");
}

Reverse Engineer an Existing SharePoint Farm using PowerShell DSC

PowerShell Desired State Configuration (DSC) is a technology that allows you to define a configuration for an environment, and ensure it automatically gets configured tat way. It can also help you monitor the delta between the current state of the environment and what the defined configuration (a.k.a. Desired State) is. So as an example in the case of SharePoint, if you were to specify that you needed to register a Managed Account for user sp_services, and that someone was to delete that account, PowerShell DSC would automatically report the discrepancy in the logs or attempt to automatically recreated it based on how you configured your environment (Configuration Mode).

The SharePoint module for DSC is great to build new environments from the ground up. Simply specify what the Desired State should be (Web Applications, Site Collections, Service Applications, etc.) and let DSC do its job and automatically configure it. It acts as some kind of abstraction layer for the user, where they only need to define resources and their various settings for DSC to create and configure everything for them in the background. They don’t need to know the cmdlets to use, know what the objects’ properties are, and even less understand the object model behind it all. They only need to know what the DSC resource accepts.

Existing Environments

In many cases however, clients who want to get into DSC already have investments in SharePoint, where they have an existing SharePoint 2013 or 2016 farm that they would like to replicate somewhere else with DSC. Writing a DSC configuration script from scratch that represents their exact environment is a complex job and often results in the customer writing tens of thousands of lines of code. This is where the ReverseDSC orchestrator for SharePoint, called SharePointDSC.Reverse comes into play. SharePointDSC.Reverse is a PowerShell script (.ps1) that you execute against an existing SharePoint 2013 or 2016 farm to extract its PowerShell Desired State Configuration. The script will scan every component of the farm, down to the SPWeb level. This means that everything that can be defined in Central Administration, with the addition of Sites can be extracted with the SharePointDSC.Reverse. The output will be another .ps1 script that will contain the DSC configuration of your farm. That script can then be executed to compile a .MOF file that can be used to replicate your SharePoint environment elsewhere.

Other Scenarios

The ReverseDSC Orchestrator for SharePoint can also be used for several scenarios other than replicating a farm. It can be used to:

  • Extract a SharePoint 2013 Farm Configuration and replicate it as a SharePoint 2016 one;
  • Compare deltas between two extractions to see what changed between two point in time;
  • Create standalone copies of a Farm for developers;
  • Move an on-premises SharePoint Farm into Azure Infrastructure-as-a-Service;
  • On-board an existing farm onto PowerShell DSC to monitor it and ensure it remains in its desired state.

How to Use

The ReverseDSC Orchestrator for SharePoint is now officially available in the PowerShell Gallery. It can be installed, alongside all of its dependencies, by executing the following PowerShell one-liner from a machine that has internet connectivity:

Install-Script SharePointDSC.Reverse -Force 

Note that this has to be done on only one server in the farm; you don’t need to install it on all servers. If your SharePoint server doesn’t have internet connectivity, simply run the above line of PowerShell from a machine that does have internet connectivity, and manually copy the ReverseDSC, SharePointDSC from the PowerShell modules folder (C:\Program Files\WindowsPowerShell\Modules), and the SharePointDSC.Reverse.ps1 script from the PowerShell script folder (C:\Program Files\WindowsPowerShell\Scripts) onto the destination server.

Additional Resources

To learn more about the ReverseDSC Orchestrator for SharePoint, you can watch my Microsoft Virtual Academy session on the topic at the following URL: https://mva.microsoft.com/en-US/training-courses/sharepoint-automation-with-dsc-17843