Remove Item from the Search Index in SharePoint using PowerShell

Nik Charlebois Tutorial 287 Leave a comment

While working at a client site this week, we encountered an issue where they had duplicate entries for the same Document ID, meaning that when they tried to access that document via its DocId Url, they were presented with a search page showing the two items instead. Without going into the details as to what was actually causing this issue to surface, the easy fix for this was to go and remove one of the duplicate from the Search Index. Through this interface, this can be done as follow:

1 – Navigate to the Search Service Application Page in Central Administration.

Search Service Application page in Central Administration

Search Service Application page in Central Administration

2 – Click on Crawl Log in the left navigation.

SharePoint Search Crawl Log

SharePoint Search Crawl Log

3 – In the top navigation, click on URL View.

URL View in Search Crawl Log

URL View in Search Crawl Log

4 – Search for the URL (or pattern) you want to remove.

Search Crawl Log for URL

Search Crawl Log for URL

5 – Find the item in the list and click on it to expand the contextual menu.

Crawl Log Entry Options

Crawl Log Entry Options

6 – From the list of options, select Remove the item from the Index.

The PowerShell Equivalent

In order for my clients to automate this process for the URLs they want to remove from the Search Index, I created a quick cmdlet called Remove-SPEnterpriseSearchURLFromIndex which simply takes in a URL pattern. Upon detecting URL entries in the Crawl Log that match the provided URL, the cmdlet will prompt the user to remove the item from the index or not.

[CmdletBinding]
function Remove-SPEnterpriseSearchURLFromIndex
{
    param( 
        [System.String]
        $Url = "Default"
    )

    $ssas = Get-SPEnterpriseSearchServiceApplication

    foreach($ssa in $ssas)
    {
        $cl = New-Object Microsoft.Office.Server.Search.Administration.CrawlLog $ssa
        $logEntries = $cl.GetCrawledUrls($false,100,$Url,$true,-1,-1,-1,[System.DateTime]::MinValue, [System.DateTime]::MaxValue)

        foreach($logEntry in $logEntries.Rows)
        {
            Write-Host "You are about to remove " -NoNewline
            Write-Host $logEntry.FullUrl -ForegroundColor Green

            do{
                $deletionAnswer = Read-Host "Do you confirm the deletion (y/n)"
            }while($deletionAnswer.ToLower() -ne 'n' -and $deletionAnswer.ToLower() -ne 'y')

            switch($deletionAnswer)
            {
                'y'
                {
                    $catch = $cl.RemoveDocumentFromSearchResults($logEntry.FullUrl)
                    if($catch)
                    {
                        Write-Host "Deleted" -ForegroundColor Yellow
                    }
                    else
                    {
                        Write-Host "Could not delete the item" -ForegroundColor Red
                    }
                }
                'n'
                {
                    break
                }
            }
        }
    }
}

Easily Retrieve IIS ApplicationPool Credentials

Nik Charlebois Tutorial 117 Leave a comment

This is going to be a very short Blog Post, not to say a brain dump, but if you ever need to retrieve the credentials used in a Specific ApplicationPool in IIS, you can use the following snippet of PowerShell code to do so:

$appPools = Get-WebConfiguration -Filter '/system.applicationHost/applicationPools/add'

foreach($appPool in $appPools)
{
    if($appPool.ProcessModel.identityType -eq "SpecificUser")
    {
        Write-Host $appPool.Name -ForegroundColor Green -NoNewline
        Write-Host " -"$appPool.ProcessModel.UserName"="$appPool.ProcessModel.Password
    }
}

This will output something similar to the following:

Now, I am sure some of you are probably freaking out by now, realizing that people with access to the server can easily retrieve the credentials from IIS app pools that are running as a specific user. Let me assure you that there is nothing magic about the PowerShell code above. When you specify credentials for an IIS application pool, after verifying against Active Directory that the provided credentials are valid, IIS will go and actually encrypt and store those credentials locally. Using the Get-WebConfiguration cmdlet allows you to retrieve and decrypt those.
So yes, the moment a user has access to run the PowerShell cmdlet on the server, he is also able to retrieved stored credentials for users running the app pool.

SharePoint DSC 1.9.0.0 is Released

Nik Charlebois Announcements 94 Leave a comment

The latest version of the SharePoint DSC module has been released this last Wednesday, October 5th. You can now obtain the latest bits of the module by running:

Install-Module SharePointDSC

If you wish to update the configurations on a given SharePoint farm to run these latest bits, make sure each .MOF file is first recompiled using the latest package and that the new module bits are copied to every server in the farm.

Here are some of the major changes that were introduced in this version:

  • SPServiceIdentity: A new DSC Resource has been introduced in the module to represent the managed account that is assigned to a service instance.

  • SPWebAppSiteUseAndDeletion: Fixed an issue with the scheduling where is was incorrectly setting the delays in weeks instead of in days.

  • SPWebAppGeneralSettings: Fixed an issue where web applications that weren’t assigned a TimeZone were incorrectly returning -1 instead of $null.

  • SPProductUpdate: Fixed an issue for updating a SharePoint 2013 farm were the resource was always complaining that the farm was missing the OSearch16 (SharePoint 2016 version).

  • Several other fixes related to the Get-TargetResource methods not properly returning values.

Along with the SharePointDSC 1.9.0.0 release, the SharePoint Orchestrator script for ReverseDSC, which allows you to extract the configuration out of an existing SharePoint 2013 or 2016 farm, has also been updated. To leanr more about what is new in the ReverseDSC Orchestrator Script, take a look at my previous blog post What’s New With SharePointDSC.Reverse 1.9.0.0

Patching your SharePoint Farm with PowerShell DSC

Nik Charlebois Tutorial 497 Leave a comment

Patching a SharePoint 2013/2016 farm with the help of PowerShell Desired State Configuration (DSC) is a common ask I get from customers almost every single time I deliver a DSC engagement. As part of the SharePointDSC module, we offer two main resources to help you automate the patching process for your farm: SPProductUpdate and SPConfigWizard.

  • SPProductUpdate resource is responsible for installing the patch’s bits onto a server in the farm. It is the equivalent of manually running the installer for a Cummulative/Public update onto the given server. It is very important to note that declaring a resource block of this type in your DSC configuration ONLY installs it on the given node. You need to make sure that this resource block gets defined on every server in your farm to make sure all servers have the bits installed on them. This resource allows you to speed up the installation process on the various nodes by automatically shutting down the various Search Services that normally slow down the installation process. In order to shutdown those services during the installation, you need to specify the ShutdownServices parameter to $true
  • SPConfigWizard on the other hand, is the equivalent of running PSConfig on a given server. It is responsible for committing the installed bits into the configuration database to finalize the farm’s upgrade process. Just like the SPProductUpdate resource, this one needs to be defined against every server in the farm.

Patching Process

In this article, I will demo the process of patching a SharePoint 2016 farm, however the process is the same if you wish to patch a SharePoint 2013 farm. To properly demonstrate the patching process, I will be using a SharePoint 2016 RTM farm, and will be patching it to the September 2017 Public Update, which includes the Feature Pack 2 bits.

  1. The first step is to go an download the SharePoint 2016 – September 2017 Public Update from the web. Decide where you wish to save it. My recommendation is to put it on a Shared Network Location that all servers will be able to access. However, you need to understand the implications of running the Update installer from a Network location using DSC, because your installation process may get stuck due to the User Account Control protection. I’ve put together a short article that lists the most common gotchas for when using DSC and solutions to them. In my case, the file will be put under \\DSC-Share\SP16-Sept16PU\sts2016-kb4011127-fullfile-x64-glb.exe

  2. The second step is to add the DSC Resource blocks into your PowerShell configuration script. The recommendation here is for you to put them right after the SharePoint binaries have been installed via SPInstall, and right before your are actually attempting to have the server join the farm via SPFarm. This would also be the recommendation as far as location within the script for where to install the Language Packs. That is if you are using DSC to install your farm from the ground up.

    For this article however, I am going to demonstrate the case where you already have a SharePoint 2016 Farm built and all you are trying to do in apply a Public Update on it via DSC. The following is the complete script I will be using to achieve this:

    Configuration SP2016September2017PU
{
    Import-DscResource -ModuleName "SharePointDSC" -ModuleVersion "1.9.0.0"
    $CredsspFarm = Get-Credential -Message "Farm Account"

    Node $AllNodes.NodeName
    {
        SPProductUpdate Sept2017PU
        {
            SetupFile = "\\DSC-Share\SP16-Sept16PU\sts2016-kb4011127-fullfile-x64-glb.exe"
            ShutdownServices = $true
            Ensure = "Present"
            PsDscRunAscredential = $CredsspFarm
        }

        SPConfigWizard PSConfig
        {
            Ensure = "Present"
            PsDscRunAscredential = $CredsspFarm
        }
    }
}

$ConfigurationData = @{
    AllNodes = @(
        @{
            NodeName = "SPWFE1"
            PSDscAllowPlainTextPassword = $true;
            PSDscAllowDomainUser = $true;
        },
        @{
            NodeName = "SPWFE2"
            PSDscAllowPlainTextPassword = $true;
            PSDscAllowDomainUser = $true;
        },
        @{
            NodeName = "SPAPP1"
            PSDscAllowPlainTextPassword = $true;
            PSDscAllowDomainUser = $true;
        }
    )
 }

SP2016September2017PU -ConfigurationData $ConfigurationData

  3. Initiate the Start-DSCConfiguration SP2016September2017PU -Wait -Verbose -Force cmdlet to initiate the configuration of the servers in the farm.

That was easy enough wasn’t it? Now, whenever a new update comes in that you wish to apply to your farm, simply update the SetupFile parameter’s value to the new PU file. DO NOT ever include more than one SPProduct update block for a given server in your DSC configuration. Updates in SharePoint are cumulative, meaning that if your goal is to update a farm to the October 2017 PU, you don’t need to install the September 2017 PU first.

SharePoint DSC Gotchas and Best Practices

Nik Charlebois Announcements 277 2 Comments

In the Microsoft Virtual Academy on SharePoint Automation with DSC my colleague Brian Farnhill and I recorded in June of 2017, we dedicated an entire module to SharePointDSC best practices. I thought I’d take a few minutes to put the items we discussed during that session on paper. This article will act as a list of the most frequent issues and problem encountered by people using SharePoint DSC. It will be updated as the tool evolves.

Frequent Issues

Installation of Prerequisites, Binaries or Language Pack is Stuck

Symptom:

This is one of the most frequent issue people encountered when starting off with SharePoint DSC. When running the configuration using Start-DSCConfiguration, or when in Pull mode, the installation seems to get stuck when trying to install components (prerequisites, binaries, or language pack). No errors are being thrown.

Most Likely Reason:

There is some level of User Account Control set on the installation file. In most cases this issues will happen when the installation is done from a Network Drive. Upon executing the installation of the component, windows triggers a prompt for the user to confirm the execution of an executable that was launched from an untrusted location, but because this was initiates from a background thread, it is impossible to ever approve that prompt and the installation gets stuck.

In order to test if this is your issue, launch a new PowerShell session, running as the user defined in the PsDscRunAsCredential parameter of the execution block. From there, run the following line of PowerShell. If there are any prompts that appear, that means that the installation is most likely stuck because of the User Account Control.

Start-Process -FilePath <Path to the Installer file> -Wait -PassThru

Solution:

The solution here is two fold. First off, we recommend you turn off the User Account Control feature for the user running the PsDscRunAsCredential installation block in your script. As a second step, launch Internet Explorer and go to Internet Explorer Settings. From there, go to the Security tab, click on the Local Intranet Zone, then click on the Sites button. In the prompt, click on the Advanced button, this will open the window that let’s you specify addresses to identify as being part of the Local Intranet Zone. In that window, type in the location of the Network share (or other common location) where the installer file is located, and click on Add.

Using a completely DSC based solution, you could use the xSystemSecurity module to disable the User Account Control prompt as follow:

# Make sure you import the module
Import-DSCResource -ModuleName "MSFT_xSystemSecurity"

# Put this in the node's configuration;
xUAC NeverNotifyAndDisableAll
{
    Setting = "NeverNotifyAndDisableAll"
}

Best Practices

Master Node

SharePoint is a distributed system, meaning that not all components run on the same server, and that most of the content and configuration is actually stored in a SQL Server database, that all servers in the farm can access at any time. When you define a DSC configuration file, you should identify one of the servers in your farm to be what I refer to has your Master Node. this is the node on which you will define most of the components in the Farm. All other nodes, referred to as secondary nodes in this scenario, should only define resources that allow them to install the required binaries and updates, to join the farm, and the service instances to run on that server. All other components should ONLY be defined on the Master Node.For example, don’t go and define an SPSite entry on multiple node, it is not necessary. SPSite objects are defined in the Content Databases anyway, so you don’t need every server in the farm to attempt to create them.

Take another example where you wish to update the DSC definition of a certain feature on a Site Collection to be disabled instead of enabled. If you properly defined your configuration, you only have to update the DSC definition for that feature within a single node (Master Node) in the DSC configuration file, but let’s imagine you have it defined within every node instead. You now have to go and modify it multiple times within your configuration, and that opens the door to mistakes. Let’s say our farm has 3 servers, that you updated the feature’s definition for Server #1 and #2, but forgot to update the definition in server #3. To make things worst, you’ve set all 3 servers’ Local Configuration Manager (LCM) to be set to ApplyAndAutocorrect. We all know that by default, the ConfigurationMode for the LCM will check every 15 minutes to make sure it is in its desired state, but that 15 minutes may not happen exactly at the same time on each of the servers in the farm. Imagine that at 10h a.m., server one has its LCM kick in, and realize that the feature has to be turned on based on the new definition. It will go and disable the feature. Now, let’s pretend that server #3’s LCM kicks in at 10h05 a.m., but since that server doesn’t have the proper definition, it re-disables the feature that was just turned on 5 minutes ago by Server #1. You can see where this is going, and that can cause a lot of headaches for your admin team. DO NOT DECLARE components (other than SPFarm, SPIntall, SPInstallPrereqs, and SPServiceInstance) on more than one server in the configuration.

Use ApplyAndAutocorrect with Caution

We took a decision a few years back not to let SharePointDSC deal with content such as SPList, SPContentTypes, etc. However, we do deal with items such as SPSite and SPWeb that are stored in the Content Database. The real beauty behind SharePoint is that it lets users deal with their content as they see fit. Users can easily go and create a new list, play with it for a few days, and then decide to delete it after because their tests are done.

PowerShell Desired State Configuration is very powerful. When you set the Local Configuration Manager of a server to ApplyAndAutocorrect, the server will automatically attempt to bring itself back into its defined Desired State Configuration whenever it detects that it drifted away from that configuration. While this feature is very powerful, my recommendation is not to set the LCM in ApplyAndAutocorrect the moment your configuration includes either and SPWeb or SPSite resource declaration, unless you are absolutely certain and aware of what this will do to your environment.

Imagine the following scenario where you have an environment which declares 3 SPWeb (say Web1, Web2, and Web3), and for which the LCM is set to ApplyAndAutocorrect. Come spring, the Information Management (IM) team at your organization undertakes a major cleanup project where they want to get rid of sites that have not been active in the past 6 months. There 3 SPWeb sites defined in the configuration, have been identified by the IM team to be deleted. The IM manager goes in and deletes the 3 sites, then sends an email to the IM director to let him know that the cleanup has been done. Two weeks after the IM director stumbles upon SPWeb2, which the IM Manager claimed to have deleted 14 days ago. Well, the moment the site was originally deleted by the IM Manager, 15 minutes after, the LCM detected that it was no longer in its desired state, and since its LCM is set to ApplyAndAutocorrect, it automatically recreated the site. Imagine the confusion this can cause.

Do Not Have Multiple SPProductUpdate & SPConfigWizard Entries

The SPProductUpdate resource is responsible for installing the bits from a Cumulative or Public Update (.exe) onto a server. When using it in a script that configures a farm from the ground up, the recommendation is to have it executed right after the SPInstall bits have installed the SharePoint binaries, and right before using the SPFarm resource to join the server to the farm. You need to make sure that this resource is applied to EVERY server in the farm, therefore it needs to be defined for each node in your Configuration Script.

The SPConfigWizard resource is responsible for executing the PSconfig process on the servers, committing the bits back to the farm’s configuration database. Once again, it needs to be run on every server in the SharePoint farm, and should therefore appear in every node in your DSC config.

It is obvious why you do not want to have more than one instance of the SPConfigWizard resource defined per server. There are no reasons why PSconfig should be ran multiple time on the server during a DSC consistency check. You also do not want to have more than one SPProductUpdate instance per server. Cumulative and Public updates are cumulative by nature, meaning that for example, you do not need to install the September 2017 Public update before installing the October 2017 Public Update. You simply install the October 2017 PU which contains all the bits from the previous updates as well. If you define multiple SPProductUpdate instances per server, the latest PU or CU will always overwrite the previous version, so you will end up installing old versions of patches, just so that they can be overwritten by more recent ones. Always remember to only have one instance of these two resources per node in your Configuration Script.

this article will be updated on an on-going basis.