Patching a Multi-Server Farm using PowerShell DSC and Cross-Node Dependency

While I already have an article that covers the details of how you can use PowerShell Desired State Configuration (DSC) to patch a SharePoint Farm, I wanted to write a second article that covers the process of patching a multi-server farm, while respecting a sequential upgrade logic. Starting with PowerShell 5.0 we can now specify cross-node dependencies, which mean a node can wait for a specific resource to be in the desired state on another server before proceeding with the configuration of a given resource on the local node. By combining this new feature with the SharePointDSC module, we can fully automate our sequential upgrade sequence.

Scenario

The scenario is the following: the client has a 4 servers SharePoint 2013 Farm and wishes to upgrade it to the November 2017 Cumulative Update (CU). The client’s maintenance window is on Saturdays and Sundays, between 2AM and 8AM Eastern time. The SharePoint farm had 2 Web Front-End servers (SPWFE1 & SPWFE2), and 2 Application servers (SPAPP1 & SPAPP2). The upgrade sequence should be as follow:

If we were to put this into words, we should be able to install the binaries on all servers in parallel. The installation process may take longer to complete on some servers compared to the others. Once that is completed on all servers, then we should be running PSConfig on one server in the farm, and whenever the process finishes on that first server, then we can go an run it on all other servers in parallel. The orange dotted lines on the diagram represent “logic gates”. The first one, indicates that you should not be running PSConfig on the first server until the binaries installation process has completed on every server in the farm. The second one, specifies that we need to wait for the PSConfig process to finish on the first server before attempting to run it on the another server. These dotted lines will be represented by the Cross-Node Dependency resources WaitForAll in our DSC script.

Script

We will start off by defining our Node structure in our configuration script. Since SPWFE2, SPAPP1 & SPApp2 all follow the same “logic”, we will put them all 3 in the same basket. The other server, SPWFE1 has unique logic and therefore needs to have its own Node definition. The skeleton for this will look like the following:

Configuration SPFarmUpdate
{
    $CredsSPFarm = Get-Credential -UserName "contoso\sp_farm" -Message "Farm Account"
    Import-DscResource -ModuleName SharePointDSC -ModuleVersion 1.9.0.0
    Node "SPWFE1"
    {
        # First Node
    }

    Node @("SPWFE2", "SPAPP1", "SPAPP2")
    {
        # Other Nodes
    }
}

Since we can go ahead and install the binaries on all 4 servers at the same time, we can put the same logic block in both Node sections. The DSC resource used to install binaries update is SPProductUpdate. Because my client’s maintenance window is on Saturdays and Sundays between 2 and 8 AM, I will make sure to specify that window in my resource block.I will also specify that I wish to shutdown all services on my server to speed up the installation process. This will cause and outage of the SharePoint server. Just to keep things clear however, I will give them different names:

Configuration SPFarmUpdate
{
    $CredsSPFarm = Get-Credential -UserName "contoso\sp_farm" -Message "Farm Account"
    Import-DscResource -ModuleName SharePointDSC -ModuleVersion 1.9.0.0
    Node "SPWFE1"
    {
        SPProductUpdate November2017CUFirstNode
        {
            SetupFile = "C:\Patch\November2017CU.exe"
            ShutdownServices = $true
            BinaryInstallDays = @("sat", "sun")
            BinaryInstallTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
        }
    }

    Node @("SPWFE2", "SPAPP1", "SPAPP2")
    {
        SPProductUpdate November2017CUSecondaryNode
        {
            SetupFile = "C:\Patch\November2017CU.exe"
            ShutdownServices = $true
            BinaryInstallDays = @("sat", "sun")
            BinaryInstallTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
        }
    }
}

Once the binaries have been installed on the servers, we need to use the SPConfigWizard DSC resource to execute PSConfig and actually apply the update to the servers and databases. Once again, I will be specifying my maintenance window to ensure DSC doesn’t trigger an upgrade during business hours, and I will give the two DSC resource block different names to better illustrate the components involved. Our script now looks like the following at this stage:

Configuration SPFarmUpdate
{
    $CredsSPFarm = Get-Credential -UserName "contoso\sp_farm" -Message "Farm Account"
    Import-DscResource -ModuleName SharePointDSC -ModuleVersion 1.9.0.0
    Node "SPWFE1"
    {
        SPProductUpdate November2017CUFirstNode
        {
            SetupFile = "C:\Patch\November2017CU.exe"
            ShutdownServices = $true
            BinaryInstallDays = @("sat", "sun")
            BinaryInstallTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
        }

        SPConfigWizard PSConfigFirstNode
        {
            Ensure = "Present"
            DatabaseUpgradeDays = @("sat", "sun")
            DatabaseUpgradeTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
        }
    }

    Node @("SPWFE2", "SPAPP1", "SPAPP2")
    {
        SPProductUpdate November2017CUSecondaryNode
        {
            SetupFile = "C:\Patch\November2017CU.exe"
            ShutdownServices = $true
            BinaryInstallDays = @("sat", "sun")
            BinaryInstallTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
        }

        SPConfigWizard PSConfigSecondaryNode
        {
            Ensure = "Present"
            DatabaseUpgradeDays = @("sat", "sun")
            DatabaseUpgradeTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
        }
    }
}

Since by default, everything in DSC is executed sequentially at the node level, there is a risk that whatever server finishes installing its binaries automatically calls PSConfig even if the other servers are still in the middle of installing their binaries, or that multiple servers try calling PSConfig at the same time. To prevent this from happening, we will use the WaitForAll DSC resource which allows us to tell a node to wait for something else to complete on another node. In our case, we will tell the First server to wait for the binaries to be done installing on all servers in the farm and we will also tell the Secondary servers not to start their PSConfig process until it is first completed on the first node. In order for us to tell a DSC resource not to execute until something else on a different server has completed, we will use the DependsOn clause, and will “point” it to the associated WaitForAll instance. I will also tell the cross-node dependency to retry and check every minutes, for a maximum of 30 minutes to see if the dependency on the other server has completed. By doing so, we now have a complete script that looks like the following:

Configuration SPFarmUpdate
{
    $CredsSPFarm = Get-Credential -UserName "contoso\sp_farm" -Message "Farm Account"
    Import-DscResource -ModuleName SharePointDSC -ModuleVersion 1.9.0.0
    Node "SPWFE1"
    {
        SPProductUpdate November2017CUFirstNode
        {
            SetupFile = "C:\Patch\November2017CU.exe"
            ShutdownServices = $true
            BinaryInstallDays = @("sat", "sun")
            BinaryInstallTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
        }

        WaitForAll ServersToHaveBinariesInstalled
        {
            ResourceName = "[SPProductUpdate]November2017CUSecondaryNode"
            NodeName = @("SPWFE2", "SPAPP1", "SPAPP2")
            RetryIntervalSec = 60
            RetryCount = 30
            DependsOn = "[SPProductUpdate]November2017CUFirstNode"
        }

        SPConfigWizard PSConfigFirstNode
        {
            Ensure = "Present"
            DatabaseUpgradeDays = @("sat", "sun")
            DatabaseUpgradeTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
            DependsOn = "[WaitForAll]ServersToHaveBinariesInstalled"
        }
    }

    Node @("SPWFE2", "SPAPP1", "SPAPP2")
    {
        SPProductUpdate November2017CUSecondaryNode
        {
            SetupFile = "C:\Patch\November2017CU.exe"
            ShutdownServices = $true
            BinaryInstallDays = @("sat", "sun")
            BinaryInstallTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
        }

        WaitForAll PSConfigToBeFinishedOnFirstServer
        {
            ResourceName = "[SPConfigWizard]PSConfig"
            NodeName = @("SPWFE1")
            RetryIntervalSec = 60
            RetryCount = 30
            DependsOn = "[SPProductUpdate]November2017CUSecondaryNode"
        }

        SPConfigWizard PSConfigSecondaryNode
        {
            Ensure = "Present"
            DatabaseUpgradeDays = @("sat", "sun")
            DatabaseUpgradeTime = "2:00am to 8:00am"
            PsDscRunAsCredential = $CredsSPFarm
            DependsOn = "[WaitForAll]PSConfigToBeFinishedOnFirstServer"
        }
    }
}
$ConfigData = @{
    AllNodes = @(
        @{
            NodeName = "SPWFE1"
            PSDscAllowPlainTextPassword = $true;
            PSDscAllowDomainUser = $true;
        },
        @{
            NodeName = "SPWFE2"
            PSDscAllowPlainTextPassword = $true;
            PSDscAllowDomainUser = $true;
        },
        @{
            NodeName = "SPAPP1"
            PSDscAllowPlainTextPassword = $true;
            PSDscAllowDomainUser = $true;
        },
        @{
            NodeName = "SPAPP2"
            PSDscAllowPlainTextPassword = $true;
            PSDscAllowDomainUser = $true;
        }
    )
}

SPFarmUpdate -ConfigurationData $ConfigData