MID Server Administration in 2022

My second most popular blog article since founding this blog was “Best Practices: MID Server Administration”, and while I think it has aged well, things are ever changing. I won’t be rehashing everything again, just where I think some updates are needed!

1. Basics

   a. Sizing / Server Specs

   MID Servers have gotten more resource intensive over recent years, some newer applications (ACC, CPG, EM) are being recommended to add more CPU (now up to 8 core), and more memory (8gb normal, 16 on the high end). I’ve also seen some Discovery Patterns like Kubernetes chug with 4 core CPU and 6 GB dedicated wrapper memory - but it’s very dependent on the scale of discovery. I would recommend a base load out in Production with 8 Core CPU, 12 GB RAM, 60GB Disk, and updating the MID Server config to leverage more than 25 threads, and 6GB or more of usable RAM. You might be able to get away with less for Orchestration, and you’ll need more for specialized cases like ACC.

   b. Multiple Services on a Single Server

   It’s possible to install the MID Server and run it multiple times on the same physical/virtual server. For sub productions this is an amazing way to scale to add additional MID Server services, while maximizing Server resources. I think the practice makes a lot of sense in certain situations, but there are some key limitations. MID Servers can interfere with each other and cause resource spikes, they will compete for network bandwidth directly, and they can make issues harder to debug. I would stick to only using these in subprod, and regardless how beefed up the server is, I wouldn’t go any higher than 3 services running on the same box.

2. Counts - How Many?

   a. MID Server Calculator / Discovery Estimator

   ServiceNow provides a worksheet to go through and calculate based on how many devices (servers, routers, switches, storage arrays, load balancers) you have, and how often you need the discovery data refreshed. If you have 10k servers, but only need them discovered weekly over the weekend, then you can get away with fewer MID servers. If you need them discovered daily, but can only discover them at night, to reduce system load, then you need a much higher number of MID servers. There are definitely companies out there that have over 100 production MID servers to handle such large amounts of infrastructure and time requirements.

   b. Redundancy

   Besides having enough MID servers to do the job, the next level of maturity is introducing redundancy measures. If most of your MID servers are in a single data center, and that goes down, you need a backup plan. There should be multiple redundant servers in different data centers and for different use cases like orchestration and discovery.

3. Time Sliced Usage

   To make the most out of your MID server, it is important to think about spreading out the load, both across multiple MID servers, as well as spreading it out along the timeline. Most people are familiar configuring discovery schedules, but it is also important to think about this in the sense for other applications as well, to try and spread out the run times. You could for instance have another service running an integration load during the day, and a separate service on the same server running discoveries at night.

4. MID Servers in Containers - The Future

   I would be amiss if I didn’t talk about the very obvious future of MID Servers in the ecosystem. More and more applications are moving towards containerized approaches, such as with a Docker image, and I would imagine MID server would move that way as well. A docker image can be set up with a specific operating system and configuration, and can be spun up and down with a click of a button. It would be a more cost effective approach to just use what you need, and it would be easier to quickly scale up and down devices. I hope ServiceNow invests in this technology in the future.

I hope MID Server administration practices become much more mainstream as time progresses!

Log4j Defect - It's Patched!

I wrote an article back in January (Log4j MID Server Fix Script) regarding Log4j defect on MID servers where they had a vulnerable version of the Log4j jar files on them. Great news - the issue has now been patched for good!

As of Rome Patch 7 and San Diego Patch 1, MID servers are now packaged with Log4j version 2.17.1 which is the currently safe version of the library.

All you need to do is make sure to upgrade your instance version to one of those patches or higher, and then also upgrade your MID server (typically happens automatically), and then you’re set.

I haven’t seen much official information posted that it was fixed, but will be sure to update the article if I find reference to an official announcement.

I also wanted to include as a bonus some instructions on how to verify the version of jar files on your MID server!

1. Log into the MID server

2. Open a CMD terminal as an administrator

3. Type in the following commands:

   > F:

   > cd: D:\MIDServer\agent\jre\bin\

   > jar xf D:\MIDServer\agent\lib\log4j-core.jar

4. Navigate to to D:\MIDServer\agent\jre\bin\META-INF\MANIFEST.MF file

5. Open with a text editor, and verify the version number (ex: in the case of Log4j we want 2.17.1)

Note: After completing the instructions you can delete the META-INF folder that is created.

Log4j MID Server Fix Script

I know many people have been scrambling after the announcement of the Log4J vulnerability, and ServiceNow is not unscathed entirely. The MID server packages that are provided by ServiceNow, even in the latest version at the time of writing contain the vulnerability as a dependency, even though they don’t claim to use them. If you’re on the current MID server patch your Log4J is probably like version 2.14 or older and you need this fix.

The only workaround is to manually stop the service, delete and replace the jar files, and restart the service (described on KB1001211). Rinse and repeat for every MID server, and after every single patch or upgrade to the system, until ServiceNow decides to update it.

If you’re like me and have started going a little crazy, I’ve come up with a decent partially scripted solution for at least Windows MID servers! I hope others can make use of this trick to remediate their issue in a quicker and consistent manner.

Step 1. Download the latest jars of Log4j that are secure from logging.apache.org (currently v2.17.1). Save the ZIP to your desktop, extract and locate the log4j-api.jar, log4j-core.jar, and log4j-slf4j-impl.jar files.
Note: If they aren’t renamed exactly like log4j-core.jar, you will need to rename the file manually.

Step 2. Create a Local update set in your instance.

Step 3. Upload the 3 Jar files as a MID Server Jar file. Navigate to MID Server > JAR Files, click New, and fill out the name, version, description, and upload the file and save. Repeat for the 3 jar files.

Step 4. Create this MID server script file with the name as “Log4j.ps1” and this as the script:

#Run the following command using admin powershell terminal
#Usage>."D:\MyMIDServer\agent\scripts\Log4j.ps1"

#Primary MID Server Install path
$scriptsPath = split-path -parent $MyInvocation.MyCommand.Definition
$agentPath = split-path -parent $scriptsPath
Write-Output "Auto detected agent path as $agentPath"

#Path where all the new jars are located
$extlib = $agentPath + "\extlib\"

$startbat = $agentPath + "\start.bat"
$stopbat = $agentPath + "\stop.bat"

$extfile1 = $extlib + "log4j-api.jar"
$extfile2 = $extlib + "log4j-core.jar"
$extfile3 = $extlib + "log4j-slf4j-impl.jar"
$mainlib = $agentPath + "\lib\"
$file1 = $mainlib + "log4j-api.jar"
$file2 = $mainlib + "log4j-core.jar"
$file3 = $mainlib + "log4j-slf4j-impl.jar"

$file1exist = Test-Path $extfile1
$file2exist = Test-Path $extfile2
$file3exist = Test-Path $extfile3
#Write-Host "($file1exist) -and ($file2exist) -and ($file3exist)"

if (($file1exist) -and ($file2exist) -and ($file3exist)){
  Write-Output "Stopping MID Server Process"
  Start-Process -FilePath $stopbat -Wait -NoNewWindow

  Remove-Item -Path $file1 -Force
  Remove-Item -Path $file2 -Force
  Remove-Item -Path $file3 -Force
  Write-Output "Old Jar Files Removed successfully."

  Copy-Item -Path $extfile1 -Destination $mainlib
  Copy-Item -Path $extfile2 -Destination $mainlib
  Copy-Item -Path $extfile3 -Destination $mainlib
  Write-Output "New Jar Files Copied successfully."

  Start-Sleep -Seconds 1.5
  Write-Output "Starting MID Server Process"
  Start-Process -FilePath $startbat -Wait -NoNewWindow

  Clear-RecycleBin -DriveLetter D -Force -ErrorAction SilentlyContinue
  Get-ChildItem "D:`$Recycle.bin\" -Force | Remove-Item -Recurse -force
  Write-Output "Recycle Bin cleared successfully."
}else{
  Write-Output "New Jar files not detected in extlib folder, restarting MID"
  Start-Process -FilePath $stopbat -Wait -NoNewWindow
  Start-Sleep -Seconds 1.5
  Start-Process -FilePath $startbat -Wait -NoNewWindow
  Write-Output "Re-run script after Jar files have been propogated"
}

Step 5. Restart the MID server service so it force syncs the files.

Step 6. Remote into the server, or connect remotely using PowerShell.

Step 7. Open Powershell terminal as an admin and execute the line below. You will need to change the path to whatever path the MID server agent folder is using.

>. "D:\MyMIDServer\agent\scripts\Log4j.ps1"

Step 8. Validate the results. Open the lib folder and see the updated dates on the jar files.

Step 9. Close and move your update set and re-do steps 5-8 for any other environments and MID servers

My script includes some logic to auto-detect the folder, and detect if the jars are synced, otherwise it just restarts the service. The script also includes logic to clear the recycle bin, since the file can still be found even if in the recycle bin. If you are not using the D drive, you will need to slightly update the two lines to whichever drive you are using.

Also a word of caution, just deleting the log4j jar files without replacing will break the MID server service from starting all together. We can only hope that ServiceNow properly patches this soon.

Best Practices: MID Server Administration

Below are some of my personal thoughts and advice on MID Server configuration, set-up and maintenance. Some of these are pretty well known and adopted, and some are based on my personal experiences.

1. Basics

   a. Sizing / Server Specs

   Specification wise, RAM/Memory and CPU Core Count/(and associated threads) are the most important. Usually 8GB RAM, 4 Core CPU is recommended with 40+GB disk and 64bit OS which is a requirement being enforced going forward. Within the MID server parameters there are options to use more RAM, Threads, etc. Generally specifying to use more RAM is recommended, and using more threads, results can vary.

   b. Network (Bandwidth and Location)

   This point is the key reason that MID servers are needed in the first place. They are a necessary appliance so that you can securely interact with other resources and devices on a network. It’s important to have fast connection speed (usually 100mb/s upload is recommended), and put the MID server physically close to the devices it is interacting with, for quick and optimized interactions. If you have multiple data centers, the servers should be spread out in those data centers, and put ones on similar subnets as the majority of devices.

2. Segregate by Purpose

   a. History

   The biggest flaw I’ve always seen with enterprise MID server set ups, is that there is no separation between Discovery and Orchestration usage. Many Discovery credentials are server admin or granted powerful rights to log into machines and run discovery commands. There are a couple ways you can force orchestration tasks to use specific credentials, using credential tags, and hardcoding MID server IP relationships with the target Orchestration device. However you can’t lock down the Discovery credentials to only be used for discovery, which can be a major security issue.

   b. Resolution

   The only guaranteed mechanism for locking down those credentials is to lock them down to only be allowed to use on specific MID servers. Additionally on those MID servers you have the options to not allow Orchestration application. Thus it’s critical to have separate MID servers for Discovery, Orchestration, JDBC and other activities to enforce proper credential usage.

3. Security

   Review the MID Server Hardening Guide.

   There are a number of overlooked recommendations, for setting specific security parameters, like disabling SSL, TLS 1.0 and 1.1, setting the DH group value and encrypting proxy passwords (if applicable).

4. Closing Thoughts

   Besides just initial setup, every administrator knows that you have to keep current with upgrades, restarts, and rekeying the credentials. Besides those standard activities to take it a step further, you can set up proactive monitoring, such as using the built in MID Server resource threshold alerts, or advanced tools like PerfMon, Microsoft SCOM or Datadog.

Hope everyone learned something! I’ve been absent, but hopefully we can start off 2021 right and have a lot of content. Please comment your ideas or anything you want to see!