Log4j Defect - It's Patched!

in

I wrote an article back in January (Log4j MID Server Fix Script) regarding Log4j defect on MID servers where they had a vulnerable version of the Log4j jar files on them. Great news - the issue has now been patched for good!

As of Rome Patch 7 and San Diego Patch 1, MID servers are now packaged with Log4j version 2.17.1 which is the currently safe version of the library.

All you need to do is make sure to upgrade your instance version to one of those patches or higher, and then also upgrade your MID server (typically happens automatically), and then you’re set.

I haven’t seen much official information posted that it was fixed, but will be sure to update the article if I find reference to an official announcement.

I also wanted to include as a bonus some instructions on how to verify the version of jar files on your MID server!

  1. Log into the MID server

  2. Open a CMD terminal as an administrator

  3. Type in the following commands:

    > F:

    > cd: D:\MIDServer\agent\jre\bin\

    > jar xf D:\MIDServer\agent\lib\log4j-core.jar

  4. Navigate to to D:\MIDServer\agent\jre\bin\META-INF\MANIFEST.MF file

  5. Open with a text editor, and verify the version number (ex: in the case of Log4j we want 2.17.1)

Note: After completing the instructions you can delete the META-INF folder that is created.

State of Scoped Apps in 2022

in

My third most popular article since founding this blog was my “Scoped Apps, the good, bad and ugly”, and there have been quite a few changes in the space since 2019, it’s a bit overdue to revisit! I’m not going to re-hash my old sentiment, since I think it mostly still holds true (except for the workflow issues that have kind become old news with the explosion of Flows). Instead I’ll be focusing on the “new” things, either new in general or new to me!

Scoped applications are merely a encapsulated way to build applications on top of ServiceNow, instead of building everything in the default space. We continue to see more and more ServiceNow created scoped applications as time progresses (notable ones being HR and SecOps).

Note: At the time of writing this, I’m in a Rome Patch 4 instance.

The Good

  1. CI/CD Spoke. If you haven’t seen ServiceNow CI/CD spoke yet, and all the automations available, you’re missing out. This has the potential to automate the entire scoped app release pipeline, from publishing, installing, scanning and testing, it can do it all. Except for update sets.

  2. Flow Designer Improvements. With Flow designer blowing up, and so many new spokes available, and a re-designed UI layout, it’s better to use than ever. You can create re-usable actions in scoped apps and use them anywhere across the instance - it’s amazing.

  3. Uninstalling. I should have covered this in my original post, but a major advantage that Scoped Apps have over other development, is that you can simply uninstall an app (and have options to leave the data). You can’t do anything that quick on regular global development to revert everything. If you need to sunset the app down the road, there is a clear path to do so.

The Bad

  1. App Builder. I think the idea was good, but the implementation is poor, and I find it more of a hindrance and myself skipping the options as quick as I can. I would suggest instead of a “builder” approach, switch to a “template” approach, since most scoped application can be generalized in a couple of different ways. Also if people could create and publish their own templates, this would be an amazing win for the community.

  2. Many still discourage it’s use (among ServiceNow and major partners). I’ve gotten involved on setting up new ServiceNow instances, and have had a chance to work with various professional services developers. The resounding message I get is that most don’t want to work in scoped applications. There isn’t much push towards scoped applications in the industry still.

  3. Don’t Make Catalog Items in Scopes. One of the bigger pieces of best practice I’ve picked up on, is to not make catalog items in Scoped applications. There is too much value for re-use in variable sets, and in Flows for putting Catalog items in scoped to make sense. It’s better to build the catalog in global.

The Ugly

  1. Flow Issues. With Flows being the big new thing that ServiceNow is advocating, there sure are a lot of issues with flows and scoped applications. The main one being that flows being activated don’t move when you install a scoped application, it only moves in an update set. Additionally Flows are not always captured properly in a scoped application and can be corrupted. Just stick to update sets or Global if you want to use Flows.

  2. Versioning. Something lacking overall in the ServiceNow environment is version control, and with the addition of GitHub syncing, it’s a god send. However there is a sore need to being able too roll back development versions, and start over from a prior time within the instance. I’ve seen way too many apps just abandoned because they got to a point it wasn’t maintainable, or someone deleted a table, and couldn’t revert it.

  3. Cross Scope Permissions are Buggy. ServiceNow has a feature to allow cross scope access to tables, script includes, etc, and it’s stored on several tables in the instance. Typically you never have to mess with these records, and cross scope records are automatically generated the first time you try to run something. However more and more either these records don’t get generated correctly, or when you move an app, they don’t carry over. The only fix I’ve found is to delete them and manually re-create them to toggle the permissions.

  4. Admins can’t do Scoped ACLs, but non-admins Can !? There is a weird bug existing in newer ServiceNow instances, where if you’re a scoped app developer for an app you can edit and update the ACL within that app. However if you’re an admin (unless you have security_admin) you can’t edit it. This just further complicates the lines on the developer landscape with the 3 levels of scoped app developer, admin, and security_admin.

Log4j MID Server Fix Script

in ,

I know many people have been scrambling after the announcement of the Log4J vulnerability, and ServiceNow is not unscathed entirely. The MID server packages that are provided by ServiceNow, even in the latest version at the time of writing contain the vulnerability as a dependency, even though they don’t claim to use them. If you’re on the current MID server patch your Log4J is probably like version 2.14 or older and you need this fix.

The only workaround is to manually stop the service, delete and replace the jar files, and restart the service (described on KB1001211). Rinse and repeat for every MID server, and after every single patch or upgrade to the system, until ServiceNow decides to update it.

If you’re like me and have started going a little crazy, I’ve come up with a decent partially scripted solution for at least Windows MID servers! I hope others can make use of this trick to remediate their issue in a quicker and consistent manner.

Step 1. Download the latest jars of Log4j that are secure from logging.apache.org (currently v2.17.1). Save the ZIP to your desktop, extract and locate the log4j-api.jar, log4j-core.jar, and log4j-slf4j-impl.jar files.
Note: If they aren’t renamed exactly like log4j-core.jar, you will need to rename the file manually.

Step 2. Create a Local update set in your instance.

Step 3. Upload the 3 Jar files as a MID Server Jar file. Navigate to MID Server > JAR Files, click New, and fill out the name, version, description, and upload the file and save. Repeat for the 3 jar files.

Step 4. Create this MID server script file with the name as “Log4j.ps1” and this as the script:

#Run the following command using admin powershell terminal
#Usage>."D:\MyMIDServer\agent\scripts\Log4j.ps1"

#Primary MID Server Install path
$scriptsPath = split-path -parent $MyInvocation.MyCommand.Definition
$agentPath = split-path -parent $scriptsPath
Write-Output "Auto detected agent path as $agentPath"

#Path where all the new jars are located
$extlib = $agentPath + "\extlib\"

$startbat = $agentPath + "\start.bat"
$stopbat = $agentPath + "\stop.bat"

$extfile1 = $extlib + "log4j-api.jar"
$extfile2 = $extlib + "log4j-core.jar"
$extfile3 = $extlib + "log4j-slf4j-impl.jar"
$mainlib = $agentPath + "\lib\"
$file1 = $mainlib + "log4j-api.jar"
$file2 = $mainlib + "log4j-core.jar"
$file3 = $mainlib + "log4j-slf4j-impl.jar"

$file1exist = Test-Path $extfile1
$file2exist = Test-Path $extfile2
$file3exist = Test-Path $extfile3
#Write-Host "($file1exist) -and ($file2exist) -and ($file3exist)"

if (($file1exist) -and ($file2exist) -and ($file3exist)){
  Write-Output "Stopping MID Server Process"
  Start-Process -FilePath $stopbat -Wait -NoNewWindow

  Remove-Item -Path $file1 -Force
  Remove-Item -Path $file2 -Force
  Remove-Item -Path $file3 -Force
  Write-Output "Old Jar Files Removed successfully."

  Copy-Item -Path $extfile1 -Destination $mainlib
  Copy-Item -Path $extfile2 -Destination $mainlib
  Copy-Item -Path $extfile3 -Destination $mainlib
  Write-Output "New Jar Files Copied successfully."

  Start-Sleep -Seconds 1.5
  Write-Output "Starting MID Server Process"
  Start-Process -FilePath $startbat -Wait -NoNewWindow

  Clear-RecycleBin -DriveLetter D -Force -ErrorAction SilentlyContinue
  Get-ChildItem "D:`$Recycle.bin\" -Force | Remove-Item -Recurse -force
  Write-Output "Recycle Bin cleared successfully."
}else{
  Write-Output "New Jar files not detected in extlib folder, restarting MID"
  Start-Process -FilePath $stopbat -Wait -NoNewWindow
  Start-Sleep -Seconds 1.5
  Start-Process -FilePath $startbat -Wait -NoNewWindow
  Write-Output "Re-run script after Jar files have been propogated"
}

Step 5. Restart the MID server service so it force syncs the files.

Step 6. Remote into the server, or connect remotely using PowerShell.

Step 7. Open Powershell terminal as an admin and execute the line below. You will need to change the path to whatever path the MID server agent folder is using.

>. "D:\MyMIDServer\agent\scripts\Log4j.ps1"

Step 8. Validate the results. Open the lib folder and see the updated dates on the jar files.

Step 9. Close and move your update set and re-do steps 5-8 for any other environments and MID servers

My script includes some logic to auto-detect the folder, and detect if the jars are synced, otherwise it just restarts the service. The script also includes logic to clear the recycle bin, since the file can still be found even if in the recycle bin. If you are not using the D drive, you will need to slightly update the two lines to whichever drive you are using.

Also a word of caution, just deleting the log4j jar files without replacing will break the MID server service from starting all together. We can only hope that ServiceNow properly patches this soon.

Prototype.js, ServiceNow, and Objects

in

Disclaimer: This article assumes some entry level computer science concepts, and it’s a bit dry!

A well known programming concept is Classes, and their instance, Objects, which is a common staple of Object Oriented Programming (OOP) languages. Older versions of javascript (ES5 and below) which is what ServiceNow uses don’t have support for Classes. In an attempt to solve this problem, to add support for the concept of classes, ServiceNow used Prototype.js. A common place prototype is used for all the classes we define in ServiceNow's Script Includes.

Let’s take a look at an example Prototype Class that also could be a script include.

var Pirate = Class.create();
Pirate.prototype = {
  initialize: function(name) {
    this.name = name;
  },
  say: function(message) {
    return this.name + ': yarr ' + message;
  }
  type: 'Pirate'
};

The key parts are defining a class, it’s prototype, initialize method (aka the constructor, which can be optional), functions (aka methods), and the type variable which ServiceNow uses to reference script includes.

Inheritance & Polymorphism

Prototype also has support for common OOP concepts like Inheritance, with super and sub classes. A well known example for this is the GlideAjax super class. Along with Inheritance support, there is also support for method overriding and polymorphism. Since methods can be overridden, you don’t want to override like initialize function when extending the AbstractAjaxProcessor, otherwise it would break.

var myAjax = Class.create();
myAjax.prototype = Object.extendsObject(AbstractAjaxProcessor, {
      getMessage: function () {
        return "success"
      }
});

Encapsulation & Private Elements

Another capability is encapsulation, and support for private functions and data variables. You can set a function or data element as private by using the “_” underscore prefix in front of the name. The data or function can only be called and accessed from within the class itself by using the this._privateVar syntax. All other methods and classes are by default considered public, and there is no concept of protected variables.

This Object

I’ve referenced it in the initial example, but Prototype provides a very helpful “this” object, which allows you to store data variable and call other functions within the current function. This also is incredibly helpful because with the type variable on all script includes, it’s very easy to log and identify where something is coming from. This is very similar to the “current” object available within business rules and workflows, but provides you with real accurate information regarding the script location.

Closing Thoughts

I’m not entirely sure which version of Prototype ServiceNow is running, but it seems to be older than 1.6.0 since there isn’t support for superclass or subclass properties. I can only assume a version was selected around 2004 when ServiceNow was initially developed and released. Script includes are one of the greatest capabilities that ServiceNow has to offer for re-usable server side code snippets, and prototype.js enables it to be that much more powerful.

Node Switcher UI Page!

in ,

If you aren’t familiar, when you log into ServiceNow, you are assigned a node dynamically, to serve all your transactions. There many be many situations where you may want to switch to a different node. There isn’t any functionality available to the general public for this (though I know you exist hop2node!)

I developed my own node switching tool as an alternative to Chrome Extensions, since many companies have security restrictions for those. This page instead acts as a native tool within ServiceNow!

A number of you might be wondering how I was able to get this to actually work…. Or even if you aren’t I’m here to explain anyways!

Step 1 - Figuring out how to get the node IP address cookie value for all nodes. For this I had two big hints:
This little tool on the ServiceNow Share: Generate Node BigIP - Switch Between ServiceNow Nodes

https://developer.servicenow.com/connect.do#!/share/contents/4214057_generate_node_bigip_switch_between_servicenow_nodes?v=1.0&t=PRODUCT_DETAILS

This gave me the biggest hint, that part of nodes name on typical deployments, includes the last numbers for the IP address of that node. That saves so much time from a lookup perspective, since you just need to get the ballpark initial parts of the IP address first, since the nodes should be in the same data center.

The next part, which is the even harder part, most tools make use of querying stats.do to get the node IP address. However, that page is pretty long, parsing it isn’t as reliable, and most security advise is to lock down that page. I got a tip from Ahmed Hmeid from ServiceNow Gems to use SOAP against InstanceInfo.do. This is the same way the MID servers get the instance details. Basically just have to spoof a MID server!

Step 2 - Figuring out what the user’s current node is. This part was also harder than I expected. The value of the node stored in the cookie is an HTTP only flag cookie, which means it can’t be read client side at all. However, there are two server side workarounds!

var request = GlideTransaction.get().getRequest();
result.user.node_cookie = GlideCookieMan.getCookieValue(request.getCookies(), 'BIGipServerpool_' + this.instanceName);

result.user.node_name = (GlideServlet.getSystemID() + '').split(':')[1]; //get the second half of the node name

Step 3 - Mechanism for switching the node. Since it’s a special cookie, you can’t just use any normal client side function to change it. I found out though after a bit of digging, that Processors have the ability to write cookies! There is an undocumented function called g_response.addCookie which can set and override even HTTP only cookies.

var cookie = new Packages.javax.servlet.http.Cookie('BIGipServerpool_' + instanceName, node_id);
cookie.setPath("/");
g_response.addCookie(cookie);

All together the flow looks like this: UI Page Pre-Load > Script Include > SOAP > UI Page > Processor > UI Page. I’ve also added some other data enrichment in the script include to grab node metrics as well.

Closing Thoughts

I hope to one day package this up and post it on the ServiceNow share. Just a couple more things I have to get cleared up first (like getting it to work with other node naming conventions, simplifying the MID server user requirement, etc.). Developing this tool was quite the ride and challenge, but I’m happy with the result!

Top 6 Wish List Features for ServiceNow Administrators

in

We can’t help but always want more! Here are my personal thoughts on 6 features that would round out the product and make ServiceNow’s administrator’s day to day life so much easier.

  1. Restart a node

    I couldn’t count the number of times when some rogue Performance Analytic job got hung, and an entire node got hung. You have to reach out to HI, have them verify, and then they have to initiate the bounce. It’s 2021, and we don’t have the option to turn it off and back on in ServiceNow.

  2. Trace a transaction, log, error, etc

    Something that an absolute pain in ServiceNow is figuring out something about a server side transaction, a log or anything, and trying to trace it down to what might be the root cause. You have to have multiple tabs open, recording timestamps, session ids, user ids, etc., to find anything at all. I would kill for some call stack type of technology, or better debugging tools.

  3. Enhanced Self Monitoring

    ServiceNow actually has a self monitoring capability for Event Management, so it can manage itself. It would be great to take the step even further and build in self monitoring for nodes, database, event queue, email queue, or literally anything else besides just Event Management and MID servers resource alerts. I know some things are in the pipeline, but I just wish this formally gets expanded upon. So much potential.

  4. AI DB Index Enhancer

    ServiceNow has some little tools to suggest database indexes, but wouldn’t it be great if it was smarter? Or you could set it up with some rules to automatically make suggested index tweaks on the fly? Some automation and algorithm to use indexes better could make a mediocre instance an amazing instance.

  5. Workspace with quick actions dashboard for SysAdmins

    Workspaces are beginning to be all the rage, and they are great for so many different user personas. One such user persona is a ServiceNow admin. It would be awesome if there was a workspace that delivered to them their days work right in a single interface with powerful buttons and actions to manage that world. Simplify so many clicks. I know there are some good community dashboards and tools out there for sysadmins, so I hope this is just a matter of time!

  6. Release Automation

    ServiceNow has been iterating on CI/CD for a number of releases now, but we still don’t have any automated capabilities for update sets, the backbone of ServiceNow instances for over a decade. I also hope they build some interfaces and flows for customers that want to use those CI/CD spoke actions, so they can get going fast.

Undocumented: Table Extension APIs

in

There are many undocumented ServiceNow APIs that have existed since the beginning. These are a couple that you may come across needing either as a system administrator or developer.

The first is the GlideTableParentChange if you need to move a table to be extended from another table. An example use case is that you may have accidently created it under the wrong table, like the wrong CI or task table.

var table = ''; //table you want to extend
var old_parent = '';
var new_parent = ''; //the table you want to extend to, like cmdb_ci or task
var changer = new GlideTableParentChange(table);
changer.change(old_parent, new_parent);

The second is GlideDBUtil.promoteColumn which is a single one line script that lets you move a column up to the parent level table. This also might be useful in CMDB management, where you may have created a column under a table, but it really need to be moved to the parent so more CIs can use the field.

GlideDBUtil.promoteColumn('table_to_move_from', 'table_to_move_to', 'field_to_move', true);

Finally there is a documented TableUtils API that also offers some useful functions around dropping tables.

https://docs.servicenow.com/bundle/rome-application-development/page/app-store/dev_portal/API_reference/TableUtils/concept/c_TableUtilsAPI.html?cshalt=yes

Monitoring Series: Database Size Report

in

ServiceNow instance performance is a cause of concern and notice for any ServiceNow admin. To get ahead of the curve, it is a good idea to keep an eye on your instance database size.

Database size, the sum of all the tables and table index data stored in the instance database server. It’s easy to ask ServiceNow directly or fill out their self service form on the Support page, but even easier to just make a report and put in on a dashboard yourself!

Step by Step

  1. First go to Reports > Create New

  2. Set a name like “Current Database Size (MB)”, and select the table UA Instance Info [ua_instance_state].

  3. Select type as Speedometer (you can also do single score, but it’s nice to set ranges!)

  4. On the configure tab, select Aggregation as Sum and field as “Primary database size (MB)”

  5. On the style tab, you can set the direction as Minimize, and set reasonable ranges you think your database size should fall under.

  6. Finally, you can go and add it to a dashboard.

The finished result should look something like this:

db_size_detail.PNG

This report is an excellent addition to any system administrator’s dashboard.