LogicMonitor Downtime Scheduler
AutoSDT on Shutdown
2025-09-12
We use LogicMonitor for monitoring our systems and for on-call alert routing. One issue that quickly arose after implementation was false positive alert generation due to planned reboots. LogicMonitor checks system uptime, and if it's lower than a specific threshold, it chirps. Rather than requiring app-owners to schedule downtime in the platform prior to rebooting a server, we implemented a simple script to schedule SDT as part of the reboot process on our Windows and Linux boxes.
Important: Generate a key with a limited access service account in LogicMonitor. The account should have minimal privileges to manage SDT and read all devices.
Automating SDT on Linux
On Linux, we deployed two scripts, a Python script to schedule SDT in LogicMonitor, and a shell script to execute the SDT script prior to rebooting the system.
Shell Wrapper
The shell script just checks to make sure the SDT script is present and executes it prior to the reboot command if so. If it's not available for some reason, it just runs reboot without attempting to schedule downtime. This will result in alerts, but it won't delay the reboot process unnecessarily.
#!/bin/sh
if [ ! -f /usr/bin/lm_reboot_sdt ]; then
reboot
fi
/usr/bin/lm_reboot_sdt
reboot
Downtime Scheduler
The Generate_Request method is largely derived from the LogicMonitor API Docs. The script first confirms that we have a node with a matching hostname in LogicMonitor. This is determined using platform.node() which is helpfully what we also use when adding systems into the monitoring platform as part of our automated build process. Assuming a matching node is found, we construct an SDT payload and fire off the request to place the node into SDT. Note that it would be better to log the output rather than print it to the screen since the reboot command follows immediately after.
#!/bin/python
import sys
import json
import logging
import requests
import json
import hashlib
import base64
import time
import hmac
import platform
import time
def Generate_Request(method, path, request_data='', params=''):
access_id = '<access_id>'
access_key = '<access_key>'
org_name = '<org_name>'
url = f"https://{org_name}.logicmonitor.com/santaba/rest{path}"
epoch = str(int(time.time() * 1000))
request_vars = method + epoch + request_data + path
hmac1 = hmac.new(access_key.encode(),msg=request_vars.encode(),digestmod=hashlib.sha256).hexdigest()
signature = base64.b64encode(hmac1.encode())
auth = 'LMv1 ' + access_id + ':' + signature.decode() + ':' + epoch
headers = {'Content-Type':'application/json','Authorization':auth}
response = requests.request(method, url= url+params, data=request_data, headers=headers)
if response.status_code >= 200 and response.status_code <= 206:
logging.debug(response.json())
return response.json()
else:
print(response.text)
logging.error(response.text)
return False
def main():
device_name = platform.node()
lm_device_name = ''
device_list = Generate_Request(method="GET", path="/device/devices", params="?filter=name~" + device_name + "*")
if device_list:
if device_list['data']['total'] == 1:
lm_device_name = device_list['data']['items'][0]['name']
else:
for dev in device_list['data']['items']:
if dev['name'] == device_name:
return dev['name']
else:
pass
if lm_device_name == '':
print("Unable to locate " + device_name + " in LogicMonitor")
sys.exit("Exiting")
start_time = int(time.time())
# LogicMonitor wants epoch times in miliseconds
start_date_time = start_time * 1000
end_date_time = (start_time + 1200) * 1000
# Define payload
data = {
'type': "DeviceSDT",
'sdtType' : 1,
'comment' : "Issued reboot command",
'startDateTime' : start_date_time,
'endDateTime' : end_date_time,
'deviceDisplayName' : lm_device_name
}
# Schedule downtime and confirm result
result = Generate_Request(method="POST", path="/sdt/sdts", request_data=json.dumps(data))
if result:
if 'status' in result:
if result['status'] == 200:
print('SDT scheduled successfully (20 minutes)')
elif 'errmsg' in result:
print("Failed to schedule downtime. Error:" + result['errmsg'])
else:
print('Failed to schedule downtime. Unknown error. Response:' + result)
else:
print('Failed to schedule downtime. Unknown error.')
if __name__ == '__main__':
main()
Both scripts are deployed via Ansible using the playbook below. The final step in the playbook creates an alias for the reboot command in the bash profile for admin. Now, when an interactive user runs a reboot, the server is automatically placed into SDT long enough to suppress system uptime alerts.
---
- name: Install the LogicMonitor SDT utility and alias
hosts: all
tasks:
- name: Copy sdt script and set permissions
copy:
src: files/lm_reboot_sdt.py
dest: /usr/bin/lm_reboot_sdt
mode: '0775'
- name: Copy reboot script and set permissions
copy:
src: files/reboot_sdt.sh
dest: /usr/bin/rebootsdt
mode: '0775'
- name: Add alias to .bash_profile
lineinfile:
path: /admin/.bash_profile
line: "alias reboot='rebootsdt'"
create: yes
Automating SDT on Windows
Since we don't consistently deploy Python on Windows, we needed a PowerShell script to do the same thing. And while we manage our Windows hosts with Ansible, it's simpler to deploy scripts with Group Policy. The New-LMRequest function is provided in the LogicMonitor REST API examples.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
function New-LMRequest($method,$path,$request_data,$request_parameters) {
if($request_data){
$request_data = $request_data | ConvertTo-Json -Compress
}
$api_key = "<api_key>"
$api_secret = "<api_secret>"
$org_name = "<org_name>"
$credential = New-Object pscredential($api_key, (ConvertTo-SecureString $api_secret -AsPlainText -Force))
$epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds)
$request_vars = $method + $epoch + $request_data + $path
$access_id = $credential.GetNetworkCredential().username
$access_key = $credential.GetNetworkCredential().password
$hmac = New-Object System.Security.Cryptography.HMACSHA256
$hmac.Key = [Text.Encoding]::UTF8.GetBytes($access_key)
$signature_bytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($request_vars))
$signature_hex = [System.BitConverter]::ToString($signature_bytes) -replace '-'
$signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signature_hex.ToLower()))
$auth_key = "LMv1 $access_id`:$signature`:$epoch"
$url = "https://$($org_name).logicmonitor.com/santaba/rest" + $path
$headers = @{
'Content-Type'='application/json'
'Authorization'=$auth_key
}
$response = Invoke-RestMethod -Uri "$($url)$($request_parameters)" -Method $method -Headers $headers -Body $request_data
if($response){
return $response.data
}
elseif($response.errmsg){
Write-Output $response.errmsg
return $false
}
}
$start_epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds)
$end_epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date (Get-Date).AddMinutes(20)).ToUniversalTime()).TotalMilliseconds)
$data = @{
"type" = "DeviceSDT"
"sdtType" = 1
"comment" = "Reboot"
"startDateTime" = $start_epoch
"endDateTime" = $end_epoch
"deviceDisplayName" = $env:COMPUTERNAME.ToLower()
}
$result = New-LMRequest -method POST -path "/sdt/sdts" -request_data $data
The GPO copies the script to a protected local path and schedules it as a shutdown script to execute during the normal shutdown process.
