Integrations

We use LAPS to rotate the default Administrator account password on Windows. LAPS conveniently stores passwords in a secure portal (or now in InTune), and RBAC permits adminstrators to access them on demand.

There was a notable gap in how we managed (or didn't) local account passwords on Linux. Since we're using key-based authentication, it was easy to ignore that local passwords hadn't changed since deployment and were shared across multiple machines. This awkwardly came to light during an audit, and it was time to do something about it.

Working with Existing Tools

Secret Server

Changing passwords is a straightforward process, but we had some requirements we needed to sort out. - Passwords need to be unique to each system - Passwords need to be accessible to administrators in the event that SSH is unavailable - Password access needs to be audited

Since we currently use Secret Server, and it meets the requirements for restricted access and an audit trail, this was a logical place to start. Their API is incredibly easy to work with, and they provide plenty of examples to get started.

I set up an application account in secret server, created a new root folder, and granted permissions on the folder to our team and the application account. This limits the scope of the application's access to just the secrets in this folder in the event that the account is compromised. The folder permissions are set to Add secret and the secret permissions are Edit. This allows the application to create new secrets and modify existing secrets within the folder.

Ansible

Since our Linux hosts are all managed by Ansible, that was the simplest option for processing password changes. Note that most of these tasks are using the no_log: True option to prevent the password from being recorded in ansible.log.

To begin, we needed to define a few variables. Most of these are hardcoded into the playbook but could just as easily be input variables. The template ID and folder ID can be found in the URL associated with each object in Secret Server. The name of the secret will match the inventory_hostname value in Ansible, and the default user for the password change is root.

  vars:
    secret_template_id: 101
    secret_folder_id: 101
    secret_site_id: 1
    secret_username: "{{ root_user | default('root') }}"
    secret_name: "{{ inventory_hostname }}"

Next, we needed to generate a random password that meets our requirements. The task below generates a random 32-character string.

- set_fact:
    new_password: "{{ lookup('community.general.random_string', length=32, special=true) }}"
    no_log: true

We now need to fetch an authentication token and store it as ss_token using the two tasks below. The ss_url, ss_username, and ss_password variables are securely stored and fetched in our ansible vault.

- name: Authenticate to Secret Server
    uri:
    url: "{{ ss_url }}/oauth2/token"
    method: POST
    body_format: form-urlencoded
    body:
        username: "{{ ss_username }}"
        password: "{{ ss_password }}"
        grant_type: password
    return_content: yes
    register: ss_auth
    delegate_to: localhost
    run_once: true
    no_log: True

- name: Set auth token fact
    set_fact:
    ss_token: "{{ ss_auth.json.access_token }}"
    delegate_to: localhost
    run_once: true
    no_log: True

With our token in hand, we can query the secrets API to determine if an entry matching the inventory_hostname already exists.

- name: Search for existing secret
    uri:
    url: "{{ ss_url }}/api/v2/secrets?filter.searchtext={{ secret_name }}"
    method: GET
    headers:
        Authorization: "Bearer {{ ss_token }}"
    return_content: yes
    register: ss_search
    delegate_to: localhost
    no_log: True

If the entry exists, we'll send a PUT request to update it. If the entry doesn't exist, we'll send a POST request to create it.

- name: Create or update secret
    uri:
    url: >-
        {{ ss_url }}/api/v1/secrets{{
        '' if ss_search.json.records|length == 0
        else '/' + (ss_search.json.records[0].id|string) + '/fields/password'
        }}
    method: >-
        {{ 'POST' if ss_search.json.records|length == 0 else 'PUT' }}
    headers:
        Authorization: "Bearer {{ ss_token }}"
        Content-Type: application/json
    body: >-
        {{
        {
            "secretTemplateId": secret_template_id,
            "folderId": secret_folder_id,
            "siteId": secret_site_id,
            "name": secret_name,
            "items": [
            {"fieldName": "Host", "itemValue": secret_name},
            {"fieldName": "Username", "itemValue": secret_username},
            {"fieldName": "Password", "itemValue": new_password}
            ]
        }
        if ss_search.json.records|length == 0
        else {"value": new_password}
        }}
    body_format: json
    return_content: yes
    delegate_to: localhost
    no_log: True

Finally, we'll update the password on the remote server.

- name: Set root password on target host
    become: yes
    user:
    name: "{{ secret_username }}"
    password: "{{ new_password | password_hash('sha512') }}"
    no_log: True

If the Secret Server API request fails, the playbook will halt, and the remote password will not be changed. If the Secret Server update is successful but the remote password change fails, we could end up with a mismatching password on file. While this isn't ideal, we can access password history for the device in Secret Server and attempt older versions. If we set the remote password first, then the Secret Server password update failed, we wouldn't have a way to recover the value of the remote password to update Secret Server with. This is less desirable.

To spot when errors occur and to incorporate this into our scheduler in Azure Automation, I wrapped the process in a Python script. Note that pySATLogger is a custom module I wrote to extend Python's logging engine to automatically route messages to the HTTP endpoint of our logging platform. In the event that the subprocess fails, an alert is generated, and we have an opportunity to re-run the playbook against whatever system(s) fails to update.

#!/usr/bin/env python3
import pySATLogger
import os
import subprocess
import sys

job_name = "Ansible_Rotate_Passwords"
log_type = "Text"
log_to_console = True
console_level = 'INFO'
log_to_file = False
file_level = 'INFO'
log_retention_days = 0
log_to_monitor = True
monitor_level = 'ERROR'

def main():
    command = [ "/usr/bin/ansible-playbook",
                "-i", 
                "/etc/ansible/inventory_rotate.yml",
                "--vault-password-file", 
                "/.vault",
                "/etc/ansible/play_rotate_passwords.yml",
            ]
    process = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, env=dict(os.environ, ANSIBLE_STDOUT_CALLBACK='json'))

    if process.returncode == 0:
        logger.info(f'Passwords rotated successfully')
        sys.exit(0)
    else:
        logger.error(f'Errors encountered during password rest operation. Review ansible.log for more details. Return code: {process.returncode}')
        sys.exit(1)

if __name__ == '__main__':

    logger = pySATLogger.configure_logging(
        job_name = job_name,
        log_type = log_type,
        log_to_console= log_to_console,
        console_level = console_level,
        log_to_file = log_to_file,
        file_level = file_level,
        log_retention_days = log_retention_days,
        log_to_monitor = log_to_monitor,
        monitor_level = monitor_level,
    )
    main()

At this point, we're ready to schedule the wrapper script to run on our hybrid worker. I'll save the details of our Azure Automation setup for a later post.

Playbook

The whole playbook is below.

---
- name: Rotate root password and push to Secret Server
  hosts: linux
  gather_facts: no
  vars:
    secret_template_id: 101
    secret_folder_id: 101
    secret_site_id: 1
    secret_username: "{{ root_user | default('root') }}"
    secret_name: "{{ inventory_hostname }}"
  vars_files:
  - /vault.yml
  - /vars.yml
  tasks:
    - set_fact:
        new_password: "{{ lookup('community.general.random_string', length=32, special=true) }}"
      no_log: true

    - name: Authenticate to Secret Server
      uri:
        url: "{{ ss_url }}/oauth2/token"
        method: POST
        body_format: form-urlencoded
        body:
          username: "{{ ss_username }}"
          password: "{{ ss_password }}"
          grant_type: password
        return_content: yes
      register: ss_auth
      delegate_to: localhost
      run_once: true
      no_log: True

    - name: Set auth token fact
      set_fact:
        ss_token: "{{ ss_auth.json.access_token }}"
      delegate_to: localhost
      run_once: true
      no_log: True

    - name: Search for existing secret
      uri:
        url: "{{ ss_url }}/api/v2/secrets?filter.searchtext={{ secret_name }}"
        method: GET
        headers:
          Authorization: "Bearer {{ ss_token }}"
        return_content: yes
      register: ss_search
      delegate_to: localhost
      no_log: True

    - name: Create or update secret
      uri:
        url: >-
          {{ ss_url }}/api/v1/secrets{{
            '' if ss_search.json.records|length == 0
            else '/' + (ss_search.json.records[0].id|string) + '/fields/password'
          }}
        method: >-
          {{ 'POST' if ss_search.json.records|length == 0 else 'PUT' }}
        headers:
          Authorization: "Bearer {{ ss_token }}"
          Content-Type: application/json
        body: >-
          {{
            {
              "secretTemplateId": secret_template_id,
              "folderId": secret_folder_id,
              "siteId": secret_site_id,
              "name": secret_name,
              "items": [
                {"fieldName": "Host", "itemValue": secret_name},
                {"fieldName": "Username", "itemValue": secret_username},
                {"fieldName": "Password", "itemValue": new_password}
              ]
            }
            if ss_search.json.records|length == 0
            else {"value": new_password}
          }}
        body_format: json
        return_content: yes
      delegate_to: localhost
      no_log: True

    - name: Set root password on target host
      become: yes
      user:
        name: "{{ secret_username }}"
        password: "{{ new_password | password_hash('sha512') }}"
      no_log: True