A Salty API

Extending Salt Master Access with CherryPy

written by Alan Cugler, edited by Benjamin Echelmeier on 2022-10-17

Why an API?

Integrating CherryPy will enable us to automate our Salt masters.

The normal workflow for executing Salt commands involves an SSH session into a Salt master. This suits Salt's original purpose of controlling minions from a master perfectly. If, however, you want to automate masters from afar (e.g., a remote web server or client, managing multiple masters), this can grow cumbersome. CherryPy allows us to utilize web API calls in place of SSH, vastly simplifying the automation process of one or more masters.

In pursuit of full-automation, SSH and private-key management rapidly grows in complexity. In contrast, the REST API provided by CherryPy exists as a far more standard interface for extension.

Instruments of Choice

Salt is a configuration management tool composed of Salt masters and Salt minions. This blog post assumes a basic knowledge of Salt fundamentals. For a brief overview of Salt, checkout their docs.

CherryPy is “a minimalist Python web framework.” For our use, we will be taking advantage of the REST API functionality, which is already provided as a Salt module. Once updated for optimal performance, it can be activated to provide a WSGI for managing Salt.

Traditional Salt workflow

As an example, we will execute a simple test.ping on a couple of minions

$ # SSH into SaltMaster
$ ssh -i ~/.ssh/<KEY> ubuntu@<saltmaster_hostname>

$ # Execute minion commands
$ sudo salt <minion-1> test.ping
minion-1:
    True

$ sudo salt <minion-2> test.ping
minion-2:
    True

This is simple and effective for a user to perform, but it rapidly grows clunky if we attempt to do the same thing with a script.

#!/bin/bash
ssh -i ~/.ssh/<KEY> ubuntu@<saltmaster_hostname> 'sudo salt <minion-1> test.ping && sudo salt <minion-2> test.ping'

```console minion-1: True minion-2: True ``` This can rapidly grow nastier as more complex Salt commands and sequences of commands must be nested inside of ssh sessions. Additionally, it is difficult to implement such solutions outside of bash, such as in a python script.

Example Python

This would be the same implementation using the python file at the end of this guide This is obviously cleaner and more readable than commands nested within ssh

# test_ping.py
master = API_Master(URL, LOGIN_CREDENTIALS)
master.local_cmd("test.ping", target="minion-1")
master.local_cmd("test.ping", target="minion-2")

Rest_cherrypy workflow

With CherryPy, once we have acquired a token we can directly execute or script any commands required.

$ # API call to get auth token
$ curl -sSk https://<saltmaster_hostname>:8000/login \
    -H 'Accept: application/x-yaml' \
    -d username=saltdev \
    -d password=saltdev \
    -d eauth=pam

$ # API call using token from prior call. Replace
$ curl -sSk https://<saltmaster_hostname>:8000 \
    -H 'Accept: application/x-yaml' \
    -H 'X-Auth-Token: 1234abcd1234abcd1234abcd1234abcd1234abcd' \
    -d client=local \
    -d tgt='minion-1' \
    -d fun=test.ping

Responses

return:
- eauth: pam
  expire: 1665126040.9847946
  perms:
  - .*
  - '@wheel'
  - '@runner'
  - '@jobs'
  start: 1665082840.9847934
  token: 5d8c0bcb81065088815147a2e093f83418dd962d
  user: saltdev

return:
- minion-1: true

Steps to configure SaltMaster with CherryPy

Perform these steps on each Salt Master where you want to make API calls.

##1. Establish Dependencies

The following dependencies are required for properly configuring CherryPy on each Salt master.

Note: Be very careful when using Salt’s built-in pip module; pip.upgrade often breaks masters by upgrading version-locked dependencies. Always take care and update packages ad hoc using :

$ pip.install <pkg> upgrade=True.

Install salt-api

Redhat

  $ yum install salt-api -y

Ubuntu

  $ apt install salt-api -y
  $ apt install python3-pip

Install remaining dependencies

$ salt local-minion pip.install pip upgrade=True
$ salt local-minion pip.install pyOpenSSL
$ salt local-minion pip.install cryptography==35.0
$ salt local-minion pip.install CherryPy upgrade=True

2. Configure Masters:

The following command will now execute correctly with Salt’s updated dependencies from the previous step.

$ salt-call --local tls.create_self_signed_cert

CherryPy’s PAM authentication requires a non-root user. Add a user and password for API authentication. We’re using saltdev for both the user and password in our example. This is unsecure for production environments.

useradd **saltdev**
passwd **saltdev**

Enable PAM on the Salt master by creating a new configuration file in the /etc/salt/master.d/ directory and adding the following code block. Replace saltdev with the username you added in the previous step.

$ echo "external_auth:
  pam:
    **saltdev**:
      - .*
      - '@wheel'   # allow access to all wheel modules
      - '@runner'  # allow access to all runner modules
      - '@jobs'    # allow access to the jobs runner and/or wheel module
" > /etc/salt/master.d/auth.conf

Enable CherryPy on the Salt master by creating a new configuration file in the /etc/salt/master.d/ directory and adding the following code block. The first step, salt-call --local tls.create_self_signed_cert should have created SSL keys with the default paths indicated below. If you move these keys, make sure to correct these paths.

$ echo "rest_cherrypy:
  port: 8000
  ssl_crt: /etc/pki/tls/certs/localhost.crt
  ssl_key: /etc/pki/tls/certs/localhost.key
" > /etc/salt/master.d/api.conf

Restart both the salt-master and salt-api services to apply the new configuration.

$ systemctl restart salt-master
$ systemctl restart salt-api

3. Execute Salt Command from API:

Auth Call: Collect the User Token

Authenticate using the PAM username and password

$ curl -sSk https://<saltmaster_hostname>:8000/login \
    -H 'Accept: application/x-yaml' \
    -d username=saltdev \
    -d password=saltdev \
    -d eauth=pam

This should return something like the following

return:
- eauth: pam
  expire: 1657430221.5378942
  perms:
  - .*
  - '@wheel'
  - '@runner'
  - '@jobs'
  start: 1657387021.5378923
  token: 1234abcd1234abcd1234abcd1234abcd1234abcd
  user: saltdev

Take note of the user token returned to you in the yaml. You must replace following example tokens with the one you have received. It will expire in 24 hours.

Execute a Salt Command

Using the token from the auth response, run your Salt command. This directly exposes Salt's Python API, with arguments passed in header and data flags. A brief guide is given below, and full documentation can be found here. An example in python is given at the end of this blog.

-H 'Accept: application/<RETURN>' determines the response format. x-yaml returns yaml, and json returns json.
-d client=<CLIENT> determines where the command is invoked. Most use cases will use local to issue commands to minions.
-d tgt='<TARGET>' determines the target minion(s). minion-1 will target a minion with id minion-1, and * will target all minions.
-d fun=<FUNCTION> will execute the given function, such as test.ping.
-d arg='<ARG>' will pass the given string as an argument to the given function.

Sample 1: test.ping

As an example, run test.ping on all minions (‘*’)

$ curl -sSk https://<saltmaster_hostname>:8000 \
    -H 'Accept: application/x-yaml' \
    -H 'X-Auth-Token: 1234abcd1234abcd1234abcd1234abcd1234abcd' \
    -d client=local \
    -d tgt='*' \
    -d fun=test.ping

```yaml return: - minion-0: true minion-1: true ```

Sample 2: cmd.run_all

As an example, execute a basic bash script, /root/script.sh, on minion-1 using cmd.run_all.

Note on cmd.run_all

cmd.run_all will provide a retcode for errors whereas cmd.run will not.

# /root/script.sh
touc /root/i_was_here.txt

Use ssh to create this file on your minion, or challenge yourself to create it using your API

$ curl -sSk https://<hostname>:8000 \
    -H 'Accept: application/json' \
    -H 'X-Auth-Token: 1234abcd1234abcd1234abcd1234abcd1234abcd'\
    -d client=local \
    -d tgt=minion-1 \
    -d fun=cmd.run_all \
    -d arg='bash /root/script.sh'

Response:

Note: /root/script.sh contains a deliberate error to demonstrate retcode functionality of cmd.run_all

{"return": [{"minion-1": {"pid": 16306, "retcode": 127, "stdout": "", "stderr": "bash: /root/script.sh: No such file or directory"}}]}

Correcting this will simply remove the stderr response

{"return": [{"minion-1": {"pid": 16332, "retcode": 0, "stdout": "", "stderr": ""}}]}

Moving On

Now that you have a web API, you can automate the token reclamation and implement scripts across one or multiple masters, as suits your needs. You might find something like this a helpful starting point:

# salty_api.py
import requests
import yaml

# Store these in some sort of config file. NEVER store creds in public repositories.
URL = "https://<hostname>:8000"
LOGIN_CREDENTIALS = {"username": "saltdev", "password": "saltdev"}


class API_Master:
    def __init__(self, url, login_creds):
        self.url = url
        self.login_creds = login_creds
        self._token = None

    def _new_pam_token(self):
        """Returns a PAM authentication token"""
        headers = {"Accept": "application/json"}
        data = {**self.login_creds, "eauth": "pam"}
        response = requests.post(
            self.url + "/login", headers=headers, data=data, verify=False
        )
        return response.json()["return"][0]["token"]

    def set_pam_token(self):
        """Renews login token"""
        self._token = self._new_pam_token()

    def local_cmd(self, cmd, arg=None, target="*", retry=True, return_yaml=False):
        "runs a command on the local client"
        if not self._token:
            self.set_pam_token()
        headers = {
            "Accept": "application/json",
            "X-Auth-Token": self._token,
            "Content-Type": "application/x-www-form-urlencoded",
        }
        data = {
            "client": "local",
            "tgt": target,
            "fun": cmd,
        }
        if arg:
            data["arg"] = arg
        try:
            response = requests.post(self.url, headers=headers, data=data, verify=False)
            assert response.status_code == 200
            resp = response
        except Exception as e:
            if retry:
                resp = self.local_cmd(target=target, cmd=cmd, arg=arg, retry=False)
            else:
                raise e
        if return_yaml:
            return yaml.dump(resp.json())
        return resp

You can use this inside a package to build a simpl script. Make sure to create an empty init.py.

# leave_a_mark.py
import requests

from salty_api import LOGIN_CREDENTIALS, URL, API_Master


def leave_a_mark():
    """
    This is just a sample run. A more thorough implementation should store the
    token and only reinitialize when needed.
    """
    # Silence warning on unsigned cert
    requests.packages.urllib3.disable_warnings()
    master = API_Master(URL, LOGIN_CREDENTIALS)
    leave_a_mark_cmd = 'echo "echo \\\'Hello from CherryPy API\\\'" > /root/script.sh ;\
             echo "touch /root/i_was_here.txt" >> /root/script.sh'
    print(master.local_cmd("cmd.run_all", leave_a_mark_cmd, return_yaml=True))
    print(master.local_cmd("cmd.run_all", "bash /root/script.sh", return_yaml=True))


if __name__ == "__main__":
    leave_a_mark()

Then run the script!

return:
- minion-0:
    pid: 142743
    retcode: 0
    stderr: ''
    stdout: ''
  minion-1:
    pid: 10413
    retcode: 0
    stderr: ''
    stdout: ''

return:
- minion-0:
    pid: 142761
    retcode: 0
    stderr: ''
    stdout: '''Hello from CherryPy API'''
  minion-1:
    pid: 10418
    retcode: 0
    stderr: ''
    stdout: '''Hello from CherryPy API'''

Now that you're rolling, refer to the documentation to help solve any unique problems you have. Good luck!

Wrestling with Salt? Reach out to see how we can help.