Python Mercenaries
When working with salt there is a learning curve present in just understanding the salt filesystem defaults and how different salt feature systems are managed in the filesystem. For example the pillar system and the state system both use .sls files and have a top.sls file in the root of their respective directories. However, the state system is at its core a logic driven instruction set, while the pillar system is intended as a central data system available to the cluster as configured. Two very different parts of salt and yet so many similarities from how a user interacts with it.
This isn't a bad design, I think its good to present users with reusable workflow to provide a lower learning curve, but because these systems aren't well contrasted. This is resulting in a user's experience being an increased learning curve.
For this article we are looking at how different salt features and their filesystems can be adjusted to provide more consistency and/or clarity on the salt master.
If you are using GitFS backend with your salt-master, I still recommend this filesystem structure to be reflected in the git repository.
In the state system there are actually three salt features intertwined. These are the orchestration states, reactor states, and the salt state system. All three have different syntax for their respective salt states but they all share the same salt state id name space.
The default roots for each are as follow:
/srv/salt/reactor/srv/salt/orch/srv/saltA nested file tree with sibling features at first doesn't seem that big of an issue since two of the salt feature's file roots are children directories of the root of the main salt state system. However, for a beginner this does pose some confusion right away.
If they have learned all salt states should be referenced in /srv/salt/top.sls, then should orchestration and reactor states be included in that?
All three systems have different syntax for their states. However, that isn't always obvious or users forget and often try to use syntax from one system in another. This can build up a lot of frustration especially when the user checks syntax and its technically valid from a rendering standpoint. Coupled with this, the error information though accurate is not helpful to non-experts in salt (a.k.a. using salt jargon in the error message).
All three salt features use the same salt state ID namespace. Inside of every salt state file (.sls) you will have logic blocks that are "salt states" and at the top of these is a description ID the user creates to uniquely identify this salt state. The problem is you cant have two salt states with the same ID... across all the files in the salt state system. so your orchestration, reactor, and regular salt states must all be unique. That's mostly easy in the same context, but as you are working on different features its more likely you will over use a state ID causing name space collision.
When I make a salt cluster I always change these roots in the salt-master configuration files to reflect orchestration, reactor, and salt states as sibling features. This doesn't alleviate the namespace ID issue but it does help engineers to think of them as different tools (point 2) and makes it easier for beginner engineers to understand orch and reactor are not regular salt states (point 1).
# /etc/salt/master.d/filesystem.conf file_roots: base: - /srv/salt/state - /srv/salt/reactor - /srv/salt/orch
Don't forget to restart the salt master after adjusting configurations. After restart the salt state system will check for a given salt state a user has requested in the order listed. First checking the state/ directory, then reactor/, and then orch/.
$ systemctl restart salt-master
The salt pillar systems root is in /srv/pillar. Just as with the salt state system, the pillar system is a specific feature. However in some cases pillar is a required dependency for other salt features, and in others it is a supporting feature to add additional value to other salt systems.
The main two features that need (or optionally use) pillar are Proxy minions and Beacons. Both of these can use pillar for instantiation of their respective feature offerings. This can be convoluted though, as these pillar and beacon instantiation .sls files are sharing the same pillar filesystem as pillar files supporting salt states, orch, etc.
I recommend to users to move pillar file roots under the "salt" directory so as to further clarify to engineers this falls under open-salt. Its not the biggest deal in the world, but it often saves one or two guesses in the terminal weekly when an engineer is troubleshooting salt. I further recommend bifurcating pillar roots into sibling directories to separate out the types of pillar roots for each feature that utilizes their corresponding .sls files.
# /etc/salt/master.d/filesystem.conf pillar_roots: base: - /srv/salt/pillar - /srv/salt/beacon - /srv/salt/proxy
Don't forget to restart the salt master after adjusting configurations. After restart the salt pillar system will check for a given salt state a user has requested in the order listed. First checking the pillar/ directory, then beacon/, and then proxy/.
$ systemctl restart salt-master
Salt experts need to remember that the advantage of salt is to abstract and centralize management of many devices, Virtual devices, and applications for users. This centralized control is suppose to be advantageous for engineers over making lots of scripts. But if salt is to hard to navigate and to much to remember all its quarks then it will not be adopted.
The abstraction should not become more difficult then writing the literal equivalent for engineers, otherwise the abstraction isn't beneficial.
A lot of salt users are not full time salt engineers. They are System Administrators, Developers in specific fields, SecOps and DevOps engineers. If they have to be 'experts' in salt in order to leverage it for their projects, infrastructure, and companies then salt will not be a mainstream tool to reach for.