Mastering Configuration Management with Salt: The Essentials

Introduction to Configuration Management

Configuration management is a process for maintaining computer systems, servers, applications, network devices, and other IT components in a desired state. It’s a way to help ensure that a system performs as expected, even after many changes are made over time.

Why Salt?

SaltStack, also known as Salt, is a configuration management and orchestration tool. It uses a central repository to provision new servers and other IT infrastructure, to make changes to existing ones, and to install software in IT environments, including physical and virtual servers, as well as the cloud.

Features of Salt:

The Salt system is a Python-based, open-source remote execution framework for configuration management, automation, provisioning, and orchestration.

Salt delivers a dynamic communication bus for infrastructure to leverage in:

• Remote execution.

• Configuration management.

• Automation and orchestration.

Salt supports the Infrastructure as Code approach to deployment and datacenter management.

Getting Started with Salt

Before you start the installation, check the system requirements to ensure your platform is supported in the latest version of Salt and open the required network ports. Ensure you also have the correct permissions to install packages on the targeted nodes.

• Install the salt-master service on the node that will manage your other nodes, meaning it will send commands to other nodes. Then, install the salt-minion service on the nodes that the Salt master will manage.

• For Linux-based operating systems, the recommended installation method is to use the bootstrap script or you can manually install Salt using the instructions for each operating system.

• For Windows or macOS operating systems, you need to download and run the installer file for that system.

• Configure the Salt minions to add the DNS/hostname or IP address of the Salt master they will connect to. You can add additional configurations to the master and minions as needed.

• Start the service on the master, then the minions.

• Accept the minion keys after the minion connects.

• Verify that the installation was successful by sending a test ping.

• Install third-party Python dependencies needed for specific modules.

Setting up SALT Master

Turning on the Salt Master is easy -- just turn it on! The default configuration is suitable for the vast majority of installations. The Salt Master can be controlled by the local Linux/Unix service manager:

• On Systemd-based platforms (newer Debian, openSUSE, Fedora):

systemctl start salt-master

• On Upstart-based systems (Ubuntu, older Fedora/RHEL):

service salt-master start

• On SysV Init systems (Gentoo, older Debian etc.):

/etc/init.d/salt-master start

• Alternatively, the Master can be started directly on the command line:

salt-master -d

• The Salt Master can also be started in the foreground in debug mode, thus greatly increasing the command output:

salt-master -l debug

Setting Up the SALT Minion

The Salt Minion can operate with or without a Salt Master. This walk-through assumes that the minion will be connected to the master.

start the minion in the same way as the master; with the platform init system or via the command line directly:

As a daemon:

salt-minion -d

In the foreground in debug mode:

salt-minion -l debug

Writing Salt States

You can write a state file in the Config–> File Server section of SaltStack Config and list the packages that you want to apply to the Windows systems. Basically just enter the name of the sls file without the sls extension. Create a job to call the state file you created in the File Server.

1. Set up the return dictionary and perform any necessary input validation (type checking, looking for use of mutually-exclusive arguments, etc.).

def myfunc():

ret = {"name": name, "result": False, "changes": {}, "comment": ""}

if foo and bar:

ret["comment"] = "Only one of foo and bar is permitted"

return ret

2. Check if changes need to be made. This is best done with an information-gathering function in an accompanying execution module. The state should be able to use the return from this function to tell whether or not the minion is already in the desired state.

result = salt["modname.check"](name)

3. If step 2 found that the minion is already in the desired state, then exit immediately with a True result and without making any changes.

def myfunc():

if result:

ret["result"] = True

ret["comment"] = "{0} is already installed".format(name)

return ret

4. If step 2 found that changes do need to be made, then check to see if the state was being run in test mode (i.e. with test=True). If so, then exit with a None result, a relevant comment, and (if possible) a changes entry describing what changes would be made.

def myfunc():

if opts["test"]:

ret["result"] = None

ret["comment"] = "{0} would be installed".format(name)

ret["changes"] = result

return ret

5. Make the desired changes. This should again be done using a function from an accompanying execution module. If the result of that function is enough to tell you whether or not an error occurred, then you can exit with a False result and a relevant comment to explain what happened.

result = salt["modname.install"](name)

6. Perform the same check from step 2 again to confirm whether or not the minion is in the desired state. Just as in step 2, this function should be able to tell you by its return data whether or not changes need to be made.

ret["changes"] = salt["modname.check"](name)

As you can see here, we are setting the changes key in the return dictionary to the result of the modname.check function (just as we did in step 4). The assumption here is that the information-gathering function will return a dictionary explaining what changes need to be made. This may or may not fit your use case.

7. Set the return data and return!

def myfunc():

if ret["changes"]:

ret["comment"] = "{0} failed to install".format(name)

else:

ret["result"] = True

ret["comment"] = "{0} was installed".format(name)

return ret

Salt Best Practices

GENERAL RULES

1. Modularity and clarity should be emphasized whenever possible.

2. Create clear relations between pillars and states.

3. Use variables when it makes sense but don't overuse them.

4. Store sensitive data in pillar.

5. Don't use grains for matching in your pillar top file for any sensitive pillars.

STRUCTURING STATES AND FORMULAS

When structuring Salt States and Formulas it is important to begin with the directory structure. A proper directory structure clearly defines the functionality of each state to the user via visual inspection of the state's name.

Reviewing the MySQL Salt Formula it is clear to see the benefits to the end-user when reviewing a sample of the available states:

/srv/salt/mysql/files/

/srv/salt/mysql/client.sls

/srv/salt/mysql/map.jinja

/srv/salt/mysql/python.sls

/srv/salt/mysql/server.sls

This directory structure would lead to these states being referenced in a top file in the following way:

base:

'web*':

• mysql.client

• mysql.python

  'db*':

• mysql.server

This clear definition ensures that the user is properly informed of what each state will do.

The usage of a clear top-level directory as well as properly named states reduces the overall complexity and leads a user to both understand what will be included at a glance and where it is located.

STRUCTURING PILLAR FILES

• Pillars are used to store secure and insecure data pertaining to minions. When designing the structure of the /srv/pillar directory, the pillars contained within should once again be focused on clear and concise data which users can easily review, modify, and understand.

• The /srv/pillar/ directory is primarily controlled by top.sls. It should be noted that the pillar top.sls is not used as a location to declare variables and their values. The top.sls is used as a way to include other pillar files and organize the way they are matched based on environments or grains.

• An example top.sls may be as simple as the following:

/srv/pillar/top.sls:

base:

'*':

• packages

Any number of matchers can be added to the base environment. For example, here is an expanded version of the Pillar top file stated above:

/srv/pillar/top.sls:

base:

'*':

• packages

  'web*':

• apache

• vim

It is clear to see through these examples how the top file provides users with power but when used incorrectly it can lead to confusing configurations. This is why it is important to understand that the top file for pillar is not used for variable definitions.

Each SLS file within the /srv/pillar/ directory should correspond to the states which it matches.

VARIABLE FLEXIBILITY

• Salt allows users to define variables in SLS files. When creating a state variables should provide users with as much flexibility as possible. This means that variables should be clearly defined and easy to manipulate and that sane defaults should exist in the event a variable is not properly defined.

• Although it is possible to set variables locally, this is generally not preferred:

/srv/salt/apache/conf.sls:

{% set name = 'httpd' %}

{% set tmpl = 'salt://apache/files/httpd.conf' %}

include:

• apache

apache_conf:

file.managed:

• name: {{ name }}

• source: {{ tmpl }}

• template: jinja

• user: root

• watch_in:

  • service: apache

This flexibility provides users with a centralized location to modify variables, which is extremely important as an environment grows.

MODULARITY WITHIN STATES

• Ensuring that states are modular is one of the key concepts to understand within Salt. When creating a state a user must consider how many times the state could be re-used, and what it relies on to operate.

• It begins to address the referencing by using - name, as opposed to direct ID references:

/srv/salt/apache/init.sls:

apache:

pkg.installed:

• name: httpd

  service.running:

• name: httpd

• enable: True

apache_conf:

file.managed:

• name: /etc/httpd/httpd.conf

• source: salt://apache/files/httpd.conf

• template: jinja

• watch_in:

  • service: apache

The above init file it has several issues which lead to a lack of modularity. There is also still the concern of the configuration file data living in the same state as the service and package.

In the next example steps will be taken to begin addressing these issues. Starting with the addition of a map.jinja file (as noted in the Formula documentation), and modification of static values:

/srv/salt/apache/map.jinja:

{% set apache = salt['grains.filter_by']({

'Debian': {

'server': 'apache2',

'service': 'apache2',

'conf': '/etc/apache2/apache.conf',

},

'RedHat': {

'server': 'httpd',

'service': 'httpd',

'conf': '/etc/httpd/httpd.conf',

},

}, merge=salt['pillar.get']('apache:lookup')) %}

STORING SECURE DATA

• Secure data refers to any information that you would not wish to share with anyone accessing a server. This could include data such as passwords, keys, or other information.

• As all data within a state is accessible by EVERY server that is connected it is important to store secure data within pillar. This will ensure that only those servers which require this secure data have access to it. In this example, a use can go from an insecure configuration to one which is only accessible by the appropriate hosts:

/srv/salt/mysql/testerdb.sls:

testdb:

mysql_database.present:

• name: testerdb

/srv/salt/mysql/user.sls:

include:

• mysql.testerdb

testdb_user:

mysql_user.present:

• name: frank

• password: "test3rdb"

• host: localhost

• require:

  • sls: mysql.testerdb

Many users would review this state and see that the password is there in plain text, which is quite problematic. It results in several issues which may not be immediately visible.