░░ fzysqr By Justin Slattery

This is a cross post from a post I wrote for the MLS Digital Dev Blog. -Justin

Introduction

For those of us who come from traditional sysops backgrounds, learning Salt means un-learning many of our shell scripting habits. Instead of linearly specifying the operations you need a server to perform like this (contrived example, the salt-minion package would ask to install python):

#!/bin/bash
apt-get update
apt-get install python 
apt-get install salt-minion
cp ~/temp/salt-minion-config /etc/salt/minion
chown root.root /etc/salt/minion
chmod 600 /etc/salt/minion
/etc/init.d/salt-minion restart

You instead specify a series of states that will be applied the server. States are evaluated independently from each other and may live in separate files. Each specifies its own dependencies and SaltStack will make sure they get applied in the correct order. The example above translates to something like:

python:
  pkg:
    - installed

salt-minion:
  pkg:
    - installed
    - require:
      - pkg: python

/etc/salt/minion:
  file:
    - managed
    - source: salt://salt/salt-minion-config
    - mode: 600
    - user: root
    - group: root
    - require:
      - pkg: salt-minion

salt-minion:
  service:
    - running
    - watch:
      - file: /etc/salt/minion
    - require:
      - pkg: salt-minion

Ok, so the Salt version is a tad longer. But even for this not-so-realistic example, there are several benefits:

• In the shell script version, the minion will be restarted every time. In the Salt version, the minion is only restarted when the config file is changed.
• It would require a lot more tricky shell script coding if we decided to split the installation of python into a separate file.
• Where does the salt-minion-config file even come from? The shell script version will need it delivered to ~/temp somehow. Salt handles this for us.

Real Power

The real power of Salt starts to become apparent when you realize you can start adding flow control logic to your jinja states. Example to add users to a server:

{% for usr in 'Tom','Dick','Harry' %}
{{ usr }}:
  user:
    - present
    - fullname: {{ usr }}
    - home: True
    - shell: /bin/bash
    # group names for sudo very by platform
    - optional_groups:
      - admin
      - ubuntu
      - wheel
  ssh_auth:
    - present
    - user: {{ usr }} 
    - source: salt://users/keys/{{ usr }}_id_rsa.pub
    - require:
      - user: {{ usr }}
{% endfor %} 

Slick huh? We have a list of usernames and we auto-generate the rest of the jinja file to create each user’s corresponding state. We even use a simple naming convention to get their SSH keys out there!

This is a real example that we used here at MLS Digital to add users to all of our servers. It worked great, but as the team grew, the need to constantly be editing the user list became tedious. Furthermore, we started to find that we didn’t want every developer to have full access to every server.

The next level

We had just recently started using Salt Pillars to achieve better separation between data and code and we thought this would be a great application for pillars. Our first attempt resulted in this:

pillar/top.sls:

'prod_servers*':
  - produsers
'dev_servers*':
  - devusers

salt/top.sls:

'*_servers*':
  - sudoers

pillar/users/produsers.sls:

sudoers_users: ['Tom','Jane','Harry','Sue','Mike','Janet']

pillar/users/devusers.sls:

sudoers_users: ['Dick', 'Tom','Jane','Harry','Sue','Mike','Janet']

salt/users/sudoers.sls:

{% for usr in pillar['sudoers_users'] %}
{{ usr }}:
  user:
    - present
    - fullname: {{ usr }}
    - home: True
    - shell: /bin/bash
    # group names for sudo very by platform
    - optional_groups:
      - admin
      - ubuntu
      - wheel
  ssh_auth:
    - present
    - user: {{ usr }} 
    - source: salt://users/keys/{{ usr }}_id_rsa.pub
    - require:
      - user: {{ usr }}
{% endfor %} 

Jinja templates allow basic flow control structures, such as loops, conditionals, and macros. Pillars allow you to assign chunks of data (think strings and lists) scoped in the same way you scope the states themselves. Pulling the data out of the states allows you to create more dynamic states and achieve better re-usability across your code base. In this example, we are using the same basic sudoers state, but getting different behavior depending on the target server. In case you were wondering, Dick crashed production with some untested code and is on temporary production probation :)

Trouble in paradise

That’s pretty nice! But us OCD programmer types like to keep it DRY and seeing the produsers’ names listed twice makes us start twitching… uncontrollably.

The first thought was to try and aggregate the separate user lists in the pillars themselves. It looked something like:

pillar/users/produsers.sls:

sudoers_users: ['Tom','Jane','Harry','Sue','Mike','Janet']

pillar/users/devusers.sls:

{% if 'sudoers_users' in pillar %}
sudoers_users: pillar['sudoers_users'].extend(['Dick'])
{% else %}
sudoers_users: ['Dick']
{% endif %}

Unfortunately, that doesn’t work and several similar variations failed as well. The pillar constructs are simply not available to be used by the pillars themselves.

The even next-er level

After some research and some experimenting, we realized that between the limitations in the Pillar system and the minimal tools available in Jinja, that we would have to write our Python state. Oh, did I mention you can do that? Yes! All you have to do is prefix the sls file with #!py and Salt will execute everything inside the run() function. The only requirement is that you return a valid Salt high state data structure.

pillar/users/mls.sls:

users_add_mls: ['Tom','Jane','Harry','Sue','Mike','Janet']

pillar/users/probation.sls:

users_add_probation: ['Dick']

salt/users/sudoers.sls:

#!py

def run():
    '''
    Manage sudo enabled user adds/removes
    '''

    users_root = []
    [users_root.extend(value) for key,value in pillar.iteritems() if key.startswith('users_add')]

    generated_user_hsd = {}

    for user in users_root:
        generated_user_hsd[user] = {}

        generated_user_hsd[user]['user'] = [
            'present',
            {'fullname' : user},
            {'home' : True},
            {'shell' : '/bin/bash'},
            {'optional_groups' : ['admin','ubuntu','wheel']}
        ]
        generated_user_hsd[user]['ssh_auth'] = [
            'present',
            {'user' : user},
            {'source' : 'salt://users/keys/' + user + '_id_rsa.pub'},
            {'require' : [{'user': user}]}
        ]

    return generated_user_hsd

It certainly is refreshing to have the full power of Python available! Our new state is now smart enough to search for available pillars with users_add as part of their name, aggregate the results, and generate the correct high state data structure for Salt to process. This version above is very similar to what we currently run in production and it works quite well.

Wrapping it up

We started with shell scripting, progressed to generated jinja states, and ended up with custom python states. What fun! A small warning though. Writing a custom python state should be your last resort. There is a major tradeoff between readbility and keeping things DRY. We try to stick to jinja states unless the benefits of the custom code are significant. As you become more familar with Salt, you will start to get a feel for what level of complexity is required for various tasks.

Remember, you’re DevOps now, not some unix cowboy. This is code. You’re a programmer and all the best practice coding rules apply.