STORIES
This blog is written by Rogier van der Heide, Engineer bij Triple.
One of the departments of Triple is Technology Operations (Techops), which I’m part of. The purpose of this department is to manage servers and cloud-hosting solutions for our clients. We handle things such as patching, monitoring, change management and incident handling. In order to provide these services, we use several types of tooling. One of those tools is named Saltstack. This blog will elaborate on what it is, and how and why we use it.
Wat is Saltstack?
Salt, also known as Saltstack, is a Python application created roughly 11 years ago. Its purpose is to automate software configuration using state management. It supports Infrastructure-as-code, allowing you to use Git to store your states. Using YAML templates enriched with Jinja code, you can write specific states which will make all the adjustments you wish on the particular targets you specify.
In layman’s terms, this means that we can write how we want to install software on our Virtual Machines and use Git to store these templates so that we can re-use them over and over to make sure that our servers are configured the same way.
Saltstack is essential in managing all our Virtual Machines and ensures that different environments (DEV/ACC/PRD) for the same solution are similar, with only required differences.
There is a multitude of different software solutions which achieve the same effect as Saltstack, such as Ansible, Puppet or Chef. We chose Saltstack due to two reasons: The first is its network behaviour, where you have a Salt master and, for each target, a minion. In the case of Saltstack, it’s not the master who communicates with the minions but the other way around. The minion requests work from the master.
This means that the minions initiate the communication and is considered outbound traffic. With a large number of different customers, we usually run into issues with inbound traffic due to the use of firewalls. However, the cases where we run into outbound traffic being blocked by firewalls are quite slim. Hence we chose Saltstack, as it’s easier to set up initially.
The second reason is that Saltstack has a relationship between Master and Minion. This offers the option to target minions based on their characteristics actively. Due to our large fleet of diverse types of servers and customers, this fits our wishes perfectly.
Our Saltstack setup
We’ve set up Saltstack using a MongoDB for pillar data and attached it to a git repository where we store all our states.
Saltstack has three data sources which you use to manipulate its behaviour. Those being:
State files
Pillar data
Grains
State files are the YAML templates which we store in git and carry the extension .sls. These contain state modules which specify a particular behaviour. For example file.managed. Which manages a specific file on a server and ensures that a specific file on a Virtual Machine is equal to that specified in this state. Or pkg.installed which checks if a package is installed through the default package manager (e.g. yum, apt) and installs any missing packages from the list. There is a multitude of state modules to be found on the Saltstack documentation site. In order to add logic to these state files, one can use Jinja to make them more dynamic and incorporate data from either Pillar or Gains.
Pillar data contains information which we add to a minion. It comes from a MongoDB database. We use this data to give extra information to the minion, which we then use in our states or to target a specific group of minions. Think of information such as to which project they belong. Who their customer is. But also state-specific information as to which static IP we wish to be configured on the NIC, or the timezone to which the server’s date/time needs to be configured.
The minion itself primarily creates grain data. It contains information such as the OS the minion is running on, what kind of CPU the machine has, or the hostname. It can also be used to store certain information. For example, setting certain flags to indicate that a specific package of software is installed so that we don’t install it every time we run the same state (if it’s not installed through the standard package manager)
A simple example
Let’s take a simple example of a state file which we use to install a pretty standard piece of software used when dealing with websites, namely Apache.
{% set customer = salt.pillar.get( 'cmdb:customer:label' ).lower() %} {% set project = salt.pillar.get( 'cmdb:project:label' ).lower() %} {% set prefix = 'httpd' %} {{prefix}}-packages: pkg.installed: - names: - httpd - mod_ssl - httpd-itk {{prefix}}-httpd-file: file.managed: - name: /etc/httpd/conf/httpd.conf - source: salt://customer/{{customer}}/{{project}}/web/files/httpd/httpd.conf - user: root - group: root - mode: 644
In the given example, we have a syntax of YAML mixed with Jinja. This gets rendered by Salt before applying the YAML. So let’s say we have the following data in our pillar: - cmdb.customer.label = example - cmdb.project.label = foo
The resulting YAML will look as follow:
httpd-packages: pkg.installed: - names: - httpd - mod_ssl - httpd-itk httpd-httpd-file: file.managed: - name: /etc/httpd/conf/httpd.conf - source: salt://customer/example/foo/web/files/httpd/httpd.conf - user: root - group: root - mode: 644
Just using these two state modules, we will achieve the following:
Install Apache with mod_ssl and ITK using the default package manager (in our case, yum).
Push our default Apache config.
This handles the standard, a webserver with your generic config. Next comes how to deal with your website configs. Now, for simplicity's sake, I will just show how to handle the individual website configs. More is required, such as the creation of folders and certificate handling, which makes it too complex for this example.
{% for dom in pillar.custom.domains %} {% set domain = pillar['custom']['domains'][dom] %} {{domain.name}}-user: user.present: - name: {{domain.name}} - home: /home/domains/{{domain.name}}/ - createhome: false - usergroup: true - password: {{domain.password}} {{domain.name}}-vhost-file: file.managed: - name: /etc/httpd/conf.d/{{domain.name}}.conf - source: salt://customer/{{customer}}/{{project}}/files/httpd/vhost.conf.jinja - user: root - group: root - mode: 644 - template: jinja - context: domain: {{domain}} - require: - {{domain.name}}-user {% endfor %}
The Pillar data that belongs to this part is the following: "custom" : { "domains" : { "www_example_com" : { "name" : "www.example.com", "password" : "$1$snip" } } },
Combining the state with the pillar data will result in a loop which will create a vhost config for each domain we define in the custom.domains pillar. Now combine this with states who create the proper folders, and you can create a complete config for each website you define. Each domain will then have its own home folder to push data to, belonging to its own user, and Apache will be configured with the proper hostnames configured.
A living CMDB
You’ve now seen a simple but powerful example of what you can do with Saltstack in regard to installing and maintaining software.
But Saltstack offers another feature which is really essential when maintaining a large, diverse fleet of servers. Due to the grains being set based on the type of server you have and the fact that you can request the status of all minions from a single master, you essentially have a living CMDB.
Through the use of compounds, you can quickly request, for example, how many Ubuntu 20 servers you have. Or see which version OpenSSL you have installed.
$ salt -C "G@os:Ubuntu and I@cmdb:customer:label:example" cmd.run 'openssl version' acc-server.example.local: OpenSSL 1.1.1f 31 Mar 2020 prd-server.example.local: OpenSSL 1.1.1f 31 Mar 2020 ------------------------------------------- Summary ------------------------------------------- # of minions targeted: 2 # of minions returned: 2 # of minions that did not return: 0 # of minions with errors: 0 -------------------------------------------
Another feature which is really handy is to assign servers to specific maintenance groups so that you can patch all your non-production servers during the day and only patch your production servers during the night. You can simultaneously patch all your servers with a simple command if you want.
A farewell word
As you can see in the simple examples above, Saltstack offers a wide range of possibilities when dealing with Virtual Machines.
Using state files and git allows you to ensure that your environments are similar, and making changes can quickly be done without fear of getting differences.
And the ability to update all your servers at the same time will drastically reduce the time you need in order to patch a large number of servers.
Once you start creating more elaborate state files and standardizing your pillar data, you can use the same templates over and over so that you require fewer hours to set up any new Virtual Machine. And if you build in the necessary configuration checks, deploying changes will be even safer.
All in all, Saltstack made our work a lot easier, a lot more standardized, and reduced incidents since the time we started using it.
Explore the Triple Universe
Learn more about Triple, our culture or check out our vacancies.
