Alexander Weidinger 5952aa24aa feat(unbound): use TOFS for configuration

2021-04-19 22:35:32 +02:00

20 KiB

Raw Blame History

`map.jinja`: gather formula configuration values

The documentation explains the use of a map.jinja to gather parameters values for a formula.

As pillars are rendered on the Salt master for every minion, this increases the load on the master as the pillar values and the number of minions grows.

As a good practice, you should:

store non-secret data in YAML files distributed by the fileserver
store secret data in:
- pillars (and look for the use of something like pillar.vault)
- SDB (and look for the use of something like sdb.vault)

Current best practice is to let map.jinja handle parameters from all sources, to minimise the use of pillars, grains or configuration from sls files and templates directly.

Table of Contents

For formula users

Quick start: configure per role and per DNS domain name values

We will see a quick setup to configure the unbound formula for different DNS domain names and several roles.

For this example, I'll define 2 kinds of fileserver sources:

formulas git repositories with hard-coded version reference to avoid breaking my setup randomly at upstream update. they are the last sources where files are looked up
parameters of the formulas in the file backend roots

Configure the fileserver backends

I configure the fileserver backends to serve:

files from roots first
gitfs repositories last

Create the file /etc/salt/master.d/fileserver.conf and restart the master:

---
##
## file server
##
fileserver_backend:
  # parameters values and override
  - roots
  # formulas
  - gitfs

# The files in this directory will take precedence over git repositories
file_roots:
  base:
    - /srv/salt

# List of formulas I'm using
gitfs_remotes:
  - https://github.com/saltstack-formulas/template-formula.git:
    - base: v4.1.1
  - https://github.com/saltstack-formulas/openssh-formula.git:
    - base: v2.0.1
...

Create per DNS configuration for `unbound` formula

Now, we can provides the per DNS domain name configuration files for the unbound formulas under /srv/salt/unbound/parameters/.

We create the directory for dns:domain grain and we add a symlink for the domain grain which is extracted from the minion id:

mkdir -p /srv/salt/unbound/parameters/dns:domain/
ln -s dns:domain /srv/salt/unbound/parameters/domain

We create a configuration for the DNS domain example.net in /srv/salt/unbound/parameters/dns:domain/example.net.yaml:

---
values:
  config: /etc/template-formula-example-net.conf
...

We create another configuration for the DNS domain example.com in the Jinja YAML template /srv/salt/unbound/parameters/dns:domain/example.com.yaml.jinja:

---
values:
  config: /etc/template-formula-{{ grains['os_family'] }}.conf
...

Create per role configuration for `unbound` formula

Now, we can provides the per role configuration files for the unbound formulas under /srv/salt/unbound/parameters/.

We create the directory for roles:

mkdir -p /srv/salt/unbound/parameters/roles

We will define 2 roles:

unbound/server
unbound/client

We create a configuration for the role unbound/server in /srv/salt/unbound/parameters/roles/unbound/server.yaml:

---
values:
  config: /etc/template-formula-server.conf
...

We create another configuration for the role unbound/client in /srv/salt/unbound/parameters/roles/unbound/client.yaml:

---
values:
  config: /etc/template-formula-client.conf
...

Enable roles and the `dns:domain` and `domain` grains for `map.jinja`

We need to redefine the sources for map.jinja to load values from our new configuration files, we provide a global configuration for all our minions.

We create the global parameters file /srv/salt/parameters/map_jinja.yaml:

---
values:
  sources:
    # default values
    - "Y:G@osarch"
    - "Y:G@os_family"
    - "Y:G@os"
    - "Y:G@osfinger"
    - "C@{{ tplroot ~ ':lookup' }}"
    - "C@{{ tplroot }}"

    # Roles activate/deactivate things
    # then thing are configured depending on environment
    # So roles comes before `dns:domain`, `domain` and `id`
    - "Y:C@roles"

    # DNS domain configured (DHCP or resolv.conf)
    - "Y:G@dns:domain"

    # Based on minion ID
    - "Y:G@domain"

    # default values
    - "Y:G@id"
...

The syntax is explained later at Sources of configuration values.

Bind roles to minions

We associate roles grains to minion using grains.append.

For the servers:

salt 'server-*' grains.append roles unbound/server

For the clients:

salt 'client-*' grains.append roles unbound/client

Note

Since we used Y:C@roles, map.jinja will do a salt['config.get']('roles') to retrieve the roles so you could use any other method to bind roles to minions (pillars or SDB) but grains seems to be the preferred method.

Note for Microsoft Windows systems

If you have a minion running under windows, you can't use colon : as a delimiter for grain path query (see bug 58726) in which case you should use an alternate delimiter:

Modify /srv/salt/parameters/map_jinja.yaml to change the query for dns:domain to define the alternate delimiter:

---
values:
  sources:
    # default values
    - "Y:G@osarch"
    - "Y:G@os_family"
    - "Y:G@os"
    - "Y:G@osfinger"
    - "C@{{ tplroot ~ ':lookup' }}"
    - "C@{{ tplroot }}"

    # Roles activate/deactivate things
    # then thing are configured depending on environment
    # So roles comes before `dns:domain`, `domain` and `id`
    - "Y:C@roles"

    # DNS domain configured (DHCP or resolv.conf)
    - "Y:G:!@dns!domain"

    # Based on minion ID
    - "Y:G@domain"

    # default values
    - "Y:G@id"
...

And then, rename the directory:

mv /srv/salt/unbound/parameters/dns:domain/  '/srv/salt/unbound/parameters/dns!domain/'

Format of configuration YAML files

When you write a new YAML file, note that it must conform to the following layout:

a mandatory values key to store the configuration values
two optional keys to configure the use of salt.slsutil.merge
- an optional strategy key to configure the merging strategy, for example strategy: 'recurse', the default is smart
- an optional merge_lists key to configure if lists should be merged or overridden for the recurse and overwrite strategy, for example merge_lists: 'true'

Here is a valid example:

---
strategy: 'recurse'
merge_lists: 'false'
values:
  pkg:
    name: 'some-package'
  config: '/path/to/a/configuration/file'
...

Using Jinja2 YAML template

You can provide a Jinja2 YAML template file with a name suffixed with .yaml.jinja, it must produce a YAML file conform to the Format of configuration YAML files, for example:

---
strategy: 'overwrite'
merge_lists: 'true'
values:
{%- if grains["os"] == "Debian" %}
  output_dir: /tmp/{{ grains["id"] }}
{%- endif %}
...

Sources of configuration values

The map.jinja file aggregates configuration values from several sources:

YAML files stored in the fileserver
pillars
grains
configuration gathered with salt['config.get']

For the values loaded from YAML files, map.jinja will automatically try to load a Jinja2 template with the same name as the YAML file with the addition of the .jinja extension, for example foo/bar/quux.yaml.jinja.

After loading values from all sources, it will try to include the salt://parameters/post-map.jinja Jinja file if it exists which can post-process the mapdata variable.

Configuring `map.jinja` sources

The map.jinja file uses several sources where to lookup parameter values. The list of sources can be configured in two places:

globally
1. with a plain YAML file salt://parameters/map_jinja.yaml
2. with a Jinja2 YAML template file salt://parameters/map_jinja.yaml.jinja
per formula
1. with a plain YAML file salt://{{ tplroot }}/parameters/map_jinja.yaml
2. with a Jinja2 YAML template file salt://{{ tplroot }}/parameters/map_jinja.yaml.jinja

Note

The map.jinja configuration files must conform to the format of configuration YAML files.

Each source definition has the form [<TYPE>[:<OPTION>[:<DELIMITER>]]@]<KEY> where <TYPE> can be one of:

Y to load values from YAML files from the fileserver, this is the default when no type is defined
C to lookup values with salt['config.get']
G to lookup values with salt['grains.get']
I to lookup values with salt['pillar.get']

The YAML type option can define the query method to lookup the key value to build the file name:

C to query with salt['config.get'], this is the default when no query method is defined
G to query with salt['grains.get']
I to query with salt['pillar.get']

The C, G or I types can define the SUB option to store values in the sub key mapdata.<KEY> instead of directly in mapdata.

All types can define the <DELIMITER> option to use an alternate delimiter of the <KEY>, for example: on windows system you can't use colon : for YAML file path name and you should use something else like exclamation mark !.

Finally, the <KEY> describes what to lookup to either build the YAML filename or gather values using one of the query methods.

Note

For the YAML type:

if the <KEY> can't be looked up, then it's used a literal string path to a YAML file, for example: any/path/can/be/used/here.yaml will result in the loading of salt://{{ tplroot }}/parameters/any/path/can/be/used/here.yaml if it exists
map.jinja will automatically try to load a Jinja2 template, after the corresponding YAML file, with the same name as the YAML file extended with the .jinja extension, for example any/path/can/be/used/here.yaml.jinja

The built-in map.jinja sources are:

- "Y:G@osarch"
- "Y:G@os_family"
- "Y:G@os"
- "Y:G@osfinger"
- "C@{{ tplroot ~ ':lookup' }}"
- "C@{{ tplroot }}"
- "Y:G@id"

This is strictly equivalent to the following map_jinja.yaml.jinja:

values:
  sources:
    - "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml"
    - "parameters/osarch/{{ salt['grains.get']('osarch') }}.yaml.jinja"
    - "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml"
    - "parameters/os_family/{{ salt['grains.get']('os_family') }}.yaml.jinja"
    - "parameters/os/{{ salt['grains.get']('os') }}.yaml"
    - "parameters/os/{{ salt['grains.get']('os') }}.yaml.jinja"
    - "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml"
    - "parameters/osfinger/{{ salt['grains.get']('osfinger') }}.yaml.jinja"
    - "C@{{ tplroot ~ ':lookup' }}"
    - "C@{{ tplroot }}"
    - "parameters/id/{{ salt['grains.get']('id') }}.yaml"
    - "parameters/id/{{ salt['grains.get']('id') }}.yaml.jinja"

Loading values from the configuration sources

For each configuration source defined, map.jinja will:

load values depending on the source type:
- for YAML file sources
  - if the <KEY> can be looked up:
    - load values from the YAML file named salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml if it exists
    - load values from the Jinja2 YAML template file named salt://{{ tplroot }}/paramaters/<KEY>/{{ salt['<QUERY_METHOD>']('<KEY>') }}.yaml.jinja if it exists
  - otherwise:
    - load the YAML file named salt://{{ tplroot }}/parameters/<KEY>.yaml if it exists
    - load the Jinja2 YAML template file named salt://{{ tplroot }}/parameters/<KEY>.yaml.jinja if it exists
- for C, G or I source type, lookup the value of salt['<QUERY_METHOD>']('<KEY>')
merge the loaded values with the previous ones using salt.slsutil.merge

There will be no error if a YAML or Jinja2 file does not exists, they are all optional.

Configuration values from `salt['config.get']`

For sources with of type C declared in map_jinja:sources, you can configure the merge option of salt['config.get'] by defining per formula strategy configuration key (retrieved with salt['config.get'](tplroot ~ ':strategy') with one of the following values:

recurse merge recursively dictionaries. Non dictionary values replace already defined values
overwrite new value completely replace old ones

By default, no merging is done, the first value found is returned.

Global view of the order of preferences

To summarise, here is a complete example of the load order of formula configuration values for an AMD64 Ubuntu 18.04 minion named minion1.example.net for the libvirt formula:

parameters/defaults.yaml
parameters/defaults.yaml.jinja
parameters/osarch/amd64.yaml
parameters/osarch/amd64.yaml.jinja
parameters/os_family/Debian.yaml
parameters/os_family/Debian.yaml.jinja
parameters/os/Ubuntu.yaml
parameters/os/Ubuntu.yaml.jinja
parameters/osfinger/Ubuntu-18.04.yaml
parameters/osfinger/Ubuntu-18.04.yaml.jinja
salt['config.get']('libvirt:lookup')
salt['config.get']('libvirt')
parameters/id/minion1.example.net.yaml
parameters/id/minion1.example.net.yaml.jinja

Remember that the order is important, for example, the value of key1:subkey1 loaded from parameters/os_family/Debian.yaml is overridden by a value loaded from parameters/id/minion1.example.net.yaml.

For formula authors and contributors

Dependencies

map.jinja requires:

salt minion 2018.3.3 minimum to use the traverse jinja filter
to be located at the root of the formula named directory (e.g. libvirt-formula/libvirt/map.jinja)
the libsaltcli.jinja library, stored in the same directory, to disable the merge option of salt['config.get'] over salt-ssh
the libmapstack.jinja library to load the configuration values
the libmatchers.jinja library used by libmapstack.jinja to parse compound like matchers

Use formula configuration values in `sls`

The map.jinja exports a unique mapdata variable which could be renamed during import.

Here is the best way to use it in an sls file:

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split("/")[0] %}
{%- from tplroot ~ "/map.jinja" import mapdata as unbound with context %}

test-does-nothing-but-display-unbound-as-json:
  test.nop:
    - name: {{ unbound | json }}

Use formula configuration values in templates

When you need to process salt templates, you should avoid calling salt['config.get'] (or salt['pillar.get'] and salt['grains.get']) directly from the template. All the needed values should be available within the mapdata variable exported by map.jinja.

Here is an example based on template-formula/unbound/config/file.sls:

# -*- coding: utf-8 -*-
# vim: ft=sls

{#- Get the `tplroot` from `tpldir` #}
{%- set tplroot = tpldir.split('/')[0] %}
{%- set sls_package_install = tplroot ~ '.package.install' %}
{%- from tplroot ~ "/map.jinja" import mapdata as unbound with context %}
{%- from tplroot ~ "/libtofs.jinja" import files_switch with context %}

include:
  - {{ sls_package_install }}

unbound-config-file-file-managed:
  file.managed:
    - name: {{ unbound.config }}
    - source: {{ files_switch(['unbound.conf'],
                              lookup='unbound-config-file-file-managed'
                 )
              }}
    - mode: 644
    - user: root
    - group: {{ unbound.rootgroup }}
    - makedirs: True
    - template: jinja
    - require:
      - sls: {{ sls_package_install }}
    - context:
        unbound: {{ unbound | json }}

This sls file expose a unbound context variable to the jinja template which could be used like this:

########################################################################
# File managed by Salt at <{{ source }}>.
# Your changes will be overwritten.
########################################################################

This is another example file from SaltStack template-formula.

# This is here for testing purposes
{{ unbound | json }}

winner of the merge: {{ unbound['winner'] }}

20 KiB Raw Blame History

map.jinja: gather formula configuration values

For formula users

Quick start: configure per role and per DNS domain name values

Configure the fileserver backends

Create per DNS configuration for unbound formula

Create per role configuration for unbound formula

Enable roles and the dns:domain and domain grains for map.jinja

Bind roles to minions

Note for Microsoft Windows systems

Format of configuration YAML files

Using Jinja2 YAML template

Sources of configuration values

Configuring map.jinja sources

Loading values from the configuration sources

Configuration values from salt['config.get']

Global view of the order of preferences

For formula authors and contributors

Dependencies

Use formula configuration values in sls

Use formula configuration values in templates

20 KiB

Raw Blame History

`map.jinja`: gather formula configuration values

Create per DNS configuration for `unbound` formula

Create per role configuration for `unbound` formula

Enable roles and the `dns:domain` and `domain` grains for `map.jinja`

Configuring `map.jinja` sources

Configuration values from `salt['config.get']`

Use formula configuration values in `sls`