Ansible is a powerful automation tool that helps us manage infrastructure, services, and databases. However, managing sensitive data such as passwords, keys, and other secrets can be challenging. Ansible Vault is a feature that provides a way to encrypt and store sensitive information within Ansible playbooks and inventory files.

Ansible Vault has two modes:

  • full file encryption: the whole contents of the file is encrypted, the type of content doesn’t matter;
  • value encryption: a string value in a yaml file is encrypted.

In both modes this mechanism is fully transparent for Ansible. After the Vault password is supplied, all Ansible operations work just the same way as without encryption. But it isn’t transparent for the user: the encrypted content is, well, encrypted.

In our projects we have several (one per environment) variable files designated for secrets. These are yaml files with a simple flat structure: secret name to secret value mapping. Often we need to add a new secret variable to these files, sometimes we would like to rename some variables. And almost never we need to see actual values: those are just random strings. We are more interested in their names, because this is how we refer them in playbooks.

Full file encryption

We used the first mode for a while, because it is much better supported. Ansible comes with the ansible-vault utility that provides some functionality to work with encrypted files. It can view, edit (using configured EDITOR), encrypt, decrypt and rekey (change password) fully encrypted files. And it mostly fulfills our needs, but lacks transparency.

Because files are fully encrypted, keys are also encrypted. In fact, we can’t even tell that it is a yaml file without decrypting it. That approach has several problems:

  • Finding variables: for plain variables we can use grep (or shortcut in IDE) to find where variables are used, but it will not work for secrets
  • Code review: diff is useless on encrypted files
  • Change history and annotation also become useless

Workaround for git diff

Git has a feature to process files with an external command before feeding it into diff. And ansible-vault view can be used as this command, so diff will show the changes on decrypted files. This can be done with the following steps:

  1. Configure a custom converter: git config diff.ansible-vault.textconv "ansible-vault view"
  2. Set git attribute echo "path/to/vault_file diff=ansible-vault merge=binary

But this solution is limited to the git utility and doesn’t work in GitHub or any IDE.

Value encryption

In order to get better transparency for secrets changes and solve the stated problems, we recently switched to value encryption mode. Unfortunately, Ansible doesn’t offer many tools for it. On the other hand, some operations can now be done in an editor: we can rename and delete secrets even without the vault password.

To make the work with secrets more convenient, we also implemented a script that:

  • automates typical operations:
    • generate and encrypt new passwords in all environments,
    • delete secrets from all environments);
  • adds missing actions:
    • get secret value,
    • view file in plain text,
    • change vault password;
  • converts fully encrypted files to value encrypted form.

Vault Python API

Since Ansible is written in Python, its vault module can be directly used in a Python script:

import ansible.parsing.vault

class AnsibleVault:
    def __init__(self, vault_password: str):
        self.vault_password = ansible.parsing.vault.VaultSecret(vault_password.encode())
        self.vault = ansible.parsing.vault.VaultLib([
          (ansible.parsing.vault.C.DEFAULT_VAULT_ID_MATCH, self.vault_password)
        ])

    def encrypt(self, secret: Secret) -> str:
        return self.vault.encrypt(secret.payload.encode(), salt=secret.salt).decode()

    def decrypt(self, ciphertext: str) -> Secret:
        salt = ansible.parsing.vault.parse_vaulttext(
          ansible.parsing.vault.parse_vaulttext_envelope(ciphertext.encode())[0]
        )[1]
        return Secret(self.vault.decrypt(ciphertext.encode()).decode(), salt=salt)

Note that in order to preserve files unchanged in a load-dump roundtrip, we also need to preserve salt values.

Ansible Vault uses a custom YAML tag !vault to mark encrypted values. The PyYAML library supports custom tags by adding callbacks to a loader and a dumper. These callbacks will convert a parser tree node to a python object and vice-versa:

class Secrets(dict):
    class YamlVaultLoader(yaml.SafeLoader):
        def __init__(self, vault: AnsibleVault, *args, **kwargs):
            super().__init__(*args, **kwargs)
            self.add_constructor('!vault', self._construct_secret)
            self.vault = vault

        def _construct_secret(self, loader: yaml.Loader, node: yaml.ScalarNode) -> Secret:
            ciphertext = loader.construct_scalar(node)
            return self.vault.decrypt(ciphertext)

    class YamlVaultDumper(yaml.SafeDumper):
        def __init__(self, vault: AnsibleVault, *args, **kwargs):
            super().__init__(*args, **kwargs)
            self.add_representer(Secret, self._represent_secret)
            self.vault = vault

        def _represent_secret(self, dumper: yaml.Dumper, secret: Secret) -> yaml.ScalarNode:
            ciphertext = self.vault.encrypt(secret)
            return dumper.represent_scalar('!vault', ciphertext, style='|')

    @classmethod
    def load(cls, path: Path, vault: AnsibleVault) -> 'Secrets':
        with path.open() as fh:
            return cls(yaml.load(fh, functools.partial(cls.YamlVaultLoader, vault)))

    def dump(self, path: Path, vault: AnsibleVault):
        with path.open('w') as fh:
            yaml.dump(dict(self), fh, functools.partial(self.YamlVaultDumper, vault))

Vault password and 1Password integration

We also use 1Password for personal passwords and sharing credentials, like Ansible Vault password. And it turns out that passing the password from 1Password to Ansible can also be automated.

With vault_password_file = vault_password.sh in ansible.cfg, ansible will call the vault_password.sh script to get the password every time it requires it. And 1Password has a command line tool that can do a wide variety of operations, including getting a specific password: op read "op://Vault Name/Item Name/password".

Therefore vault_password.sh can look like this:

#!/bin/bash
set -e

if [[ -t 1 ]]; then
    echo "I don't want to print the password to console :("
    exit 1
fi

if type op >/dev/null 2>&1; then
    exec op read "op://Vault Name/Item Name/password"
else
    echo "1Password CLI not installed"
    exit 1
fi

Summary

A convenient, transparent and customisable secrets management workflow can be built around Ansible Vault using these hints and a little bit of coding.