16 Automating Experiment Configuration

This chapter focuses on automating experiment configuration: customizing software configuration on your nodes, deploying services, and more. Manual experiment configuration is error-prone, tedious and inefficient, and difficult or impossible to reproduce. By automating your experiments and building your profiles in a more structured fashion, you can add robustness and repeatibility to your profiles, and become a more efficient Powder user—only allocating resources when you need them, since your configuration tasks are scripted and repeatable, rather than manual and time-consuming.

Several Powder profiles use the Ansible configuration management tool to automate experiment node configuration; we discuss several example profiles below.

16.1 Configuration Management: Ansible

To assist with configuration automation, we provide platform-level integration with Ansible, a widely-used and accessible configuration management tool. Ansible provides a built-in standard library of configuration modules for nearly any common UNIX-like task, and many more modules are available to integrate with other systems and software. If you haven’t used a configuration management tool before, this may seem excessive, but in addition to a standard library of configuration helpers, Ansible tasks have common error-handling support, and only make and apply changes when necessary—cumbersome to implement in shell scripts, but one-liners in a task—and this results in much more durable, repeatable configuration.

At this point, you may wish to browse through the Ansible Getting Started guide so that you understand key concepts like playbooks, roles, variables, inventories, and more.

At a high level, our integration connects your profile to configuration: you declare experiment configuration in your profile using the Ansible extensions. Once your experiment nodes boot, the bootstrap repository initializes an Ansible environment. It invokes the emulab-common collection to process the Ansible configuration declarations from the experiment manifest, generates shell scripts that play the roles to the nodes that you bound them to in your profile, and runs these scripts to invoke Ansible. This integration allows you to push as much declaration into the profile as useful, so that you can use profile parameters to override variables used in the roles—thereby changing experiment configuration when you instantiate a profile.

16.2 Ansible in Profiles: geni-lib extensions

The Ansible extensions allow you to declare and use Ansible abstractions in your profiles. Below, we describe each Ansible abstraction, and how to declare it in your profile, so that our integration can automatically run Ansible in your experiment:

• Ansible Roles group related tasks, template files, vars, etc into a well-known directory structure. Roles are typically bound to nodes in an inventory, and often include simple playbooks that execute the role’s tasks on associated nodes.

  • The geni-lib Roles abstraction allows you to declare the presence of a role in your profile repository, e.g. in an ansible/ subdirectory, as we will show in the examples below.

  • The geni-lib RoleBindings abstraction allows you to bind previously-declared Roles to nodes, so that the role and its playbook are run on the bound nodes.

• Ansible Playbooks list tasks that are executed serially to change configuration (e.g., create a file based on a template, change a line in a file, run a command, etc). A playbook is typically associated with a role to pull in per-role tasks, but it can also function as a standalone task list.

  • The geni-lib Playbooks abstraction allows you to reference a playbook in your profile’s git repository (e.g. in the ansible/ subdirectory, as in our examples below), that should be run automatically when your nodes boot.

• Ansible Variables are the typical way to configure the details of how a role configures a node. Roles define variable default values, and these can be changed, or overridden, to change configuration task behavior.

  • The Overrides abstraction binds profile parameters to Role variables, so that values from your profile override (take precedence over) the defaults. The emulab-common Ansible collection contains code to automatically generate per-role and per-node override value files, so that when you create an experiment, parameter values can be automatically mapped to Ansible configuration.

• Ansible Collections group roles and playbooks into a single reusable code repository.

  • The Collections abstraction allows you to declare that a required Ansible collection dependency be installed, and gives you control of automatic install of the collection’s own dependencies as well—before your playbooks and roles execute. This ensures the collection’s roles and playbooks are installed before your Ansible tasks invoke it.

(We attempt to support the most common Ansible usage patterns with this set of profile extensions, but there are many ways to perform a given configuration with Ansible, so we do not support every possible pattern.)

16.3 Bootstrapping Ansible in Experiments

Our small Ansible bootstrapping repository can be included as a git submodule in a git-based profile. It provides a script that can be used as your experiment’s startup command. This script installs Ansible (into a Python virtualenv via pip (default) or from Linux distribution packages; you can specify a particular version); extracts the Ansible abstractions from your experiment’s manifest; autogenerates shell wrappers (which we call entrypoints) that run the necessary Ansible playbooks; and runs the wrappers. You can customize this script’s behavior via environment variables. It’s easy to fork and modify if necessary.

To add this as a submodule to an existing or new profile’s git repository, simply run the following command, likely in your repository’s top-level directory:

git submodule add https://gitlab.flux.utah.edu/emulab/ansible/emulab-ansible-bootstrap.git

and add and commit. Then follow the instructions in the shim’s README.md to run the shim at experiment runtime. To summarize: you will choose a single node in your experiment to act as the head or manager node. The head node should add the emulab-ansible-bootstrap/head.sh script as its startup script, and managed clients should add the emulab-ansible-bootstrap/client.sh script.

You can configure the shim to install Ansible in different ways (e.g., using the system packages, choose versions, etc), but the default is to install Ansible==4.0.0 (see the declaration of ANSIBLE_VERSION) in a Python virtualenv, located on your head node in /local/setup/venv/default. If you ever need to run ansible-playbook or any of the other scripts manually, you would run (in the bash shell):

. /local/setup/venv/default/bin/activate ansible -m ping localhost deactivate

If your profile uses the geni-lib Ansible extensions to bind Roles and Playbooks to nodes (and/or Overrides to profile parameters), the shim generates an Ansible inventory for you in /local/setup/ansible/inventory.ini. This inventory contains an all group that lists all nodes in your experiment, as well as per-role groups containing the nodes that were bound to Roles. Any profile parameters that were associated with Overrides in your profile are written into /local/setup/ansible/vars (host- and group-specific variables, if any, are written into per-host files in the /local/setup/ansible/host_vars and /local/setup/ansible/group_vars directories.

Finally, the shim generates shell script wrappers (/local/setup/ansible/entrypoints/*.sh) that run each Ansible playbook defined in your profile, and a top-level driver script (/local/setup/ansible/run-automation.sh that runs them in the order listed in the profile. The per-playbook driver scripts (entrypoints) simply enter the proper Python virtualenv and run the playbook via ansible-playbook with the proper become and overrides arguments, as defined in your profile.

(The top-level automation driver script will run on your head node automatically, unless you define EMULAB_ANSIBLE_NOAUTO=1 in the profile startup command that runs head.sh from the shim.)

16.4 Using Ansible in Experiments

We provide the emulab-common Ansible collection that provides plugins and a small library of useful roles to help bridge the gap between a profile, a Powder experiment’s physical and virtual resources, and Ansible roles and playbooks.

16.4.1 Per-experiment Ansible Facts

When you call one or both of the former two modules, they add more information (facts) to the global ansible_facts dictionary. You can use these facts in playbooks, task files, and templates. For instance, suppose you wanted to add a task to run iperf in server mode, but only on a particular experiment network. You could find the IP address to listen on, assuming your node was named node-0, and is a member of a LAN named lan-0, by referencing the variable emulab_topomap["nodes"]["node-0"]["lans"]["lan-0"]. If you needed the network mask for the lan-0 network, you could find that via emulab_topomap["lans"]["lan-0"]["mask"]. If you needed node-0’s FQDN, you could find that in emulab_fqdn.

You can dump the facts from a running experiment as follows. (If you have already initialized your node using the Ansible bootstrap repository, skip the first code sample, since you already have a virtualenv and an inventory file.)

Initialize a Python virtualenv for Ansible and inventory file if you don’t already have one in /local/setup. Change the node name from node-0 to the short name of your experiment node.

sudo mkdir -p /local/setup/venv/default sudo chown -R $(id -un) /local/setup python3 -m venv –upgrade /local/setup/venv/default cat <<EOF >/local/setup/ansible/inventory.ini [all] node-0 ansible_connection=local EOF

Activate the virtualenv and run the facts plugins:

. /local/setup/venv/default/bin/activate ansible -i /local/setup/ansible/inventory.ini -m emulab.common.gather_manifest_facts node-0 ansible -i /local/setup/ansible/inventory.ini -m emulab.common.gather_emulab_facts node-0 deactivate

(Notice in the output that these facts are being added to the ansible_facts dictionary, but they are also added to the global Python namespace for use in tasks and templates.)

The emulab-common gather_manifest_facts plugin obtains information from the experiment manifest, which you can browse in the Portal web UI by clicking the Manifest tab on your experiment status page.

The emulab-common gather_emulab_facts plugin obtains information from the experiment node’s /var/emulab/boot directory, the contents of which are repopulated on each boot by querying the central configuration server.

16.4.2 emulab-common Ansible Roles

The bootstrap repository installs the latest version of the emulab-common collection.

16.4.3 Using emulab-common Roles

You will typically use these roles in your own experiments in one of two ways. For instance, suppose that your profile required Docker Compose to deploy a service.

First, you could use Ansible to get Docker installed on a node by simply modifying your profile to include the geni-lib abstractions:

from geni.rspec.emulab.ansible import (Role, RoleBinding, Override, Playbook, Collection) node.bindRole(RoleBinding("emulab.common.docker"))

If you wanted to force use of the distribution’s Docker packages, instead of the Community Edition (the default), you could set an override to tell the role:

node.addOverride(Override("emulab_docker_use_distro", value="True"))

If you wanted to set this preference for all nodes in your profile, you could instead do:

request.addOverride(Override("emulab_docker_use_distro", value="True"))

Second, if you added a custom role for your profile to also automate the deployment of the service via Docker Compose, this role could require the emulab.common.docker role as a dependency in its meta/ subdirectory, and set any desired overrides as above.

We show a complete example of the latter approach below.

16.5 Example: OpenZMS profile

OpenZMS is cloud-native spectrum management and sharing software developed as part of the POWDER-RDZ project.

The OpenZMS zms-profile (source code) automatically deploys OpenZMS in Powder experiments using Ansible. It relies on both the emulab-common and OpenZMS Ansible collections to do the bulk of the work; and includes a profile-specific openzms_powder role that includes roles from those two collections.

Briefly, the zms-profile’s profile.py script does the following:

• declares the openzms_powder role (found in ansible/roles/openzms_powder), with a playbook that plays the role onto bound nodes

• adds the openzms Collection, so that it is installed before the openzms_powder role, which requires it, is played

• defines profile parameters, some of which are later associated with Override values

• binds profile parameters to Override objects, meaning that values of these parameters when an experiment is created will be written to an Ansible variable overrides file that is passed to playbook execution

• binds the openzms_powder role to the single node, meaning that this role will be played onto the node

The openzms_powder role wraps the openzms role and depends on several of the emulab-common roles. The emulab.common dependencies are listed in the ansible/roles/openzms_powder/meta/main.yml file, and are run in-order prior to the tasks in the openzms_powder playbook. Finally, the openzms_powder role imports the openzms role into its own tasks, overriding several of the openzms role’s default variables.

16.6 Example: Kubernetes profile

We provide a fully Ansible version of our standard Kubernetes profile (source code here). All configuration in this profile is performed via Ansible playbooks, roles, and tasks, once the bootstrapping shim has ensured that Ansible is present on the designated experiment node. Like the standard Kubernetes profile, this profile uses the Kubespray installer at its core.

16.7 Example: workflow-manager profile

Our workflow-manager profile (source code here) provides custom roles that install Stackstorm and KiwiTCMS, which also use roles from the emulab-common module. Specifically, the profile’s geni-lib script

• declares two roles: stackstorm and kiwitcms, both in the ansible/ subdirectory of the repository, each associated with a playbook in the ansible/ subdirectory,

• binds the roles to nodes,

• declares a number of parameters, and binds some to Ansible overrides.

The stackstorm and kiwitcms roles in this profile demonstrate a traditional use of Ansible.