Setting up a LAVA instance¶

Installing just lava-server¶

The lava-server package is the main LAVA scheduler and frontend.

To install just the lava-server from the current packages, use:

$ sudo apt install lava-server
$ sudo a2dissite 000-default
$ sudo a2enmod proxy
$ sudo a2enmod proxy_http
$ sudo a2ensite lava-server.conf
$ sudo service apache2 restart

This will install lava-dispatcher and lava-server.

Other packages to consider:

• ntp - some actions within LAVA can be time-sensitive, so ensuring that devices within your lab keep time correctly can be important.

Installing the full lava set¶

Production installs of LAVA will rarely use the full lava set as it includes tools more commonly used by developers and test labs. These tools mean that the lava package brings more dependencies than when installing lava-server to run a production LAVA instance.

The lava package installs support for:

• lava-dev - scripts to build developer packages based on your current git tree of lava-server or lava-dispatcher, including any local changes.

  Note

  lava-dev includes a lot of packages which are not typically used on a production master or worker.

• vmdebootstrap for building your own Debian based KVM images.

• ntp - some actions within LAVA can be time-sensitive, so ensuring that devices within your lab keep time correctly can be important.

All of these packages can be installed separately alongside the main lava-server package, the lava package merely collects them into one set.

$ sudo apt install postgresql
$ sudo apt install lava
$ sudo a2dissite 000-default
$ sudo a2enmod proxy
$ sudo a2enmod proxy_http
$ sudo a2ensite lava-server.conf
$ sudo service apache2 restart

Installing master without Recommends¶

The lava-common binary package is new in 2018.5 and allows admins to choose not to install lava-dispatcher on the master if there are to be no devices assigned to the machine running lava-master. This is common for installations where there are multiple workers and the master is regularly busy. lava-server now _Recommends_ lava-dispatcher which means that admins can choose not to install it alongside lava-server:

$ sudo apt --no-install-recommends install lava-server lava-server-doc

Depending on the local configuration, some of the other recommended packages may also be desirable:

• lava-coordinator
• ntp

lava-server-doc can be omitted but this would be unusual - instances would need to be configured to have some other Help option in the menu using the CUSTOM_DOCS dictionary setting in /etc/lava-server/settings.conf and the Help links from pages within the LAVA UI would cause a 404 error for users, unless the Apache configuration was adjusted.

Installing lava-dispatcher¶

If this machine is only meant to be a dispatcher for connected devices, then just install lava-dispatcher. The lava-server package is only needed on the master in each LAVA instance.

$ sudo apt install lava-dispatcher

1. Change the dispatcher configuration in /etc/lava-dispatcher/lava-slave to allow the init script for lava-slave (/etc/init.d/lava-slave) to connect to the relevant lava-master instead of localhost. Change the port numbers, if required, to match those in use on the lava-master:

   /etc/lava-dispatcher/lava-slave
   
   # Configuration for lava-slave daemon
   
   # URL to the master and the logger
   # MASTER_URL="tcp://<lava-master-dns>:5556"
   # LOGGER_URL="tcp://<lava-master-dns>:5555"
   
   # Enable IPv6 to connect to the master and logger
   # IPV6="--ipv6"
   
   # Slave hostname
   # Should be set for host that have random hostname (containers, ...)
   # The hostname can be any unique string, except "lava-logs" which is reserved
   # for the lava-logs daemon.
   # HOSTNAME="--hostname <hostname.fqdn>"
   
   # Logging level should be uppercase (DEBUG, INFO, WARN, ERROR)
   # LOGLEVEL="DEBUG"
   
   # Encryption
   # If set, will activate encryption using the master public and the slave
   # private keys
   # ENCRYPT="--encrypt"
   # MASTER_CERT="--master-cert /etc/lava-dispatcher/certificates.d/<master.key>"
   # SLAVE_CERT="--slave-cert /etc/lava-dispatcher/certificates.d/<slave.key_secret>"
   

   See also

   Enable master encryption and Enable slave encryption

2. Restart lava-slave once the changes are complete:

   $ sudo service lava-slave restart
   

3. The administrator of the master will then be able to allocate pipeline devices to this slave.

The administrator of the master will require the following information about the devices attached to each slave:

1. Confirmation that a suitable template already exists, for each device i.e. Adding support for a device of a known type
2. A completed and tested device dictionary for each device.

This information contains specific information about the local network setup of the slave and will be transmitted between the master and the slave in clear text over ZMQ. Any encryption would need to be arranged separately between the slave and the master. Information typically involves the hostname of the PDU, the port number of the device on that PDU and the port number of the serial connection for that device. The slave is responsible for ensuring that these ports are only visible to that slave. There is no need for any connections to be visible to the master.

Configuring apache2 on a worker¶

Some test job deployments will require a working Apache2 server to offer deployment files over the network to the device:

$ sudo cp /usr/share/lava-dispatcher/apache2/lava-dispatcher.conf /etc/apache2/sites-available/
$ sudo a2ensite lava-dispatcher
$ sudo service apache2 restart
$ wget http://localhost/tmp/
$ rm index.html

You may also need to disable any existing apache2 configuration if this is a default apache2 installation:

$ sudo a2dissite 000-default
$ sudo service apache2 restart

Adding workers to the master¶

A new worker needs to be manually added to the master so that the admins of the master have the ability to assign devices in the database and enable or disable the worker.

To add a new worker:

$ sudo lava-server manage workers add <HOSTNAME>

To add a worker with a description:

$ sudo lava-server manage workers add --description <DESC> <HOSTNAME>

To add a worker in a disabled state:

$ sudo lava-server manage workers add --description <DESC> --disabled <HOSTNAME>

Workers are enabled or disabled in the Django admin interface by changing the display field of the worker. Jobs submitted to devices on that worker will fail, so it is also recommended that the devices would be made offline at the same time. (The django admin interface has support for selecting devices by worker and taking all selected devices offline in a single action.)

Using ZMQ authentication and encryption¶

lava-master and lava-slave use ZMQ to pass control messages and log messages. When using a slave on the same machine as the master, this traffic does not need to be authenticated or encrypted. When the slave is remote to the master, it is strongly recommended that the slave authenticates with the master using ZMQ curve so that all traffic can then be encrypted and the master can refuse connections which cannot be authenticated against the credentials configured by the admin.

To enable authentication and encryption, you will need to restart the master and each of the slaves. Once the master is reconfigured, it will not be possible for the slaves to communicate with the master until each is configured correctly. It is recommended that this is done when there are no test jobs running on any of the slaves, so a maintenance window may be needed before the work can start. ZMQ is able to cope with short interruptions to the connection between master and slave, so depending on the particular layout of your instance, the changes can be made on each machine before the master is restarted, then the slaves can be restarted. Make sure you test this process on a temporary or testing instance if you are planning on doing this for a live instance without using a maintenance window.

Encryption is particularly important when using remote slaves as the control socket (which manages starting and ending testjobs) needs to be protected when it is visible across open networks. Authentication ensures that only known slaves are able to connect to the master. Once authenticated, all communication will be encrypted using the certificates.

Protection of the secret keys for the master and each of the slaves is the responsibility of the admin. If a slave is compromised, the admin can delete the certificate from /etc/lava-dispatcher/certificates.d/ and restart the master daemon to immediately block that slave.

$ sudo /usr/share/lava-dispatcher/create_certificate.py master

$ sudo /usr/share/lava-dispatcher/create_certificate.py foo_slave_1

$ sudo service lava-slave stop

# Encryption
# If set, will activate encryption using the master public and the slave
# private keys
ENCRYPT="--encrypt"

# Encryption
# If set, will activate encryption using the master public and the slave
# private keys
ENCRYPT="--encrypt"

# MASTER_CERT="--master-cert /etc/lava-dispatcher/certificates.d/<master.key_secret>"
# SLAVES_CERTS="--slaves-certs /etc/lava-dispatcher/certificates.d"

# Encryption
# If set, will activate encryption using the master public and the slave
# private keys
ENCRYPT="--encrypt"

# MASTER_CERT="--master-cert /etc/lava-dispatcher/certificates.d/<master.key>"
# SLAVE_CERT="--slave-cert /etc/lava-dispatcher/certificates.d/<slave.key_secret>"

2018-02-05 11:33:55,933    INFO [INIT] Marking all workers as offline
2018-02-05 11:33:55,983    INFO [INIT] Starting encryption
2018-02-05 11:33:55,984   DEBUG [INIT] Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key_secret
2018-02-05 11:33:55,985   DEBUG [INIT] Using slaves certificates from: /etc/lava-dispatcher/certificates.d/
2018-02-05 11:33:55,986    INFO [INIT] LAVA master has started.
2018-02-05 11:33:55,986    INFO [INIT] Using protocol version 2

2018-02-05 11:34:42,035    INFO [INIT] LAVA slave has started.
2018-02-05 11:34:42,036    INFO [INIT] Using protocol version 2
2018-02-05 11:34:42,037    INFO [INIT] Starting encryption
2018-02-05 11:34:42,037   DEBUG Opening slave certificate: /etc/lava-dispatcher/certificates.d/codehelp.key_secret
2018-02-05 11:34:42,038   DEBUG Opening master certificate: /etc/lava-dispatcher/certificates.d/master.key
2018-02-05 11:34:42,038    INFO [INIT] Connecting to master as <codehelp>
2018-02-05 11:34:42,038    INFO [INIT] Greeting the master => 'HELLO'
2018-02-05 11:34:42,050    INFO [INIT] Connection with master established
2018-02-05 11:34:42,050    INFO Master is ONLINE
2018-02-05 11:34:42,053    INFO Waiting for instructions

Adding devices to a worker¶

Admins use the Django admin interface to add devices to workers using the worker drop-down in the device detail page.