# Continuous integration with Gitlab-CI

A software repository hosted in the MSO4SC Gitlab instance can setup a Continuous Integration/Delivery workflow by means of Gitlab-CI.

This document demonstrates the usage of several useful tools for the continuous in MSO4SC scope. This tools are packed in a lightweight Docker container and published in DockerHub (`mso4sc/ci:latest`).

We encourage you to use this image as base environment if you need the same features. You can see an example of how to setup CI/CD for MSO4SC in this repository. In this repository we are using this container (`mso4sc/ci:latest`).

In this example we sequentially perform several dependant steps in order to deploy a new software release:

• build: build and deliver a new Singularity image

• test: test in HPC using serverless Cloudify CLI and the HPC plugin

• deploy: [Currently EMPTY]

• undeploy: remove the Singularity image from SRegistry if any previous step failed.

Workflow:

`(1) build >> (2) test >> (3) deploy >> (4) undeploy`
 Take this into account If any step fail, the workflow breaks and next steps are cancelled. Undeploy step only occurs if any previous step failed.

Next sections describe some features of the involved tools.

## 1. Gitlab-CI

To enable the continuous integration of your project you have to place a `.gitlab-ci.yml` file in the root path of your repository.

`.gitlab-ci.yml` is a YAML file to define the workflow to be performed during the continuous integration process.

## 2. Sregistry

Transfer and storage web service for Singularity images. SRegistry is also a comand line tool to interact with the web service.

Images hosted in SRegistry are reference in a docker-like style `COLLECTION/IMAGE:TAG` (e.g. mso4sc/ci:latest).

To be able to push to the SRegistry web service from the command line you must use your user credentials.

User credentials must be stored in plain text file. The default path for this file is `$HOME/.sregistry`. If you want to use a custom path you must set the right path into `SREGISTRY_CLIENT_SECRETS` environment variable. To get your secret token you must be loged in the web service and visit your token page. SRegistry push sintax: ``````$ export SREGISTRY_CLIENT_SECRETS=SREGISTRY_TOKEN_FILE
$sregistry push --name COLLECTION/IMAGE --tag TAG SINGULARITY_IMAGE`````` • SREGISTRY_TOKEN_FILE: path to a plain text file containing your JSON token • SREGISTRY_CLIENT_SECRETS: environment variable pointing to `SREGISTRY_TOKEN_FILE` • COLLECTION: collection name • IMAGE: image name • TAG: reference tag • SINGULARITY_IMAGE: path to a singularity image SRegistry command line basic example: ``$ sregistry push --name mso4sc/example --tag latest example.simg``

## 3. How to do it

1. Create a secret variable containing your JSON token

1. Go to your project settings

2. Click on CI/CD

3. Expand Secret variables section

4. Create a new variable

5. Copy the token (e.g `{ "registry": { "token": "…​", "username": "…​", "base": "https://sregistry.srv.cesga.es" } }`)

2. Create a new `.gitlab-ci.yml` with the following workflow

1. Use `mso4sc/ci:latest` as a base image.

2. Write your JSON token to a file

3. Export `SREGISTRY_CLIENT_SECRETS` pointin to your JSON token file

5. Push the resulting singularity image to SRegistry

6. Delete the file containing the JSON token

Then the CI process will start automatically and publish the new image to SRegistry. You need to be assigned as owner of the `mso4sc` collection to be able to push.
``$singularity pull shub://sregistry.srv.cesga.es/mso4sc/example:latest`` ## 4. Cloudify CLI (cfy) Cloudify CLI (in local execution mode) allows you to execute your blueprints without connecting to the remote MSOOrchestrator instance. This use case is really usefull for HPC automated testing to ensure the correct functioning of your blueprints before using the MSOOrchestrator. The invocation pathway of `cfy` command follows the usual Cloudify flow (with an extra initial validation step) as described below: 1. Blueprint validation 2. Application registration 3. Job environment bootstrap 4. Job execution 5. Job environment revert You can see an implementation inxample in the `gitlab-ci.yml` file, this workflow is coded with the following commands: ``````$ ID=BLUEPRINT_ID
$cfy blueprints validate blueprint.yaml$ cfy init --install-plugins -b $ID -i ../inputs.yaml blueprint.yaml$ cfy executions start -b $ID install$ cfy executions start -b $ID run_jobs$ cfy executions start -b \$ID uninstall``````
You can get more info about `cfy` in the following link.