Introduction
A self-hosted Git repository provider, GitLab Community Edition comes with additional features for software development as well as project management; including GitLab CI, which is a built-in continuous integration and delivery tool.
This tutorial will elaborate on how one can set up GitLab CI to not just monitor repositories for changes but also run automated tests for validating any new code.
To start, we will use an example repository for a basic Node.js application for our running GitLab installation. After configuring the CI process; GitLab will, upon a new commit being pushed to the repository, use the CI runner to execute the test suite against the code in an isolated Docker container.
Prerequisites
You will need a secure GitLab server configured to store the code and manage the CI/CD processes. Moreover, the server that GitLab is installed on or another separate host would be required to run the automated tests. Let’s look into this in more detail before we move ahead.
SSL Secured GitLab Server
Firstly, install a GitLab instance on an Ubuntu 16.04 server to store the source code and configure the CI/CD tasks. A server with at least two CPU cores and 4GB of Ram is recommended. Secondly, use “Let’s Encrypt” for SSL protection. To complete this step, your server needs to have a domain or a subdomain name associated with it. In short, you can complete these requirements through:
- An initial Server Setup with Ubuntu 16.04: Configuring a basic firewall after creating a sudo user
- Installing and configuring GitLab on Ubuntu 16.04
- Securing the GitLab with Let’s Encrypt SSL certificate on Ubuntu 16.04
This tutorial will also discuss how to lock CI/CD runners to single projects or share them between projects. If you already didn’t do this during installation, you’ll need to restrict or disable public sign-ups for sharing the CI runners between projects. This will be to avoid possible threats of any external interference.
Servers as GitLab CI Runners
These are servers that run automated tests for validating changes made in the code. All these tests will be run with Docker containers so that the testing environment remains isolated. For this, you’ll need to install Docker on the relevant servers. Using a different server than your GitLab server will help to avoid any resource contention. This is how you can go about it:
- Do an initial Server Setup with Ubuntu 16.04: Configure a basic firewall like you did on your GitLab server.
- You won’t have to do this again if you are setting up the CI runner on the GitLab server itself.
- Then install and use Docker on Ubuntu 16.04.
After you are done with all these preparations, we’ll move ahead with our purpose.
Example Repository from GitHub
To start with, create a new project in GitLab and use the example Node.js application. For avoiding manual uploading, you can import the original repository directly from GitHub.
Log into GitLab and as shown in the image, click the plus icon on the upper right corner to select New project.
Since the repository on GitHub is called hello_hapi, we will be using that as our Project name.
Moving on, we will require the code and Git history. To do so, we’ll be importing by URL to make things easier. There is also a GitHub option but it requires a Personal access token and is used to import the repository and other information.
So, click on the Repo by URL button under the Import project from section. Then in the Git Repository URL field, enter:
https://github.com/do-community/hello_hapi.git
Things should appear like this:
Its preferable to keep the repository marked Private. When you are done, click on Create Project and you will have a project based on the imported repository
Understanding .gitlab-ci.yml File
Every repository has this .gitlab-ci.yml file for which the GitLab CI searches so as to determine how the code is to be tested. Our imported repository too has an already configured .gitlab-ci.yml file. Click on this file showing on the GitLab interface.
The GitLab CI YAML configuration syntax is being used to define the actions that should be taken, the order to be executed, the conditions involved and the necessary resources for doing this. You can use a syntax linter by visiting /ci/lint in your GitLab instance to check the formatting of your own GitLab CI files.
The CI configuration file firstly states a Docker image to be used for running the test suite. Hapi being a Node.js framework, it uses the latest Node.js image:
image: node:latest
Then it mentions the different continuous integration stages that will run:
stages: - build - test
The ordering here helps to determine the order of execution for the proceeding steps, though the names chosen are arbitrary. These stages are tags that can be applied to individual jobs and the jobs on the same stage will run together while the jobs tagged in the next stage wait to get executed. The next stage is initiated only after all the jobs in the previous stage get completed. In a situation when stages are not defined, GitLab uses three stages, namely, build, test and deploy; and assigns all the jobs to the test stage.
Moving on, the configuration then includes a cache definition to specify files that can be cached between stages:
cache: paths: - node_modules/
This often helps decrease the time required to run jobs which are dependent on resources that might not change between these runs. In our configuration, the node_modules directory is being cached. This is where the downloaded dependencies are installed by npm.
As we move forward, we can see that the first job is install_dependencies:
install_dependencies: stage: build script: - npm install artifacts: paths: - node_modules/
Descriptive names are helpful since they are being used in the GitLab UI. Generally, npm install is combined with the testing stages. In our case, we are letting it run with its own stage to better understand the interaction between stages.
The stage is marked as “build” with the stage directive. The actual commands to run are mentioned through the script directive. Multiple commands can be provided through adding lines within the script section.
The subsection artifacts is specifying the file or the directory path to save and pass between stages. Now the next step will require access to the dependencies downloaded through the npm install. The node_modules path allows for the next stage to have access to the files. This comes very useful when building artifacts like binaries since the files are available for viewing or downloading in the GitLab UI. Moreover, by replacing the entire paths section with untracked: true allows you to save everything produced during the stage.
We now move ahead towards the second job test_with_lab which is the command to run the test suite.
test_with_lab: stage: test script: npm test
This is the test stage, and has access to the artifacts, that is, the project dependencies produced by the build stage. The script section is stating the single line YAML syntax which is used only for single item cases.
So, this is how the .gitlab-ci.yml file defines CI/CD tasks. Using this understanding, we can define runners to execute the testing plan.
Triggering Continuous Integration Run
With the repository including a .gitlab-ci.yml file, the new commits will trigger a new CI run, which will be set to “pending” if there are no runners available. Let’s look at how we can trigger a CI run to understand how the pending state for a job appears. Once a runner is made available for this, the pending run immediately gets picked up.
To begin with, click on the plus sign from the hello_hapi GitLab project repository and select New File.
Enter dummy_file in the field File name and any text in the main editing window.
Once done, go to the bottom of the page and click on Commit changes. Afterwards, you’ll notice a small paused icon attached to the most recent commit when you go back to the main project page. Upon bringing the mouse over the icon, you can see it displaying “Commit:pending”, indicating that the tests to validate code changes have not yet run.
Now go to the pipeline overview page by clicking on Pipelines. Here, you will notice that the CI run is marked pending.
Click on the pending status to get details on the various stages of our run and the jobs linked to each stage. Also, you can click on CI Lint displayed on the right side of the screen to view the syntax of your.gitlab-ci.yml files.
Lastly, to get specific details about the delay in the run, click on the install_dependencies job.
You will see that the job is stuck as we haven’t configured any runners yet. Once a runner is made available, this same interface will display the output. You can also download the artifacts made during the build from here itself. After understanding what a pending job is, we can now assign a runner to take it up.
GitLab CI Runner Service Installation
Moving ahead, we will install the GitLab CI runner package and start the runner service which allows multiple runners for different projects. Whichever alternative you opt for, that is, using the same server as your GitLab instance or a different server altogether, the server needs to have Docker installed for the configuration.
Installing the CI runner service is similar to installing GitLab. You will need to download a script to add a GitLab repository to the apt source list. Then the runner package needs to be downloaded after running the script. Once done it should be configured to serve the GitLab instance.
To do this, download the latest version of the GitLab CI runner repository configuration script to the /tmp directory:
· curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.deb.sh -o /tmp/gl-runner.deb.sh
Interrogate the script to confirm that the actions it will take on running is satisfactory to you. A hosted version of it is available here:
· less /tmp/gl-runner.deb.sh
Afterwards, run the installer when and if you are comfortable with the safety of the script:
· sudo bash /tmp/gl-runner.deb.sh
The server will get set up through the script for using GitLab maintained repositories. This allows GitLab runner packages to be managed with the same package management tools used for other system packages. Upon completion, the installation can be continued using apt-get: which will install the GitLab runner package and start the runner service.
· sudo apt-get install gitlab-runner
Setting Up GitLab Runner
We will now move ahead to setting up the GitLab CI runner so that it starts accepting work. To authenticate the runner with the GitLab server, a GitLab runner token would be required, the type of which would depend on how the runner is to be used.
If there are specific requirements, then a project specific runner would be useful. If your deployment tasks require credentials, then a specific runner will be required for authenticating into the deployment environment. Another way would be to have resource intensive steps in the CI process for the project. Do remember that project specific runners do not accept other projects.
Then there’s the shared runner, which is a general-purpose runner used for running multiple projects. These take jobs from projects based on the algorithm managing the number of jobs running on each project. A shared runner can be set up using an admin account to log into GitLab.
Next, we’ll see how to get the runner tokens for our two options and you can choose the one you are comfortable with.
Collecting Information: Registering a Project Specific Runner
Start with navigating to the project page on the GitLab interface. Click on Settings, on the top then on CI/CD pipelines on the appearing submenu.
The Specific Runners section will be on the left side, showing instructions for registering project specific runners. You will have to copy the registration token provided in the third step:
While you are here and wish to disable any active shared runners, you can use the Disable shared Runners button displayed on the right. The next section would discuss collecting information from here and using it to register your runner.
Collecting Information: Registering a Shared Runner
To start with this, you will first need to be logged in with an administrative account. To view the admin area, click on the wrench icon in the navigation bar on the top. Select the Overview section and then click Runners for access to the configuration page.
You will have to copy the token displayed to use it for registering a GitLab CI runner for your project.
Registering a GitLab CI Runner with GitLab Server
You will now have to go back to the server where the GitLab CI runner service has been installed and type in the following command for registering a new runner:
sudo gitlab-runner register
After doing so, there will appear a series of requests related to the configuration as shown below:
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/)
When this comes up, enter your GitLab server’s domain name and use https:// to specify SSL. Recent versions will direct you automatically and if not, then you can append /ci to the end of your domain.
Please enter the gitlab-ci token for this runner
Here, you will enter the copied token from before.
Please enter the gitlab-ci description for this runner
Enter a descriptive name here. The list of runners will show up on the command line and in the GitLab interface.
Please enter the gitlab-ci tags for this runner (comma separated)
GitLab jobs express their dependency requirements in terms of these tags to ensure that they run on a compatible host. For now, you can leave this blank.
Whether to lock Runner to current project [true/false]
This is for assigning the runner to a specific project, after which it cannot be used by other projects. Choose “false” for this.
Please enter the executor
Choose “docker” here. This requires the method to be used by the runner for completing jobs.
Please enter the default Docker image (e.g. ruby:2.1)
This asks for the default image when the .gitlab-ci.yml file does not have an image specification. Here, it is preferred to specify a general image and then define more specific images in the .gitlab-ci.yml file. So, enter “alpine:latest” as a secure default.
After completing these prompts, you will have a new runner created for running your project’s CI/CD tasks. The runners in the GitLab CI runner service currently available can be viewed by typing:
sudo gitlab-runner list OutputListing configured runners ConfigFile=/etc/gitlab-runner/config.tomlexample-runner Executor=docker Token=e746250e282d197baa83c67eda2c0b URL=https://example.com
With a runner available for your use now, go back to the project in GitLab.
CI/CD Run in GitLab
Once you go back to GitLab, you may see the runner already running, depending on how long it has been since your registration.
Or there’s a possibility that it might have been completed.
Click on the running or passed icon to view the current state of your CI run. In case there was a problem, you will instead see a failed icon. The Pipelines menu can also be clicked for the same. It will direct you to the pipeline overview page where the status of the GitLab CI run can be viewed:
On the right side under the Stages header, there are circles that indicate the status of each of the stages within the run. Upon clicking on these circles, one can view the individual jobs in the stage the circle represents:
The first circle shows the Build stage. Move your cursor to that circle and click on the install_dependencies job which will then take you to the job overview page.
Due to the npm installing of the packages, you will now see the output of the job unlike the earlier message of no runners being available.
You can also view the other jobs by changing the stage and clicking on the runs displayed. Any artifacts produced by the run can also be viewed or downloaded.
Conclusion
This tutorial discussed the continuous integration and deployment capabilities of GitLab CI. We defined a pipeline in gitlab-ci.yml files to build and test applications and also went through the assignment of jobs to stages and their inter-relationships. Lastly, we learnt to set up a GitLab CI runner which picked up CI jobs for the tutorial project as well as understood about information about the individual GitLab CI runs. Thank you.
Would you like to know about Zuul, a CI/CD project gating tool? Download our white paper and get reading!
