Technical Deep Dive: Atmosphere's Secure Image Builds

Explore the intricate process of Atmosphere's secure image building in this technical deep dive. Understand the role of Docker Content Trust, Trivy, and Cosign in enhancing security across the stack, ensuring the reliability and integrity of images. Discover how Atmosphere addresses unpatched security issues, verifies image signatures, and maintains continuous security through daily scans.

Atmosphere is at the forefront of the cloud computing landscape, employing innovative technologies and processes to ensure the integrity and security of its virtual environment. This technical deep dive focuses on the use of the vexxhost/docker-openstack-builder image, illustrating how Atmosphere builds the virtual environment with all the OpenStack Python packages.

Docker Content Trust

The vexxhost/docker-openstack-builder image is built as a base of either Ubuntu Focal (for older releases) or Ubuntu Focal (for the newer releases). The key element here is the enforcement of Docker Content Trust when pulling down these base images. This enforcement ensures that the images are signed, verifying their authenticity and integrity, thereby enhancing the overall trust in the images used.

- uses: docker/build-push-action@v5
  ...
  env:
    DOCKER_CONTENT_TRUST: 1
  with:
    sbom: true
  ...

You can see the rest of the workflow here. You'll also notice that we generate an software bill of materials (SBOM) for the image, which is a great way to track all of the dependencies that are used in the image.

Security Issues

To further fortify the security, we implement a list of modifications to the upper-constraints.txt file. This file is used by OpenStack to pin all dependencies during testing, however they often contain many unpatched CVEs. These changes address all unpatched security issues, ensuring potential vulnerabilities are identified and patched, reducing the risk of security breaches.

# 2023.1 security updates
sed -i s/^cryptography===38.0.2/cryptography===41.0.4/ /upper-constraints.txt
sed -i s/^Django===3.2.16/Django===3.2.22/ /upper-constraints.txt
sed -i s/^pyOpenSSL===22.1.0/pyOpenSSL===23.2.0/ /upper-constraints.txt
sed -i s/^requests===2.28.1/requests===2.31.0/ /upper-constraints.txt
sed -i s/^sqlparse===0.4.3/sqlparse===0.4.4/ /upper-constraints.txt
sed -i s/^urllib3===1.26.12/urllib3===1.26.18/ /upper-constraints.txt

You can see more of the files that are updated inside upper-constraints.txt in the following Dockerfile. You can see how the other OpenStack releases have all of their own security updates as well.

Vulnerability Scanning

To ensure no vulnerabilities exist in the base image, we run a full scan on the container image with Trivy, a comprehensive open-source vulnerability scanner for container images. It effectively identifies vulnerabilities that may exist in the OS packages and language-specific packages.

- uses: aquasecurity/trivy-action@master
  with:
    image-ref: quay.io/vexxhost/openstack-builder-${{ env.RELEASE }}:${{ github.sha }}
    format: 'sarif'
    output: 'trivy-results.sarif'
    ignore-unfixed: true
- uses: github/codeql-action/upload-sarif@v2
  if: always()
  with:
    category: ${{ env.RELEASE }}
    sarif_file: 'trivy-results.sarif'

You can see the rest of the workflow here. You'll see that we also take the results and upload them into GitHub's security scanning tool, which allows us to track any new vulnerabilities that may be introduced in the future and address them immediately.

Cosign Integration

Upon completion of the vulnerability scanning, we sign the image with Cosign and publish it. Cosign ensures the authenticity of the images, providing a secure way to sign and verify container images.

- name: Install cosign
  if: ${{ github.event_name == 'push' }}
  uses: sigstore/cosign-installer@main
- name: Sign the container image
  if: ${{ github.event_name == 'push' }}
  run: cosign sign --yes quay.io/vexxhost/openstack-builder-${{ env.RELEASE }}@${{ steps.push-step.outputs.digest }}

You can see the rest of the workflow here. You'll notice that we are using keyless signatures, relying on the GitHub OpenID connect tokens to sign and validate the images. This is a great way to ensure that the images are signed by the right people, and that the images are not tampered with.

Building Services

When building services, the same base images are pulled down, and their signatures are verified. A prime example of this process can be seen in how the OpenStack Horizon image, vexxhost/docker-openstack-horizon, is built.

Once the signatures are verified, the virtual environment is built in the usual manner. Trivy is then run to expose any potential security CVEs and address them by signing the image and publishing it. This process ensures that all of our images, including the vexxhost/docker-openstack-horizon, are signed across the entire stack. You can review the entire build workflow over GitHub.

Continuous Security

It's important to note that an image that was secure yesterday might not be secure today. Understanding that security is a continuous process, we run daily scans on all images to identify any exposed CVEs. Any potential vulnerabilities are addressed immediately, ensuring continuous security for all images.

The integration of Docker Content Trust, Trivy, and Cosign into Atmosphere's image building process represents a significant step forward in enhancing security across the entire stack. This process guarantees the reliability, integrity, and security of the images used, making Atmosphere a secure and trusted choice for all your cloud computing needs.