8  Week 3 Reading: Containers

8.1 Containers

We already learned about software modules (Section 3.5) on the gizmo cluster. There is an alternative way to use software: pulling and running a software .

8.2 Terminology

In order to be unambiguous with our language, we’ll use the following definitions:

graph TD
    A["Docker Image"] --"apptainer pull/apptainer build"--> B
    B["Docker Container"] --"apptainer exec"--> C
    C["Cached Image"]

  • Registry - collection of repositories that you pull docker images from. Example repositories include DockerHub and Quay.io.
  • Image - what you download from a registry - the “recipe” for building the software environment. Stored in a registry. use apptainer pull to get image from a registry. Can also generate image from a Dockerfile
  • Container - The executable software environment actually installed and running on a machine. Generate from apptainer pull from a repository. The main point is that it is the running software.
  • Cached Image - A file that contains the Docker container on the system that can be run (for Apptainer, they are called .sif files and are converted from Docker Images). We cache images to prevent from downloading the image every time we run.
  • Dockerfile - a file that specifies how to install software and its dependencies. You often base a Dockerfile on an existing Docker image
  • Tag - a bit of metadata that is used to version a container.

8.2.1 What is a Container?

A container is a self-contained unit of software. It contains everything needed to run the software on a variety of machines. If you have the container software installed on your machine, it doesn’t matter whether it is MacOS, Linux, or Windows - the container will behave consistently across different operating systems and architectures.

The container has the following contents:

  • Software - The software we want to run in a container. For bioinformatics work, this is usually something like an aligner like bwa, or utilities such as samtools
  • Software Dependencies - various software packages needed to run the software. For example, if we wanted to run tidyverse in a container, we need to have R installed in the container as well.
  • Filesystem - containers have their own isolated filesystem that can be connected to the “outside world” - everything outside of the container. We’ll learn more about customizing these with bind paths (Section 9.2.3).

In short, the container has everything needed to run the software. It is not a full operating system, but a smaller mini-version that cuts out a lot of cruft.

Containers are . They leverage the the file system of their host to manage files. These are called both Volumes (the Docker term) and Bind Paths (the apptainer term).

Please note that when you run containers, any files you have stored in them vanish. Which is why it’s important to talk about Bind Paths (Section 9.2.3), which let you connect the container filesystem to the external filesystem.

8.2.2 Docker vs. Apptainer

There are two basic ways to run Docker containers:

  1. Using the Docker software
  2. Using the Apptainer software (for HPC systems)

In general, Docker is used on systems where you have a high level of access to the system. This is because docker uses a special user group called docker that has essentially root level privileges. This is not something to be taken lightly.

This is not the case for HPC systems, which are shared and granting this level of access to many people is not practical. This is when we use (which used to be called Singularity), which requires a much lower level of user privileges to execute tasks. For more info, see Section 9.2 .

WarningBe Secure

Before we get started, security is always a concern when running containers. The docker group has elevated status on a system, so we need to be careful that when we’re running them, these containers aren’t introducing any system vulnerabilities. Note that on HPC systems, the main mechanism for running containers is apptainer, which is designed to be more secure.

These are mostly important when running containers that are web-servers or part of a web stack, but it is also important to think about when running jobs on HPC.

Here are some guidelines to think about when you are working with a container.

  • Use vendor-specific Docker Images when possible.
  • Use container scanners to spot potential vulnerabilities. DockerHub has a vulnerability scanner that scans your Docker images for potential vulnerabilities. For example, the WILDS Docker library employs a vulnerability scanner and the containers are regularly patched to prevent vulnerabilities.
  • Avoid kitchen-sink images. One issue is when an image is built on top of many other images. It makes it really difficult to plug vulnerabilities. When in doubt, use images from trusted people and organizations. At the very least, look at the Dockerfile to see that suspicious software isn’t being installed.