Introduction to SingularityCE¶
There are no specific skills needed beyond a basic comfort with the command line and using a text editor. Prior experience installing Linux applications could be helpful but is not required.
Important: Singularity is compatible with Docker, but they do have distinct differences.
Inside a Docker container the user has escalated privileges, effectively making them root on that host system. This privilege is not supported by most administrators of High Performance Computing (HPC) centers. Meaning that Docker is not, and will likely never be, installed natively on your HPC.
Same user inside as outside the container
User only has root privileges if elevated with sudo when container is run
Can run (and modify!) existing Docker images and containers
These key differences allow Singularity to be installed on most HPC centers. Because you can run virtually all Docker containers in Singularity, you can effectively run Docker on an HPC.
2. SingularityCE Installation¶
Sylabs Singularity Community Edition (CE) homepage: https://www.sylabs.io/docs/
SingularityCE is more likely to be used on a remote system that you don’t have control of (i.e. HPC & HTC).
2.1 Install Singularity on Laptop¶
To Install SingularityCE follow the instructions for your specific OS: https://sylabs.io/guides/3.8/user-guide/quick_start.html
Load the SingularityCE module on HPC
If you are interested in working on HPC, you may need to contact your systems administrator and request they install Singularity. Because singularity ideally needs setuid, your admins may have some qualms about giving Singularity this privilege. If that is the case, you might consider forwarding this letter to your admins.
Most HPC systems are running Environment Modules with the simple command module.
You can check to see what is available:
$ module avail singularity
If Singularity is installed, load a specific version:
$ module load singularity/3/3.7.2
2.3 Atmosphere Cloud¶
CyVerse staff have deployed an Ansible playbook called
ez for software installation which includes Singularity. This command only requires you to type a short line of code to install an entire software stack with all of its dependencies.
Start any Featured instance on Atmosphere <../cyverse/boot.html>_.
Type in the following in a web shell or
$ ezs -r 3.7.3 DEBUG: set version to 3.7.3 * Updating ez singularity and installing singularity (this may take a few minutes, coffee break!) Cloning into '/opt/cyverse-ez-singularity'... remote: Enumerating objects: 24, done. remote: Total 24 (delta 0), reused 0 (delta 0), pack-reused 24 Unpacking objects: 100% (24/24), done. * ez singularity or singularity itself may not have updated successfully, but you can probably try executing it To test singularity, type: singularity run shub://vsoch/hello-world Hint: it should output "RaawwWWWWWRRRR!!")
2.4 Check Installation¶
Singularity should now be installed on your laptop or VM, or loaded on the HPC, you can check the installation with:
$ singularity pull shub://vsoch/hello-world INFO: Downloading shub image 59.75 MiB / 59.75 MiB [=====================================================================================================] 100.00% 49.24 MiB/s 1s tswetnam@tysons-box:~$ singularity run hello-world_latest.sif RaawwWWWWWRRRR!! Avocado!
Singularity’s command line interface allows you to build and interact with containers transparently. You can run programs inside a container as if they were running on your host system. You can easily redirect IO, use pipes, pass arguments, and access files, sockets, and ports on the host system from within a container.
The help command gives an overview of Singularity options and subcommands as follows:
$ singularity Usage: singularity [global options...] <command> Available Commands: build Build a Singularity image cache Manage the local cache capability Manage Linux capabilities for users and groups config Manage various singularity configuration (root user only) delete Deletes requested image from the library exec Run a command within a container inspect Show metadata for an image instance Manage containers running as services key Manage OpenPGP keys oci Manage OCI containers plugin Manage Singularity plugins pull Pull an image from a URI push Upload image to the provided URI remote Manage singularity remote endpoints run Run the user-defined default command within a container run-help Show the user-defined help for an image search Search a Container Library for images shell Run a shell within a container sif siftool is a program for Singularity Image Format (SIF) file manipulation sign Attach a cryptographic signature to an image test Run the user-defined tests within a container verify Verify cryptographic signatures attached to an image version Show the version for Singularity Run 'singularity --help' for more detailed usage information.
Information about subcommand can also be viewed with the help command.
$ singularity help pull Pull an image from a URI Usage: singularity pull [pull options...] [output file] <URI> Description: The 'pull' command allows you to download or build a container from a given URI. Supported URIs include: library: Pull an image from the currently configured library library://user/collection/container[:tag] docker: Pull an image from Docker Hub docker://user/image:tag shub: Pull an image from Singularity Hub shub://user/image:tag oras: Pull a SIF image from a supporting OCI registry oras://registry/namespace/image:tag http, https: Pull an image using the http(s?) protocol https://library.sylabs.io/v1/imagefile/library/default/alpine:latest Options: --arch string architecture to pull from library (default "amd64") --dir string download images to the specific directory --disable-cache dont use cached images/blobs and dont create them --docker-login login to a Docker Repository interactively -F, --force overwrite an image file if it exists -h, --help help for pull --library string download images from the provided library (default "https://library.sylabs.io") --no-cleanup do NOT clean up bundle after failed build, can be helpul for debugging --nohttps do NOT use HTTPS with the docker:// transport (useful for local docker registries without a certificate) Examples: From Sylabs cloud library $ singularity pull alpine.sif library://alpine:latest From Docker $ singularity pull tensorflow.sif docker://tensorflow/tensorflow:latest From Shub $ singularity pull singularity-images.sif shub://vsoch/singularity-images From supporting OCI registry (e.g. Azure Container Registry) $ singularity pull image.sif oras://<username>.azurecr.io/namespace/image:tag For additional help or support, please visit https://www.sylabs.io/docs/
3. Downloading pre-built images¶
The easiest way to use a Singularity is to
pull an existing container from one of the Registries.
library - images hosted on Sylabs Cloud
shub - images hosted on Singularity Hub
docker - images hosted on Docker Hub
localimage - images saved on your machine
yum - yum based systems such as CentOS and Scientific Linux
debootstrap - apt based systems such as Debian and Ubuntu
arch - Arch Linux
busybox - BusyBox
zypper - zypper based systems such as Suse and OpenSuse
3.1 Pulling an image from Singularity Hub¶
Similar to previous example, in this example I am pulling a base Ubuntu container from Singularity-Hub:
$ singularity pull shub://singularityhub/ubuntu WARNING: Authentication token file not found : Only pulls of public images will succeed 88.58 MiB / 88.58 MiB [===============================================================================================] 100.00% 31.86 MiB/s 2s
You can rename the container using the –name flag:
$ singularity pull --name ubuntu_test.simg shub://singularityhub/ubuntu WARNING: Authentication token file not found : Only pulls of public images will succeed 88.58 MiB / 88.58 MiB [===============================================================================================] 100.00% 35.12 MiB/s 2s
The above command will save the alpine image from the Container Library as
3.2 Pulling an image from Docker Hub¶
This example pulls an
ubuntu:16.04 image from DockerHub and saves it to the working directory.
$ singularity pull docker://ubuntu:20.04 INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures Copying blob 8f6b7df711c8 done Copying blob 0703c52b8763 done Copying blob 07304348ce1b done Copying blob 4795dceb8869 done Copying config 05ac933964 done Writing manifest to image destination Storing signatures 2020/03/09 16:14:12 info unpack layer: sha256:8f6b7df711c8a4733138390ff2aba1bfeb755bf4736c39c6e4858076c40fb5eb 2020/03/09 16:14:13 info unpack layer: sha256:0703c52b8763604318dcbb1730c82ad276a487335ecabde2f43f69a6222e8090 2020/03/09 16:14:13 info unpack layer: sha256:07304348ce1b6d24f136a3c4ebaa800297b804937a6942ce9e9fe0dac0b0ca74 2020/03/09 16:14:13 info unpack layer: sha256:4795dceb8869bdfa64f3742e1df492e6f31baf9cfc36f1a042a8f981607e99a2 INFO: Creating SIF file... INFO: Build complete: ubuntu_20.04.sif
Pulling Docker images reduces reproducibility. If you were to pull a Docker image today and then wait six months and pull again, you are not guaranteed to get the same image. If any of the source layers has changed the image will be altered. If reproducibility is a priority for you, try building your images from the Container Library.
3.3 Pulling an image from Sylabs cloud library¶
Let’s use an easy example of
alpine.sif image from the container library
$ singularity pull library://alpine:latest WARNING: Authentication token file not found : Only pulls of public images will succeed INFO: Downloading library image 2.08 MiB / 2.08 MiB [==================================================================================================] 100.00% 5.06 MiB/s 0s
You can use
singularity search <name> command to locate groups, collections, and containers of interest on the Container Library
4 Interact with images¶
You can interact with images in several ways such as
For these examples we will use a
cowsay_latest.sif image that can be pulled from the Container Library like so.
$ singularity pull library://tyson-swetnam/default/cowsay INFO: Downloading library image 67.00 MiB / 67.00 MiB [=====================================================================================================] 100.00% 5.45 MiB/s 12s WARNING: unable to verify container: cowsay_latest.sif WARNING: Skipping container verification tswetnam@tysons-box:~$ singularity run cowsay_latest.sif ________________________________________ / Expect a letter from a friend who will \ \ ask a favor of you. / ---------------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
shell command allows you to spawn a new shell within your container and interact with it as though it were a small virtual machine.
$ singularity shell cowsay_latest.sif Singularity cowsay_latest.sif:~>
The change in prompt indicates that you have entered the container (though you should not rely on that to determine whether you are in container or not).
Once inside of a Singularity container, you are the same user as you are on the host system.
$ Singularity cowsay_latest.sif:~> whoami tswetnam
shell also works with the library://, docker://, and shub:// URIs. This creates an ephemeral container that disappears when the shell is exited.
4.2 Executing commands¶
The exec command allows you to execute a custom command within a container by specifying the image file. For instance, to execute the
cowsay program within the cowsay_latest.sif container:
$ singularity exec cowsay_latest.sif cowsay container camp rocks ______________________ < container camp rocks > ---------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
exec also works with the library://, docker://, and shub:// URIs. This creates an ephemeral container that executes a command and disappears.
4.3 Running a container¶
Singularity containers contain runscripts. These are user defined scripts that define the actions a container should perform when someone runs it. The runscript can be triggered with the
run command, or simply by calling the container as though it were an executable.
singularity run lolcow_latest.sif _________________________________________ / You will remember, Watson, how the \ | dreadful business of the Abernetty | | family was first brought to my notice | | by the depth which the parsley had sunk | | into the butter upon a hot day. | | | \ -- Sherlock Holmes / ----------------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
# Exercise - 1¶
Here are the brief steps:
Go to Docker hub and look for a Dockerhub image
singularity pullcommand to pull the Docker image and convert it to .sif
singularity runcommand to launch a container from the Docker image and check to see if you get the same output that as you get from running
4.3 Running a container on HPC¶
For running a container on HPC, you need to have Singularity module available on HPC. Let’s first look to see if the Singularity module is available on HPC or not
The following instructions are from running on UA HPC. It may or may not work on other HPC. Please refer to HPC documentation to find similar commands
$ module avail singularity ------------------------------------------ /cm/shared/uamodulefiles ------------------------------------------- singularity/2/2.6.1 singularity/3/3.2 singularity/3/3.2.1 singularity/3/3.4.2 singularity/3/3.5.3
You can see that there are three different versions of Singularity are available. For this workshop, we will use
singularity/3/3.5.3. Let’s load it now
$ module load singularity/3/3.5.3