Containers and Windows: WSL2 allows Microsoft machines to create and run singularity and docker images!

Containers and Windows: WSL2 allows Microsoft machines to create and run singularity and docker images!

For those that have used Teton, you’ll know that it can be a little bit of a pain to use when you don’t have the software or packages you need and have to wait for them to be installed. It can also be a pain when different parts of your workflow require different versions of some software. These hassles can be somewhat mollified by using environments and containers on Teton. As far as I can tell, environments and containers do the same thing, but in very different ways (so, which one is preferred will depend on context): For our purposes, they both provide isolated contexts that allow one to install software and packages (among other things), without the (administrative) worries of giving users “super user” privileges (which could result in lots of problems on Teton). While Conda environments are not perfect, they do seem like a good option for most common issues. Dylan Perkins will be talking about Conda environments later this week during the Data Science Working Group.

The other option, containers, are a bit more advanced, but are also a good choice, especially if you want to share your complete Linux environment with others – this could include all code, dependencies, data, etc. making this option particularly appealing from a reproducibility perspective. There are multiple container programs, but I’m specifically talking about singularity, since that program is used by Teton.

One issue with singularity is that in order to develop the “image” that can then be used on Teton, one needs two things: 1) sudo (super user) privileges, and 2) a version of Linux with singularity install. Since regular users will never have sudo privileges on Teton, that means we have to develop our containers somewhere else. This was a problem for Windows users, because using Linux with Windows did not used to be very easy – one needed to dual-boot, setup a virtual machine, have a second computer, etc. Enter: Windows Subsystem for Linux (WSL). WSL provided a way for Windows users to interact with Linux on a Windows machine that was almost completely native to Linux. Hence we could get sudo powers in Windows – thus problem 1 from above was solved. However, WSL is not a fully integrated Linux kernel, meaning it could not directly interact with the system calls required by programs like singularity, thus problem 2 persisted: Enter Windows Subsystem for Linux 2 (WSL2).

WSL2 has full system call compatibility, allowing it to potentially support the installation of singularity. When I learned about this potential over the weekend I just had to try it out. I won’t get into the gory details here (that’s for another post, where I will try to provide instructions), but long story short: It appears it is possible to run and create singularity containers from a Windows machine. It even works with X11 forwarding (which allows Linux to display graphics in Windows) after a bit of configuring of an X11 server (Xming, in my case). Below is a screenshot of “shelling” into a singularity container (lolcow):

What I’ve done is “built” an example container from the singularity documentation that is writable (requiring the use of sudo). Now that the container is built I’m using the --writable argument flag to interactively develop said container, so that any changes I make to the container are permanent. I could have also made the container a “sandbox” such that all changes I made to the container weren’t permanent. This would be useful for certain kinds of prototyping. It is also possible to create a read-only version of the image. I could see this being useful for when one is ready to share an image as part of a paper – where you don’t want people to actually be able to change anything.

Anyway, just wanted to share that little bit of info, in case anyone is interested. Like I said, as I learn more, I’ll try to put together a How-to for 1) Installing singularity on a Windows machine via WSL2, and 2) creating a simple singularity container that you can then use with Teton.