Digging into Docker, Container and Colima
The world of containers has always been a mystery to me. Since Docker Desktop requires a license for enterprise use and colima becomes the popular alternative, I have found myself incapable of grasping what’s happening behind the scenes a few times. This has significantly slowed me down and added a lot of confusion to me; therefore, I decide to unbox this mysterious black box called Docker.
What triggered this?
I am recently spending time building my knowledge in Serverless compute. AWS SAM (Serverless Application Model) is the thing I most want to learn. It provides an abstract declarative syntax to describe the serverless application as CloudFormation (in fact, SAM is built on top of CloudFormation). What I love most about SAM is that its CLI tool provides the capability to run and debug Lambda functions locally. That means, your serverless functions do not need to be deployed before you are able to see how it behaves on the cloud.
How does SAM achieve this? It’s actually not as complicated as rocket science. All SAM has done is to spin up an AWS Lambda runtime as a Docker container and stuff the serverless functions in it, then you should be able to run your serverless functions locally as if they are on the cloud. To emulate this environment, all you need to do is simply run sam local start-api. Sounds easy, huh? That’s where my nearly one hour was spent.
Here is what I got after issuing the command. All I was running was a simple hello world API behind an API gateway. If you send a GET request to the endpoint, it will return a hello world message in JSON format. It worked nicely on the cloud, but it just didn’t run on my local. After a very prolonged investigation, I concluded it was another Docker related issue, and I managed to fixed it by running the following commands:
colima delete # Delete my current Colima VM
colima start --mount $HOME:ro # Recreate the VM and mount the home directory as read only
So, the problem that happened to me is SAM somehow failing to mount my serverless code directory to the Docker container but with no error. As you can see from the prompt, SAM did attempt to mount /Users/xudong-yang/sam-app/hello_world as /var/task: ro. It just didn’t succeed due to some Colima issue (I am still not sure what it was). As a result, I had an empty Lambda runtime and Unable to import module 'app': No module named 'app' is exactly what I should get from it.
The architecture of Docker
To demystify Docker, I decided to understand how Docker works.
Despite the evolution Docker has developed these years, this is the current Docker architecture at the time of writing.
Docker client is usually the command line interface we use.
Docker daemon, which used to be a big monolithic fully functional server side business logic, is now mostly like a controller to serve the REST API with which the docker client will interact. It also handles Docker image management, authentication and container orchestration.
The next layer under daemon is containerd. It creates the OCI bundle corresponding to the user requirement and instructs runc to start the container.
When a container is started, runc will exit and transfer the container to shim, which will keep the standard I/O and to my understanding interact with the kernel.
What is lima?
Firstly, it’s worth calling out the fact that most Docker containers are built based on the Linux kernel. Docker for Windows is a bit complicated and I will skip it there. The short summary is that Docker does support both Linux and Windows mode, but Native Windows Container is incompatible to the majority of the Docker ecosystem (which is Linux based). It is by default to, and very likely people will, choose the Linux based Docker on Windows. For mac, Native Mac Container simply doesn’t exist. If we ignore the existence of Native Windows Container, both Docker for Windows and Docker for Mac run Docker in a Linux virtual machine.
To verify this, let’s run docker version on Mac.
As you can see, the OS/Arch for my client is darwin/arm64, but the OS/Arch for my server is linux/arm64.
Now let’s talk about lima . The name of lima comes from the combination of Linux and mac. It described its motiviation in its own README file like below.
The goal of Lima is to promote containerd including nerdctl (contaiNERD ctl) to Mac users, but Lima can be used for non-container applications as well.
I digged a bit further into nerdctl, and I found it is actually an official CLI tool of containerd. It provides the nearly identical command interface as the command docker. If we have nerdctl installed, I suspect a simple alias docker=nerdctl command will do the magic to “install docker” (not really 👀).
Generally, nerdctl makes it possible to directly interact with cotainerd, without having to install the Docker CLI and going through the layer of Docker daemon. Lima brings containerd, nerdctl and Linux virtual machines to mac. Coincidentally, it just became the perfect alternative on mac to Docker Desktop when Docker Desktop became paid for enterprises.
What is colima?
Colima means “Containers in lima”.
I tried to understand it by reading its source code, but I am not very good at Go so that didn’t go very well. However, I do believe the general idea of colima is to keep using Docker CLI and serve a Docker daemon based on the foundation of lima to provide the similar functionality as Docker Desktop.
The image is my understanding of the colima architecture. Should I understand it correctly, the general architecture is still the same as Docker, and colima just replaced the daemon where Docker Desktop used to sit.
What has happened to me?
I still haven’t fully understood what has happened to me. According to the maintainer of colima, $HOME is by default mounted as writable volume. Theoretically, I shouldn’t have to recreate the Linux VM of my lima and manually mount my home directory (as read only). I didn’t try to reproduce this error because it doesn’t sound that fascinating to me 🤷.
There is indeed an issue on the GitHub page reporting not throwing an error upon the failure of mount.
I have also found a better way to change the mount point of an existing VM without deleting and recreating it: colima start --edit. Next time I won’t need to go through the whole process of recreating a new VM. Hopefully, I wish this has made me more knowledgeable next time when I have to debug Docker on my computer.