Here's a simple command to show you much disk space is being taken up by Docker images, containers, volumes and build cache.
The file docker.raw makes the backups size to appear 60GB more than it is. Attached are screen shots. The file really is not 60GB, according to.
Docker has a system
sub-command that has a few useful commands. One of them is docker system df
which reports back disk space usage stats of your Docker installation.
- Building this image results in a final size of 567.3 MB, and we can perform a docker history to see the size of its layers. Doing so demonstrates that JDK 1.8 has added 203.5 MB, and WildFly has added 159.1 MB to the total image size – that's already a lot of space!
- -t nginx-armhf:production Reclaiming the space in Docker API 1.25+ If we really want to reclaim those extra 5MiB from qemu-arm-static, we can use the new experimental –squash flag. Docker build -squash.t nginx-armhf:production. Honestly, it was about time that they did something to help us keep the image size down.
- From docker import fromenv from dockerimagesizelimit import checkimagesize oversize = checkimagesize (fromenv , 'image-name:latest', '1 GiB') assert oversize.
Docker Raw Socket
Here's what the output of docker system df
looks like on my machine:
TYPE | TOTAL | ACTIVE | SIZE | RECLAIMABLE |
Images | 212 | 0 | 63.72GB | 63.72GB (100%) |
Containers | 0 | 0 | 0B | 0B |
Local Volumes | 77 | 0 | 1.01GB | 1.01GB (100%) |
Build Cache | 0 | 0 | 0B | 0B |
This is on a machine with dozens of real world Dockerized projects that are mostly Flask, Rails, Phoenix and NodeJS apps spanning across multiple versions.
My build cache is empty because I run docker system prune
as a daily scheduled task. If you want to see how to set that up on any OS, check out Docker Tip #32.
You can pry even deeper by using the -v
flag (verbose). It will show you the unique image size for each image. You'd be surprised at how small some of them are!
What does your df output look like? Let me know in the comments below.
In this tutorial, we'll detail how to deploy self-hosted Github Actions runners to DigitalOcean with Docker. We'll also see how to scale the runners both vertically (via Docker Compose) and horizontally (via Docker Swarm).
Dependencies:
- Docker v19.03.8
- Docker-Compose v1.25.4
- Docker-Machine v0.16.2
Docker Raw Size Chart
Contents
GitHub Actions
GitHub Actions is a continuous integration and delivery (CI/CD) solution, fully integrated with GitHub. Jobs from a GitHub Actions workflow are run an applications called runners. You can either use GitHub-hosted runners or run your own self-hosted runners on your own infrastructure.
Runners can be added either to an individual repository or to an organization. We'll go with the latter approach so that the runners can process jobs from multiple repositories in the same GitHub organization.
Before beginning, you'll need to create a personal access token. Within your Developer Settings, click 'Personal access tokens'. Then, click 'Generate new token'. Provide a descriptive note and select the repo and workflow scopes.
DigitalOcean Setup
First, sign up for a DigitalOcean account if you don't already have one, and then generate an access token so you can access the DigitalOcean API.
Add the token to your environment:
Install Docker Machine if you don't already have it on your local machine.
Spin up a single droplet called runner-node
:
Docker Deployment
SSH into the droplet:
Add the following Dockerfile, taking note of the comments:
Update the RUNNER_VERSION
variable with the latest version of the runner, which can be found here.
Add the start.sh: file as well:
The ORGANIZATION
and ACCESS_TOKEN
(GitHub personal access token) environment variables are used for requesting a runner registration token.
For more, review the Create a registration token for an organization section from the documentation.
Take note of:
Essentially, the cleanup logic, for removing the runner when the container is brought down, will execute when the container is stopped.
For more on shell signal handling along with wait
, review this Stack Exchange answer.
Build the image and spin up the container in detached mode:
Make sure to replace and
with your organization and personal access token, respectively.
Take a quick look at the container logs:
You should see something similar to:
Then, under your GitHub organization name, click 'Settings'. In the left sidebar, click 'Actions'. You should see a registered runner under 'Self-hosted runners':
To test, add runs-on: self-hosted
to a repository's workflow YAML file. Then, run a new build. Back in your terminal, within the Docker logs, you should see the status of the job:
Bring down the container once done:
View the logs again. You should see:
The runner should no longer be available under 'Self-hosted runners' within your GitHub organization's Actions settings:
Remove the container:
Vertical Scaling with Docker Compose
Want to spin up more than one runner on a single droplet?
Start by adding the following docker-compose.yml file to the box:
Make sure to replace and
with your organization and personal access token, respectively.
Follow the official installation guide to download and install Docker Compose on the droplet, and then build the image:
Spin up two container instances:
You should see two runners on GitHub:
Kick off two builds. Open the Compose logs:
You should see something like:
You can scale down like so:
Exit from the SSH session and destroy the Machine/droplet:
Horizontal Scaling with Docker Swarm
Want to scale horizontally across multiple DigitalOcean droplets?
Configure Droplets
Add the DigitalOcean access token to your environment:
Spin up three new DigtialOcean droplets:
Initialize Swarm mode on the first node, runner-node-1
:
The runner should no longer be available under 'Self-hosted runners' within your GitHub organization's Actions settings:
Remove the container:
Vertical Scaling with Docker Compose
Want to spin up more than one runner on a single droplet?
Start by adding the following docker-compose.yml file to the box:
Make sure to replace and
with your organization and personal access token, respectively.
Follow the official installation guide to download and install Docker Compose on the droplet, and then build the image:
Spin up two container instances:
You should see two runners on GitHub:
Kick off two builds. Open the Compose logs:
You should see something like:
You can scale down like so:
Exit from the SSH session and destroy the Machine/droplet:
Horizontal Scaling with Docker Swarm
Want to scale horizontally across multiple DigitalOcean droplets?
Configure Droplets
Add the DigitalOcean access token to your environment:
Spin up three new DigtialOcean droplets:
Initialize Swarm mode on the first node, runner-node-1
:
Use the join token from the output of the previous command to add the remaining two nodes to the Swarm as workers:
For example:
You should see:
Build Docker Image
This time let's build the image locally and push it up to the Docker Hub image registry.
Dockerfile:
start.sh:
Build the image:
Make sure to replace with your Docker Hub username. Then, push the image to the registry:
Deploy
To deploy the stack, first add a Docker Compose file:
Again, update as well as the environment variables.
Point the Docker daemon at runner-node-1
and deploy the stack:
List out the services in the stack:
You should see something similar to:
Make sure the runner is up on GitHub:
Let's add two more nodes:
Verify:
Kick off a few jobs and ensure they successfully finish.
Scale back down to a single runner:
Verify:
Bring down the Machines/droplets once done: