Building and running containers from scratch images provides numerous advantages that enhance the security of an application. When building a scratch image, the final image is tailored to include only the binaries and libraries essential for running the application, eliminating unnecessary components. This lean design leads to fewer CVE scan findings which helps streamline maintenance by reducing the number of deploy-test cycles required to remain up to date with the latest CVE patches. Additionally, the nominal attack surface lowers the risk of container compromise, while the limited functionality further mitigates the potential impact of a security breach. These advantages closely align with the principle of least privilege by incorporating only the resources absolutely necessary for the application to function effectively and securely.
A container image can be thought of as a template for running a container and a scratch image is just that, an image with nothing in it. However, to really appreciate scratch images it helps to understand what makes containers possible in the first place. Containerization is made possible by a handful of operating system innovations, each of which is a study in its own right:
- Namespaces – Provide isolation for various system resources (mnt, pid, net, ipc, uts, user, cgroup).
- Cgroups – Allow the kernel to allocate and limit resources (CPU, memory, I/O, etc.).
- Union Filesystems – Enable the layering of file systems, where changes made in a container are stored in a writable layer while the base image remains read-only.
- Seccomp – Provides a way to restrict system calls that a containerized application can make.
- Capabilities – Break down root privileges into discrete capabilities (e.g., CAP_NET_ADMIN for network administration).
- pivot_root – a Linux system call that changes the root directory (/) of the calling process to a new directory in the mnt namespace. Unlike chroot, pivot_root makes use of the mnt namespace to further isolate the container filesystem.
Where Do I Start?
“The reserved, minimal scratch image serves as a starting point for building images. Using the scratch image signals to the build process that you want the next command in the Dockerfile to be the first filesystem layer in your image. While scratch appears in Docker’s repository on Docker Hub, you can’t pull it, run it, or tag any image with the name scratch. Instead, you can refer to it in your Dockerfile.” – docs.docker.com
Creating a scratch image is relatively straight forward if you know what to look out for along the way. Using multi-stage Docker builds, we can build an image in multiple stages. This way we can make use of a full image such as debian:latest to build our application, and then copy the essential binaries and files needed to run our application into the final scratch image. Let’s use an example application written in Ruby to illustrate how this works.
First, we need to add a builder stage to our original Dockerfile by appending AS builder to the first line of our Dockerfile and we need to add a second stage by adding FROM scratch to the end of our Dockerfile:
FROM ruby:3.3.6-slim AS builder
# install and upgrade packages
...
# set env variables
ENV GEM_HOME=/usr/local/bundle
ENV BUNDLE_SILENCE_ROOT_WARNING=1
BUNDLE_APP_CONFIG="$GEM_HOME"
PATH=$GEM_HOME/bin:$PATH
# install app dependencies
WORKDIR /app
COPY Gemfile Gemfile.lock /app/
COPY vendor/ /app/vendor/
RUN gem install bundler &&
bundle config set --local without 'development test' &&
bundle install
# copy app
COPY . /app/
# create the rails user
RUN useradd rails --create-home --shell /bin/sh &&
chown -R rails:rails .
# build the final image
FROM scratch
Now the Fun Part 😀
To build our scratch image we need to copy the essential files and settings from the builder stage into our scratch image. Figuring out exactly which files and settings are necessary to run an application can be challenging, but here’s how we did it with our example Ruby application:
- Build the application without the second stage and exec into the container.
docker run --platform linux/amd64 --rm --entrypoint "/bin/bash" -u root -d --name ruby-app ruby-app -c "while true; do sleep 1000; done"; docker exec -it ruby-app /bin/bash; docker stop ruby-app
- Find all the shared objects that are linked with the Ruby interpreter and the native extensions.
Using the ldd command we can find all the shared objects that our app requires at runtime. We need to locate all the shared objects that are required by the Ruby interpreter and the native extensions that are used by the app.> ldd /usr/local/bin/ruby libruby.so.3.3 => /usr/local/lib/libruby.so.3.3 (0x00007fffff1cd000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fffff1ac000) libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fffff12b000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fffff0ef000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fffff010000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffffee2f000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffffee0d000) /lib64/ld-linux-x86-64.so.2 (0x00007ffffffcc000) > find /usr/local/bundle/extensions -type f -name "*.so" | xargs -I{} sh -c 'echo "Dependencies for {}:"; ldd {}; echo' Dependencies for /usr/local/bundle/extensions/x86_64-linux/3.1.0/pg-1.5.6/pg_ext.so: libruby.so.3.1 => /usr/local/lib/libruby.so.3.1 (0x00007fffff356000) libpq.so.5 => /usr/lib/x86_64-linux-gnu/libpq.so.5 (0x00007fffff301000) libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007fffff222000) libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007fffff041000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fffff01f000) libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007ffffef9e000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007ffffef60000) /lib64/ld-linux-x86-64.so.2 (0x00007ffffffcc000) libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007ffffeeb7000) libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007ffffea31000) libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007ffffe9de000) libldap-2.5.so.0 => /lib/x86_64-linux-gnu/libldap-2.5.so.0 (0x00007ffffe97f000) libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007ffffe8a3000) libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007ffffe876000) libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007ffffe870000) libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007ffffe862000) liblber-2.5.so.0 => /lib/x86_64-linux-gnu/liblber-2.5.so.0 (0x00007ffffe852000) libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007ffffe835000) libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007ffffe617000) libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007ffffe610000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007ffffe5ff000) libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007ffffe4cb000) libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007ffffe49a000) libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007ffffe2e2000) libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007ffffe2cd000) libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x00007ffffe27f000) libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007ffffe236000) libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007ffffe22a000) ...
- Copy the libraries.
We could use the full path of the shared objects from the ldd output. However, in order to make our Dockerfile more flexible and to allow Ruby upgrades without needing to re-run the previous step with every upgrade, we will copy all the system libraries from the builder image. Be sure to inspect all the required shared objects from the output in step 2 to be sure all the dependencies are located in the /lib/x86_64-linux-gnu and /usr/lib/x86_64-linux-gnu folders.FROM scratch # copy system libraries COPY --from=builder /lib64 /lib64 COPY --from=builder /lib/x86_64-linux-gnu /lib/x86_64-linux-gnu COPY --from=builder /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu # copy ruby libraries COPY --from=builder /usr/local/lib/libruby* /usr/local/lib/ COPY --from=builder /usr/local/lib/ruby /usr/local/lib/ruby COPY --from=builder /usr/local/bundle /usr/local/bundle
- Copy the binaries.
# copy binaries COPY --from=builder /usr/local/bin/ruby /usr/local/bin/ruby COPY --from=builder /usr/local/bin/gem /usr/local/bin/gem COPY --from=builder /usr/local/bin/bundle /usr/local/bin/bundle
- Copy the time zone and SSL directories.
# copy timezone and SSL directories COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo COPY --from=builder /etc/ssl /etc/ssl
- Set the environment variables.
# set environment variables ENV GEM_HOME=/usr/local/bundle ENV BUNDLE_SILENCE_ROOT_WARNING=1 BUNDLE_APP_CONFIG=$GEM_HOME LANG=C.UTF-8 TZ=America/New_York SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt SSL_CERT_DIR=/etc/ssl/certs PATH=/usr/local/bundle/bin:$PATH
- Put on the finishing touches.
# copy user and group files COPY --from=builder /etc/passwd /etc/passwd COPY --from=builder /etc/group /etc/group COPY --from=builder /home/rails /home/rails # copy application COPY --from=builder /app /app # run as the rails user USER rails:rails # set work directory WORKDIR /app
Additional Considerations
Containers that need to support exec capabilities, such as docker exec, will need additional binaries such as /bin/sh and /bin/bash for the exec functionality to work correctly. Here are some binaries you may want to consider if your containers require exec capabilities.
# copy binaries to support exec capabilities
COPY --from=builder /bin/sh /bin/sh
COPY --from=builder /bin/bash /bin/bash
COPY --from=builder /bin/ls /bin/ls
COPY --from=builder /bin/cat /bin/cat
COPY --from=builder /bin/echo /bin/echo
COPY --from=builder /bin/ps /bin/ps
COPY --from=builder /bin/grep /bin/grep
COPY --from=builder /usr/bin/find /usr/bin/find
COPY --from=builder /usr/bin/env /usr/bin/env
COPY --from=builder /usr/bin/which /usr/bin/which
COPY --from=builder /usr/bin/sort /usr/bin/sort
COPY --from=builder /usr/bin/tail /usr/bin/tail
COPY --from=builder /usr/bin/top /usr/bin/top
Potential Downsides
Depending on the specific app configuration it may be difficult to find all the shared objects that are linked with the compiled objects used by the app. In our example we took a cautious approach and copied all the system libraries from our builder stage into our scratch image and we spot checked the output of the ldd commands to be sure that we copied everything we needed. In most cases the required shared objects will be in one of the two system folders that we copied, but there could be cases where a shared object is located in a non-standard folder and is not included in the scratch image. While a full regression test of the app would catch an error such as this, full regression testing can be costly. In light of this, it is important to weigh the pros and cons of using scratch images for your particular use case.
That’s a Wrap!
Utilizing scratch images and multi-stage Docker builds offers a powerful approach to enhancing container security. By stripping down containers to include only the essential components, scratch images reduce the attack surface and streamline vulnerability management. While there may be challenges in identifying all of the shared object dependencies, especially for complex applications, the effort is well worth the improvements in security and performance. As a bonus, scratch images are typically much smaller in size. For instance, the scratch image from our Ruby example is 37% smaller than the fully optimized debian-slim based builder image. For some deployments, the size differences alone warrant the effort as large microservices environments seek to reduce container start times. Despite the added complexity of building scratch images, their lean, secure, and efficient design makes them a valuable tool for modern containerized applications.
Loved the article? Hated it? Didn’t even read it?
We’d love to hear from you.
