Contain(er) (Den)izen(s)

Keidrych Anton-Oates <[email protected]>

Presentation or Coffee?



MicroOS FileSystems supported by Minimal HyperVisors.



“a person, animal, or plant that lives or is found in a particular place”


Code of Conduct

  • CNCF Architecture

  • 12 Factor Compliance

  • Container = Production Candidate

  • GitOps

    • Code Quality

    • Vulnerability Scanning

    • Config / Secret

  • Continua


Gradual increase of environmental complexity until it is indistinguishable from actual production

InfoSec Synergy

Target State

  • Red: (Offensive Security) @ Security

    • An actual Black Box

  • Blue: (Defensive Security) @ Operations

    • Vulnerability & Configuration Management

  • Yellow: (Builders) @ Development

    • Crafted Code to be proud of

via Container(s)

Containers encapsulate the deployment and runtime, and large portion of security requirements of software into an aggregated, common & mostly reproducible environment.

Why Use Containers?


Works On My Machine…​

IT organizations have come up with two broad approaches to solving this problem:

  1. Make the development environment look more like production

  2. Make production look more like the development environment


“Development looks like Production”

  • Production ⇒ Dev data Illegal via GDPR

  • Secret Management

  • Restricted API Limits

  • Stingy Resource Allocation


“Production looks like Development”

  • Red Team:

  • Blue Team:

    • everyone, please observe a moment of silence

  • Yellow Team:

    • hail your new 24x7x365 System Overlords…​

history of failure..

Constraint #1

  • cloning disks/images/VMs is time-consuming and expensive

  • Containers:

    • 'My Machine ⇐⇒ Production'

    • via Golden Image

Constraint #2

  • Constraint #2: Applications aren’t ready

  • Architecture: Retire

    • DDD

    • N-Tier

  • Architecture: Embrace

    • C4 Model & arc42

    • CNCF

Container Technology

Golden Image

Golden Container

Kernel: Linux

  • Host OS provides

    • Irrelevant for Container

    • mnt, uts, ipc, pid & net managed via RunC

Union File System: OverlayFS

  • Upper → Lower

  • Modifications

    • Copy lower → upper

  • Deletes

    • File: whiteout in OverlayFS

    • Directory: opaque directory in OverlayFS



  • accounting

  • controlling

  • prioritization

  • resource limiting



  • Container != Virtual Machine

  • Container > Application Virtualization

  • Application Virtualization is now the problem

  • Anything creating a closely bound architecture abstraction layer


  • OpenJDK / JVM Anything

    • Requires 1 machine per JVM

    • Memory Leak between System & Application layers

    • Pretends the container doesn’t exist

    • Hard wired to a specific OS folder structure


  • NodeJS…​ (what!)

  • Ruby

  • Mono

  • C#

  • …​

LXCFS seeks to address these issue.

Language Affinity

Speed & Memory

  1. C/C++

  2. D

  3. Nim ⇐ Recommended

  4. Rust


How to Choose?

  • Adopt the RoR style of development (╥﹏╥)

  • Rapid prototype in the language of choice

  • Swap out critical / high throughput parts for Nim as needed

The Base Container


  • Process Zombie

  • OS Level Health Check

  • Base Image

  • Layer Caching

Process Zombie

S6: Execline

  • Designed for Embedded Systems

  • No overhead due to interactive support

  • Occupied resources by Execline are instantly freed after task (irrespective of App execution)

  • Statically compiled, minimal C dependencies

  • Kernel Forking isn’t an issue

  • Fastest launch time of any Init Manager (Faster than sh)

S6: Services

  • Notifications

  • onStart

  • onFinish

  • deathTally (ddos prevention)

OS Level Health Check

  • NVMe can timeout!

  • Goss validates everything outside your App

    • Goss provides the 'healthz' endpoint for Kubernetes

    • Separate Kernel branch reduces NodeJS loop overloading

Base Image

  • Must compile on all all Chipsets

  • Just

    • Language dependencies

    • Non-Zero User

    • Zombie Manager

    • Other Requirements

Layer Caching

  • Not important:

    • Number of Layers…​ 125 maximum

  • Important:

    • Build Time

    • Startup Time

    • Network Utilization

    • Update Propagation Time

Layer Caching…​

  • Content addressable

    • do not need relationships to eachother

  • DockerFile’s are typically written

    • with relationships between layers in mind

    • Relationships = Vulnerability?

duplicate nodes in the graph so each node is only pointed to once


replace each leaf node with a counter, starting at 1

leaf nodes

combine counters with their children, and their children’s counters summed, then incremented


repeat this process until there is only one node & sort by popularity & by name


Enter NixOS

“The Purely Functional Package Manager” — NixPkgs Manual

( ••)O* Everything is in the _manual °Q(•_• )

No-one Reads Manuals…​

nix search hello

* nixpkgs.hello (hello)
  A program that produces a familiar, friendly greeting


{ pkgs ? import <nixpkgs> {} }:

pkgs.dockerTools.buildLayeredImage {
  name      = "hello-world";
  tag       = "latest";
  contents  = [ pkgs.hello ];
  config    = ({
    Entrypoint = ["${pkgs.hello}/bin/hello"];
Finished building layer 'hello-world-granular-docker-layers'
building '/nix/store/fry4yc2vhas7107yhkjxs9g5z27jli0l-docker-
Cooking the image...

# symlinked to 'result' in same directory
docker load -i result

d51992ba410c: Loading layer 28.73MB/28.73MB
5eadf85ad725: Loading layer 266.2kB/266.2kB
359b99fc430e: Loading layer 71.68kB/71.68kB
fe6663d99767: Loading layer 10.24kB/10.24kB
59acd48b3d90: Loading layer 10.24kB/10.24kB
3dc35514b26e: Loading layer 71.68kB/71.68kB
Loaded image: hello-world:latest

docker run -rm hello-world
Hello, world!

Containizen Base Image

  • sotekton/containizen DockerHub || GitHub

  • NodeJS, Python + pull request…​

  • Rebuilt every 24 hours

  • Crafted with NixPkgs Community

  • Extensible via DockerFile / OCI / Nix Build

  • Skarnet S6 Supervision Suite for safe Process Zero+ management

Application Code

ARG version=nodejs
FROM sotekton/containizen:$version AS base

COPY . /opt/app


nix-build example.nix

Most Common Issue

# required for some applications
# i.e. Knex-Migrate

RUN ln -s /bin/env /usr/bin/env