I’m in London for a six-day conference “marathon”: Cloud Native Rejekts (March 30–31), KubeCon co-located events (April 1), and KubeCon itself (April 2–4). Rejekts comes first – and I’ve just wrapped up day one. How was it?
What Are Cloud Native Rejekts
Rejekts is a small two-day conference for talks that didn’t make it into KubeCon.
In their own words:
Cloud Native Rejekts is the b-side conference giving a second chance to the many wonderful, but rejected talks leading to KubeCon + CloudNativeCon.
Thanks to sponsors, Rejekts is free of charge and takes place in an interesting venue, but with very limited capacity – so it’s hard to get in. I personally only got in from the overflow waitlist.
Yes, I’m also joining the ChatGPT trend of images remade in Studio Ghibli style.
What Do Rejekts and KubeCon Have in Common
As I mentioned above – Rejekts features talks that people submitted to KubeCon (and CloudNativeCon) but didn’t get accepted. Rejekts becomes their second chance to present.
What Interesting Things I Learned
On Sunday I attended eight talks. Here are the highlights of what I encountered.
Immutable Turtles All the Way Down – Image-Based Kubernetes to Power In-Place Updates
Probably the best talk I attended on Sunday!
Presented by Thilo Fromm.
How to leverage Flatcar Linux and systemd extensions for component updates.
First, a brief introduction to Flatcar Linux:
A community Linux distribution designed for container workloads, with high security and low maintenance.
PS: Flatcar is currently in the process of onboarding under CNCF.
Flatcar uses systemd to manage running processes within the system, which you configure using an ignition file.
Thanks to systemd-sysext, it’s possible to automate their updates – a new version is downloaded as a filesystem image, which is then mounted to the disk, and after a restart the new version starts running. Images are downloaded from a remote server – either the official one, or you can run your own.
systemd-sysext supports so-called staged updates, where the update is split into phases that must happen sequentially but can be triggered separately – first download and prepare, then independently perform the actual restart and activate the changes.
For managing Kubernetes node restarts, tools like Kured work well, or Flatcar offers its own operator.
Flatcar offers sysext-bakery – a collection of ready-made solutions that you just need to add to your ignition file.
My takeaway: Flatcar is even better than before! sysext is a great way to automatically update Kubernetes nodes with a bit of configuration (though Flatcar is independent of Kubernetes and can be used anywhere). Easy in-place updates, support for any Kubernetes distribution (vanilla kubeadm, k3s, rke2…), openness to new tools, independence (no need for official Flatcar servers).
This really excited me and I’m looking forward to running all clusters on Flatcar Linux – for example on k0s – without having to worry about updates. Everything just takes care of itself!
Building Air-Gapped Control Planes for a Global Pharma Leader Using Crossplane and Argo CD
Presented by Antonela Cukurin and Yury Tsarev.
What does building infrastructure look like for a regulated environment (pharma, healthcare), where the kube-apiserver has no public internet access – and how to build and manage such an environment using Crossplane.
First and foremost, we once again run into the chicken-and-egg problem: How do you build the first hub cluster when you don’t have one?
The solution is to run Crossplane in a CI pipeline using GitHub Actions on a GitHub-managed runner. This creates the first Bastion VM, network, resource group in Azure… The Bastion then registers itself as a self-hosted runner for GitHub Actions, where another pipeline runs to create the cluster – and from there, anything else can be created (databases, clusters, load balancers…).
My takeaway: Crossplane can be run in a pipeline – and it’s a nice way to solve the chicken-and-egg problem.
CRD Data Architecture for Multi-Cluster Kubernetes
Presented by Clay Baenziger.
Multi-cluster is no joke. Building multiple clusters today isn’t such a big problem thanks to tools like Cluster API or Karmada. But day-two operations are still the wild west. Everyone essentially needs the same thing and there are already several open-source projects tackling this – for example Azure Fleet.
Clay talked, among other things, about how they solve this at Bloomberg. Because their multi-cluster solution is fairly old, they use the tool Karmada.
He also mentioned the Kubernetes Enhancement Proposal (KEP) KEP-4322: Cluster Inventory and how it fits into multi-cluster management – and that we’re essentially waiting on SIG-Multicluster…
Another KEP – KEP-1645: Multicluster Services API – highlights that there is still no standard for connecting services across clusters.
He further mentioned so-called Tiers, which they use to handle similar/identical cluster configurations. Because they have so many clusters, they no longer manage individual clusters but added the concept of Tiers. This builds on the concept of Cluster Sameness, which in turn builds on Namespace Sameness (see the document from SIG-Multicluster).
When managing several clusters and using a “hub” Kubernetes cluster as a metadata database for clusters, etcd (the brain of Kubernetes) is often not fast enough – I recommend using a project like Kine, which was created as part of k3s.
Finally, he mentioned how complex authentication and authorization are in the multi-cluster world – both for users and workloads.
My takeaway: It’s complicated and requires experienced people. Cluster API handles day zero/one operations, but day two operations are still the wild west. I recommend looking at Open Cluster Management and considering replacing etcd with another database for hub clusters – for example PostgreSQL via Kine.
OCI Registry as a Secure and Single Source of Distribution for Your Container Images & Artifacts
Presented by Stephane Este-Gracias.
How the OCI registry is becoming practically the standard and the single registry for all artifacts.
OCI registry today supports practically everything:
- container images
- Kubernetes manifests
- ML models
- attestations (provenance proof, Git metadata, author…)
- signatures
Interesting fact: CloudNativePG plans to use OCI artifacts for distributing PostgreSQL extensions – thanks to Kubernetes 1.33+, which makes it possible to mount an OCI artifact as a volume mount into a running container.
Useful tools, not just for working with OCI:
- Oras – a tool for working with OCI artifacts
- Cosign – signing (primarily container images)
- Trivy – security vulnerability scanning
- Skopeo – a tool for working with remote container images
My takeaway: OCI is practically a repository for all artifacts, everything can be signed and have an SBOM attached – security without compromise. Kubernetes (1.33, beta feature) plans support for OCI volume mounts into containers.
End to End Message Authenticity in Cloud Native Systems
Presented by Lucas Kaldstrom and Micah Hausler.
mTLS and OIDC are not a silver bullet – so how do we secure the trustworthiness and authenticity of messages over HTTP?
Lucas and Micah talked about the new IETF RFC 9421: HTTP Message Signatures, which adds support for HTTP headers where you can add a signature and thus validate individual requests. The RFC also accounts for proxies and gateways – it’s possible to add headers that aren’t signed but won’t break the message and signature compatibility.
However, the problem of secure key distribution remains (how to securely distribute symmetric/asymmetric keys…).
Today, AWS sigv4 works similarly, and there’s a proposal to extend SPIFFE support with this RFC.
My takeaway: HTTP is evolving not only at the protocol level (HTTP/2, HTTP/3) but also in the area of security and request authentication.
Geographically Distributed Clusters: Resilient Distributed Compute on the Edge
Presented by Alex Bissessur.
A story about how on the island of Mauritius they’re solving cloud for a startup without any availability of common clouds (AWS, GCP, Azure…). And also about how the government long considered cloud to be dangerous (“cloud = bad”). Ping to South Africa is 60 ms, to Europe 600 ms – ouch!
The solution? “Embrace the homelab!”
They literally created a network of homelabs.
It’s a single k3s cluster stretched across 3 houses on the island (latency 1–2 ms), with at least three nodes in each house (nine total).
For storage they use Longhorn – and surprisingly have no issues, which surprised everyone.
The biggest challenge was networking – throughput and connectivity. They currently use Tailscale and have all nodes in a single Tailscale network. In the future, they plan to migrate to Headscale. Ingress is also handled through Tailscale.
My takeaway: Even a small and punk-style environment can comfortably run production workloads – right there on an island, with minimal resources and second-hand mini-PCs.
I’d sum up this talk with: “Where there’s a will, there’s a way.”
Recording
For those of you who aren’t in London and want to watch the talks at least from a recording – Rejekts has a public YouTube playlist with recordings of all talks.
More Articles
I’ll be adding more articles from each day of the conferences throughout the week.
