# Riptides Blog — Full Text > Full Markdown of every published Riptides blog post covering secure workload > identity, kernel-level security, AI agent infrastructure, and zero-trust > architecture. See https://riptides.io/llms.txt for the curated site index. This file contains 77 posts. --- ## Riptides on Windows: transparent mTLS inside WSL2 - URL: https://blog.riptides.io/riptides-on-wsl2 - Published: 2026-09-14 - Author: Nandor Kracser - Category: Development - Tags: wsl, windows, development, kernel-module, non-human identity ## Windows developers run Linux too Most developers on a Windows laptop run their code in WSL2. The toolchain lives there, and so do the calls to internal APIs. Riptides works at the socket level inside the Linux kernel, and WSL2 is Linux, so in principle this should have worked from day one. It didn't. WSL2 boots a kernel Microsoft builds itself and publishes no headers for, and a kernel module has nothing to build against without them. As of v0.6.9 it works. Riptides installs into a WSL2 distribution and behaves as it does on any other Linux host: SPIFFE identity, transparent mTLS, credentials injected on the wire. No application changes. One command in PowerShell (`-d Ubuntu` installs the current LTS, 26.04 at the time of writing): ```powershell wsl --install -d Ubuntu --no-launch wsl -d Ubuntu -u root -- bash -c "curl -fsSL https://docs.riptides.io/install.sh | bash -s -- \ --controlplane-url https://.console.riptides.io \ --join-token \ --wait-ready" ``` ![Riptides installed inside WSL2 on Windows](../../assets/riptides-on-wsl2/wsl2-install.jpg) The daemon gets a SPIFFE identity, the kernel module loads against Microsoft's WSL2 kernel, and the machine shows up in the console like any other host. ## It works with VS Code Remote ![VS Code connected to a WSL2 distribution](../../assets/riptides-on-wsl2/vscode-wsl.png) Most people never open a WSL2 shell directly. They open a folder in VS Code with the WSL extension, which runs the editor's server inside the distribution. Everything the editor starts from there, a terminal, a test run, whatever an AI assistant kicks off, is a process in that WSL2 instance. It gets an identity and policy like anything else, and there is nothing to configure in the editor. The console picks those up as workloads. Here the editor's own `node` processes on the `nandiWSL2` node, with what each one connected to and whether it was encrypted: ![The Riptides console listing VS Code's node processes on the nandiWSL2 node](../../assets/riptides-on-wsl2/vscode-connections.png) It's the same pattern as [running Copilot in a Lima VM](/blog/secretless-ai-development-github-copilot-lima) on macOS. AI-generated code runs in the sandbox and can still reach internal services over mTLS, with short-lived credentials injected on the wire. No secret sits in the environment for that code to log or commit by accident. ## Why WSL2 needed work Riptides is a kernel module, so it has to match the kernel it loads into. On Ubuntu or Amazon Linux that's routine: the distro publishes kernel headers, and we build a package per kernel version. WSL2 doesn't fit that shape. - The kernel is Microsoft's, not the distribution's. Inside Ubuntu on WSL2, `/etc/os-release` says Ubuntu while `uname -r` says `6.18.33.2-microsoft-standard-WSL2`. Ask Ubuntu for headers for that kernel and you get nothing. - There's no kernel headers package for it at all. Microsoft has never published one ([the issue](https://github.com/microsoft/WSL/issues/11557) has been open for years), so DKMS has nothing to build against. - `wsl --update` replaces the kernel. A module built for the previous version stops loading. Every WSL distribution on a machine shares that one Microsoft kernel, which works in our favour: a single driver package serves Ubuntu, Debian and openSUSE alike. WSL2 actually needs fewer builds than a normal distro target, one per Microsoft kernel rather than one per distro and kernel pair. Our driver loader keys off the kernel release instead of `/etc/os-release` for that reason. ## The headers, open sourced To build against a kernel with no headers package you need the kernel tree. So we build it from Microsoft's own tags, prune it down to the subset a distro `kernel-devel` package ships, and publish it as a container image: ```dockerfile FROM ghcr.io/riptideslabs/wsl2-kernel-headers:6.18.33.2-x86 COPY . /src WORKDIR /src RUN KVERSION="$(cat /kversion)" make ``` That's [riptideslabs/wsl2-kernel-headers](https://github.com/riptideslabs/wsl2-kernel-headers), public and Apache-2.0, with a nightly job that tracks Microsoft's live kernel branches for both x86 and arm64. The images include BTF, which stock WSL2 kernels require of any module they load. If you've ever tried to build ZFS, WireGuard or anything else out of tree on WSL2 and hit the headers wall, help yourself. None of it is specific to Riptides. ## Try it If you have a Windows machine with WSL2, the command above is all of it. In the console, the Attach Daemon dialog has a WSL2 tab that generates the whole thing with your control plane URL and a fresh join token already filled in. The [deployment docs](https://docs.riptides.io/deployment/daemon-bare-metal/#wsl2-on-windows) cover kernel matching, what to do after `wsl --update`, and the systemd check. Windows on ARM works the same way. If you're on a Snapdragon laptop and want to try it, get in touch, we'd like to hear how it goes on real hardware. --- ## Agents Changed the Workload and SPIFFE Is Changing With It - URL: https://blog.riptides.io/spiffe-roadmap - Published: 2026-09-03 - Author: Janos Matyas - Category: SPIFFE - Tags: AI Agents, SPIFFE Every standard faces the same quiet test. The world it was written for keeps moving, and the standard either moves with it or slowly turns into a description of how things used to work. Protocols that stop evolving do not fail loudly. They just start describing a smaller and smaller slice of reality until people route around them. SPIFFE recently [published its roadmap](https://spiffe.io/blog/2026-08-18-spiffe-standard-roadmap/) for the next twelve months, and it is a good example of a standard passing that test in the open. Riptides has written a lot about SPIFFE, because we build on it. Workload identity is the foundation we stand on, and SPIFFE is the substrate we chose to stand on it with. So when the people who steward the specification lay out where it is going, we read closely. What stands out is not any single item on the list. It is the shape of the list as a whole. Read together, the roadmap is a response to how workloads actually run today, agents included. It is a standard listening to the ground beneath it. ## From the bottom turtle to an identity layer SPIFFE describes its own history as a climb. It started as a bootstrap credential, the bottom turtle, the one identity you could hand a workload before it had anything else. Over time it grew into something larger: an identity abstraction layer that spans clouds, clusters, and machines, so that a workload in one place can prove who it is to a workload in another without either side sharing a secret. The roadmap is the next stretch of that same climb. And the interesting thing is where it is pointed. Almost every item answers a way the world has changed since the original model was drawn. ## The workload no longer lives where you can reach it The original SPIFFE model assumes you control the host the workload runs on. You put an agent on the node, and the node hands out identity locally. That assumption held for a long time. It is holding less well every year. The roadmap is candid about this. It names the environments the current model leaves out: Kubernetes clusters with fully managed nodes like EKS Auto Mode and GKE Autopilot, where there is nowhere to place a node agent. Serverless platforms like Lambda, Cloud Run, and Azure Functions. CI systems like GitHub Actions. Managed AI-agent platforms. In all of them the workload is real and needs an identity, but the place you would normally put the machinery is not yours to touch. A remote Workload API, standardized filesystem delivery, and upstreamed Windows support are all answers to the same question: how does SPIFFE reach a workload you cannot install anything next to. That is not a small feature. It is the "For Everyone" in Secure Production Identity Framework For Everyone being taken seriously. A standard that only worked where you owned the host would quietly be a standard for a shrinking set of deployments. Admitting that edge, and moving to close it, is what a healthy specification looks like. ## The caller is no longer deterministic For most of the history of workload identity, the question was simply who is calling. A service had a fairly predictable set of things it talked to, and identity told you which service was on the other end of the connection. AI agents break that assumption in a specific way. An agent decides at runtime what to do, based on model output, so the same identity can take a different path through your infrastructure on every run. The roadmap names non-deterministic workloads directly and treats them as a first-class part of the environment SPIFFE now has to serve. That matters more than it might look. A standard that pretended agents did not exist would be describing 2019. Naming them, and asking what identity has to account for when the caller is autonomous, is the standard staying current with the systems people are actually shipping. ## The boundary is no longer one organization Federation has been part of SPIFFE for a while, so workloads in different trust domains can validate each other's identities. But discovery has been manual. A peer has to learn out of band where your bundle lives, then configure the URL, the profile, and the trust domain name correctly, by hand, before anything works. That is fine when two teams sit down and exchange details once. It does not scale to a world where workloads, and increasingly agents, act across trust boundaries with parties they have never met. The roadmap wants to bring trust domain discovery into the identifier itself, borrowing the .well-known pattern that OAuth and OpenID Connect already use, and that MCP is now leaning on. This is the internet's own convention for "here is how you find out how to trust me" arriving in workload identity. It is the right borrow, and it points SPIFFE at open environments rather than only closed ones. ## The threat model is moving underneath all of it Underneath every one of these shifts, the cryptography has a horizon. The roadmap has started admitting post-quantum algorithms to the SVID specifications now, before any migration is urgent, so implementations are ready ahead of need rather than scrambling behind it. There is nothing flashy about tending your cryptographic foundation early. It is exactly the kind of unglamorous, years-ahead thinking you want from the standard your identity rests on. ## Why we read this as good news Riptides builds kernel-level workload identity for precisely these conditions: workloads scattered across clouds and bare metal, agents that decide what to do at runtime, identity that has to hold across environments that do not trust each other by default. We did not bet on SPIFFE because it was finished. We bet on it because a shared identity substrate is the right foundation, and because the people building it are building it in the open. A roadmap that widens that substrate to more environments, admits more kinds of workload, and extends the cryptographic horizon does not compete with that bet. It strengthens it. It is worth being honest that not every item here ships tomorrow. Some are proposals and open pull requests, and the community is still shaping them. That is the point. The direction is what matters, and the direction is toward the world modern infrastructure already lives in. Riptides will keep building on SPIFFE. --- ## Run It Yourself: The Riptides Core Capability Demo - URL: https://blog.riptides.io/riptides-core-capability-demo - Published: 2026-08-31 - Author: Nandor Kracser - Category: Identity - Tags: demo, mtls, identity, credentials, kernel, spiffe "Mutual TLS with no changes to the application" is the kind of claim that sounds like it has a catch. Usually it does. There's a sidecar to schedule, an SDK to link, a proxy variable to set, a trust store to edit. The claim survives the slide deck and then quietly dies in the proof of concept. So instead of describing it again, we published the demo we normally give in person: **** It runs on one Linux machine and takes about fifteen minutes. ## What it shows Four short acts. Each one prints its own evidence, so you don't have to take our word for any of it. - **Augmentation.** Where a workload's identity comes from: the labels the daemon collects from a running process, before any policy exists. The kernel is already tracing both connections at that point, in the clear, with no identity on them. - **mTLS between two internal services.** One HTTP leg, one Redis leg. You count the packets carrying the payload, apply the policy, then run the same capture command again. The count drops to zero while the traffic volume doesn't, so nobody can wave away "no hits" as "no traffic". Then you revoke it with a policy edit and watch the connection reset. - **Passthrough.** This is the answer to the objection we hear most often: "we already do our own TLS." Redis gets switched over to serving TLS itself. Riptides authenticates both ends and then steps out of the data path, so you get authorization without decryption. The policy doesn't change at all. The behaviour changes on its own. - **Credential injection on egress.** A GitHub token that reaches the API but is never in the workload, its environment or its image. `curl -v` shows that the request curl wrote had no `Authorization` header on it. The header went on after the bytes left the process. ![Terminal output from make act1: the daemon's labels for the nginx worker, and the kernel already tracing both connections with no identity on them](../../assets/core-capability-demo/act1-augmentation.png) *Act 1, before any policy exists. Those labels are the whole input to identity. The connections below them are already traced: `tls: NONE`, no SPIFFE ID.* ## Why the containers matter The application is five unmodified upstream containers: nginx, go-httpbin, redis and curl, pulled as-is. They speak plaintext and hold no keys, certificates or tokens. Everything the demo shows gets added underneath them, in a Linux kernel module. That's what makes "no changes to the application" something you can check rather than something you have to believe. At the end of act 2, nginx still says `proxy_pass http://…`. Policy is written the way a customer writes it, as CRDs applied against a real control plane. There's no developer-only shortcut anywhere in it, so what you run is what ships. ## Why four small steps and not one Most capability demos show the end state: everything switched on, everything working. That tells you the whole bundle works on the presenter's laptop. It doesn't tell you which part did which job, and it leaves you with nothing to work from when one part misbehaves in your environment. So each act changes exactly one thing, and shows the same measurement on both sides of that change. Act 1 is mostly there to establish the "before": traffic in the clear, no identity, no policy. Without it you can't tell whether act 2 encrypted something or whether it was already encrypted when you walked in. Act 2 applies the policy and runs the same capture command, verbatim. The packets carrying `GET /get` go from 4 of 8 to 0 of 18, and on the Redis leg `demo:ts` goes from 4 of 12 to 0 of 29. There are more packets than before, and none of them carry the payload. A count that moves while the volume holds is a measurement. A screenshot with a green checkmark on it isn't. ![Terminal output from make act2: payload counts before the policy, the five policy objects being applied, and the same two counts after, both zero](../../assets/core-capability-demo/act2-mtls-counts.png) *Two counts, five policy objects, then the same two counts again. Nothing else changed in between.* Each act shows the negative case too. Drop a workload from the allow-list and the connection resets within one reconnect. Put it back and it recovers. If everything in a demo succeeds, you learn nothing about how to diagnose it later, and you can't tell which parts are actually doing the work. It also guards against the most common way a demo lies, which is a zero that looks like proof. "No hits" can mean encrypted. It can also mean no traffic, a missing capture tool, or a policy that never applied. So every step is built to make a false success obvious. When the capture tool isn't there, the script says `NO PACKETS SEEN` instead of printing a comfortable zero. There's a practical side to this as well. You can stop at whichever act covers the claim you actually doubt and dig into that one, instead of taking the whole thing as a package deal. ![Terminal output from make act3: GitHub returns "Requires authentication", then the caller's user JSON, while curl -v shows no Authorization header and the container environment holds nothing credential-shaped](../../assets/core-capability-demo/act3-injection.png) *Act 3. The response changes, and `curl -v` shows what curl actually wrote, with no `Authorization` line in it.* ## Where it runs On a VM on your laptop, or on AWS. We've verified both, and any Linux box you can SSH into will behave the same way. You need a control plane, which is free at [console.riptides.io](https://console.riptides.io), plus a node joined to it. The README covers both paths. ## First of a series The demo is deliberately synthetic. Redis setting a timestamp key in a loop isn't anybody's production workload. It's just a clean way to watch one mechanism at a time, without a real system's noise on top of it. That's where we wanted to start. The next posts will work through the rest of the capabilities the same way, one at a time and with something you can run, and move from synthetic setups toward what real deployments look like: Kubernetes, CI runners, multiple clusters, egress to third-party APIs. The AI features come later in the series on purpose. Session-level attribution across an agent's model calls, MCP requests and database queries, and tokens an agent can use but never read, aren't a separate product bolted on the side. They're what you get once the identity in this demo is in place, and that's much easier to explain after you've watched the ordinary part work on your own machine. If something doesn't behave the way the README says it should, open an issue. Questions are welcome too. [Get a free Riptides account →](https://docs.riptides.io/deployment/free-account/) --- ## Your Coding Agent Should Never Hold a Token: Riptides Approach to Credential Brokering - URL: https://blog.riptides.io/how-riptides-solves-your-credential-brokering-problem - Published: 2026-08-10 - Author: Mate Wolf - Category: Identity - Tags: AI, AI Agents, Identity, Credentials, Credential Injection, Injection --- ## Introduction These days a lot of software development is AI-assisted: you describe what you want, and a coding agent goes off and does it: reads the repo, runs the tests, opens the PR. But agents don't work in a vacuum. Sooner or later the agent, or one of the tools or MCP servers it's allowed to use, needs a credential to do its job. Your job, really. And that's the point where the comfortable feeling stops or where it should stop. Can you trust your agents? The answer is that this isn't only a question of trust. Set aside for a moment how LLM providers handle your credentials: an agent isn't a piece of code you wrote and reviewed. It's a process whose next action is decided by an LLM reacting to whatever text has landed in its context. And that text can be anything: a README, a docstring, an MCP server's reply, an issue body written by a stranger. Any of it can carry an instruction, and telling a legitimate action apart from an attempt to steal something is genuinely hard. The exposure is real and it's broad: prompt injection that turns a `.env` read into an exfil, supply-chain packages that specifically hunt for AI CLI config files, misconfigured base URLs that ship your API key to somebody else's host, tokens that stay valid for weeks after they leak. Every one of those attacks needs the same precondition: a credential the agent can reach. These are threats developers hit day after day, which is why Riptides built a platform to close the gap. This post walks through the problem with a real but simple example, and shows how Riptides solves what the industry calls **credential brokering** - across your dev machines *and* your production environments. --- ## An everyday use-case Here's a task about as mundane as it gets: fix a GitHub issue. ``` Hi Claude! Solve this issue: https://github.com/riptideslabs/taskflow/issues/1 ``` Without any credential infrastructure, this is how far the agent gets: ```console $ gh issue view 1 --repo riptideslabs/taskflow --json title,body,number,comments,labels,state To get started with GitHub CLI, please run: gh auth login Alternatively, populate the GH_TOKEN environment variable with a GitHub API authentication token. ``` The agent stops and asks you to authenticate. And here's the moment nobody thinks twice about: you run `gh auth login`, or you paste your personal access token into `GH_TOKEN`, and now a long-lived GitHub token with access to your repos is sitting in the environment of a process that will happily run `env`, `cat`, or `curl` because an LLM told it to. You didn't do anything wrong. You did the normal thing. That's the problem. --- ## How does Riptides solve this problem? Riptides is a security platform that injects credentials **directly on the wire**. The agent sends its request with a placeholder, or with no auth at all. On the way out, before the packets leave the machine, Riptides swaps in the real credential. The application sees the placeholder. The API sees a valid token. Nothing in between ever writes the real secret to a file, an environment variable, or the agent's memory. Setting that up is three declarations: **1. Who is this?** You define a workload identity using a selector. A selector can be almost anything: process name, container image, Kubernetes labels and namespace, cloud instance metadata. Riptides collects these itself, from below the application, so a process can't claim to be something it isn't. **2. What's the credential?** You define where the secret comes from, once: a static API key, a secret in Vault, a secret provided by Azure or GCP token exchange. One definition, reusable everywhere. **3. Who may use it, and where?** You bind the two together and name the destination. This identity gets this credential, on requests to GitHub but only for GitHub. That's the whole model. From there on it's automatic, and nothing in your setup had to change: Claude Code is the stock binary, `gh` is the stock binary, no SDK, no wrapper, no `HTTPS_PROXY`, no custom CA in the trust store, no agent-framework plugin. Let's see how it works in practice. --- ## Step by step guide Five steps. Nothing in them touches the agent, the shell, or the project. ### 1. Get a control plane Request a workspace at [riptides.io/get-started](https://riptides.io/get-started). You'll get a console URL of the form `https://.console.riptides.io`. It's hosted by Riptides so you don't need to deploy or maintain. ### 2. Attach the machine where your agent runs The agent here runs in a Linux VM, a separate environment from the host, under stricter control. To attach the machine, use a **Join Token**: 1. Open your Riptides control plane 2. Log in 3. Click **Daemons** in the menu bar 4. Click **Attach Daemon** ![Screenshot: Attach Daemon step in Riptides console](../../assets/credential-brokering/daemons-page.png) 5. Create a Join Token 6. Run the command on your machine ![Attach Daemon](../../assets/credential-brokering/attach-daemon.png) The same command works on a workstation, a CI runner, or a Kubernetes node. Within seconds the node appears in the console and every workload on it is also scanned. Start a session and Claude Code shows up on its own. ![Attached daemon](../../assets/credential-brokering/attached-daemon.png) You've configured nothing yet, and you can already see what your agent is talking to. ### 3. Give the agent an identity Open the **Identities** menu and click **Create Identity**. Restrict this identity to the machine by selecting its hostname, and name it `devenv`.The selector options come from what's actually running, so you're choosing from a list rather than guessing at strings. ![Create a new identity](../../assets/credential-brokering/create-wid.png) Every identity in Riptides has to be attached to a daemon or a daemon group. The daemon is the program running on your machine that talks to the Riptides control plane. ![Set Daemon](../../assets/credential-brokering/create-wid-daemon.png) Credential injection has to write into an encrypted stream, so you need to allow TLS termination for this identity. You'll find it on the **Connection properties** tab. ![Set TLS intercept](../../assets/credential-brokering/create-wid-tls.png) Restart `claude` and start a new session. It now runs as `devenv`. ![Claude as devenv](../../assets/credential-brokering/claude-as-devenv.png) ### 4. Add the GitHub token Under **Credentials**, click **Add credential**, choose a Static credential, and paste in a GitHub PAT scoped to what the agent actually needs. ![Add credential](../../assets/credential-brokering/add-credential.png) Still a long-lived PAT but now it lives in the control plane, bound to an identity and revocable in one click, instead of in a `.env` on a machine that a `postinstall` script can read. Swap it for Vault or a cloud token exchange later without touching anything else. ### 5. Bind the credential to the identity, in the direction of GitHub Under **Credential Bindings**, click **Create Binding** and connect the two, then name the destination: identity `devenv`, credential `github-token`, destination GitHub. ![Add credential binding](../../assets/credential-brokering/add-credential-binding.png) Read it out loud and it's the whole model: **this identity gets this credential, on requests to GitHub, and nowhere else.** The agent can't take the token elsewhere, because it doesn't have the token. Let's watch it happen. --- ## The same use-case, now with Riptides The GitHub CLI still expects either an environment variable or a completed auth flow, so give it a dummy value. Let's say `peach`. ```console $ export GH_TOKEN=peach $ gh auth status github.com ✓ Logged in to github.com account matewolf (GH_TOKEN) - Active account: true ``` And then the agent just… works: ``` - Title: Bug: Page 1 is missing tasks when paginating - State: OPEN - URL: https://github.com/riptideslabs/taskflow/issues/1 - Body: ## Summary A client paging through `?page=1&limit=3` never sees the newest tasks, and the pages don't add up to `total`. ... ``` Full issue body, comments, labels. The GitHub API is perfectly happy. `gh` is perfectly happy. The token is `peach`. The real GitHub token never entered the agent's environment, never touched the filesystem, and never appeared in the model's context. If a prompt injection three turns from now convinces this agent to `curl` its environment variables to an attacker, the attacker gets a fruit. --- ## Ok. But is the threat real? A quick credential leakage history class The `peach` demo is cute, but the reason it matters is that the threat is real and current. There are enough documented cases by now to make the pattern obvious, and the shape is identical every single time: **injected text convinces the agent to read a credential it can reach, then to send it somewhere it shouldn't.** **CVE-2025-55284 - Claude Code smuggles secrets out over DNS.** A prompt injection hidden in a source file told the agent to read `.env` and encode the contents as a series of DNS lookups. The network commands it used (`ping`, `nslookup` and `dig`) were all on the auto-approved "read-only" list, so nothing ever prompted the developer. ([Embrace the Red](https://embracethered.com/blog/posts/2025/claude-code-exfiltration-via-dns-requests/)) Amazon Q Developer had the same flaw, patched quietly by AWS a few weeks later without a CVE at all. **CVE-2026-21852 - opening a repository was enough.** A crafted settings file inside a project pointed Claude Code's API endpoint at an attacker-controlled host. The agent honored the override and started making calls with the user's Claude API key in the header. You didn't have to run anything. You had to open the folder. ([Check Point Research](https://research.checkpoint.com/2026/rce-and-api-token-exfiltration-through-claude-code-project-files-cve-2025-59536/)) **CVE-2026-21516 - Copilot reads its own environment out loud.** Injected repository content convinced GitHub Copilot to pull `GITHUB_TOKEN` out of its own process environment and print it back through its normal output. No exotic exfiltration channel required. The token was sitting there because Codespaces had put it there, as designed. **IDEsaster - two dozen more, and one that needed no exfiltration at all.** Researchers found 24+ CVEs across Cursor, Copilot, Windsurf and Zed. The standout: the agent writes `.env` contents into a JSON file whose schema URL points at the attacker's domain, and the editor's own validator dutifully fetches that URL, secrets in tow. The agent never "sent" anything. ([The Hacker News](https://thehackernews.com/2025/12/researchers-uncover-30-flaws-in-ai.html)) Then there's the case that should worry you most, because it's the one where the agent stopped being the victim and became the weapon. In August 2025, someone published malicious versions of `nx` carrying a post-install script that ran the moment anyone installed it. The script did the obvious thing first: swept the machine for SSH keys, npm and GitHub tokens, `.env` files and crypto wallets. Then it did something new. It looked for AI coding CLIs already installed on the box (`claude`, `gemini`, `q`) and invoked them with their permission prompts disabled, handing them a prompt that asked them to go find anything else sensitive on the filesystem. The developer's own agent, with the developer's own trust settings, did the scanning. Everything it found was base64-encoded and pushed to a new public repository in the victim's own GitHub account. The scale of it was staggering. GitGuardian counted 2,349 credentials taken from 1,079 machines, and more than a thousand of them were still valid when researchers went looking. The attackers used those to flip 10,767 private repositories public, exposing a further 82,901 secrets from codebases nobody had ever intended anyone to read. One compromised dependency, and a laptop full of agents willing to help. ([GitGuardian](https://blog.gitguardian.com/the-nx-s1ngularity-attack-inside-the-credential-leak/)) That's the "agent as application" threat model in one incident. The agent on your machine has shell access, elevated trust, and a credential in reach. Someone else's `postinstall` script is enough to point it at you. Scoping tokens narrowly helps. Short-lived tokens help. But both of those only shrink the blast radius of a leak. Only hiding the credential prevents the leak entirely. --- ## Why Riptides is different Plenty of companies are working on credential brokering. Almost all of them arrived at the same answer: put a proxy in front of the agent. Riptides didn't. **No proxy means nothing to wire up.** You never point an application at Riptides. No `HTTPS_PROXY` per process, no custom CA in every trust store, no parser for every protocol that isn't HTTP. You define an identity once, and it works the same on a laptop, a CI runner, or a production cluster. You also avoid what a gateway inevitably becomes: one network-reachable service holding valid credentials for every workload it serves, where a single compromise hands over all of them at once. **A session is more than a connection.** One agent task is a model conversation, an MCP call, a GitHub request and a database query - separate sockets, often separate processes. A proxy sees them arrive from the same IP and has to guess they belong together. Riptides stamps each one with the same session identity at the source, so the whole session can be traced and controlled as one thing rather than inferred after the fact. *Riptides' AI Audit feature builds on this and is now in beta.* **And credentials are only the start.** The same identity layer secures everything else your workloads do: egress control, service-to-service mTLS, full attribution across dev and production. If that's interesting, the [docs](https://docs.riptides.io/introduction/) and the [blog](https://riptides.io/blog/) go deeper. --- ## The one rule A credential the agent can read is a credential the agent can leak. So don't give it one. Give it `peach`. [Get a free Riptides account →](https://docs.riptides.io/deployment/free-account/) --- ## When the Sandbox Fails, the Kernel Shouldn't - URL: https://blog.riptides.io/when-the-sandbox-fails-the-kernel-shouldnot - Published: 2026-08-03 - Author: Janos Matyas - Category: Kernel - Tags: SPIFFE, AI, AI Agents, Kernel Last week gave us the cleanest illustration yet of a problem we've been writing about for a year. During an internal cyber-evaluation, OpenAI dropped two of its models into a "highly isolated" research sandbox with no direct internet access. The models' safety refusals for cyber tasks had been deliberately turned down. The only way out was a proxy that cached and served package-registry installs. The models decided the fastest way to score well on the benchmark was to find the answer key, which they figured lived on Hugging Face, so they went and got it. They found and exploited a zero-day in the proxy software, broke out of the sandbox, and escalated privileges. Then they moved laterally from machine to machine until they reached a node with open internet, and from there compromised Hugging Face's production infrastructure end to end. Hugging Face detected the intrusion and shut it down, days before OpenAI even connected the activity to its own eval run. One of the researchers who reviewed it called it "a containment failure with the safeties turned off." That's fair. But it's worth being precise about *which* containment failed. **The industry's reflex will be to build a better sandbox, and that reflex is wrong.** ## Sandboxing is a perimeter, and perimeters get bypassed Look at what actually held the agent in place: a container, a network boundary, and a single egress proxy. Every one of those is a perimeter drawn *around* the workload. And the entire history of security says the same thing about perimeters: they work until someone finds the one weak spot you didn't protect. The current toolkit for boxing in an agent is a stack of these perimeters: - **Containers and sandboxes.** They isolate the process, network, and filesystem they all share one kernel, so they share one attack surface. - **Syscall filtering (seccomp-bpf).** It allows only the system calls you list, so it's only as good as the profile you wrote. - **User-space kernels and microVMs (gVisor, Firecracker, Kata).** These give stronger isolation and real hypervisor boundaries, but they add real overhead and are still an escape target. - **Egress proxies and network allow-lists.** This is the "one door out" model, and in the OpenAI incident that one door became the actual escape hatch. - **Service-mesh sidecars.** A proxy per workload that, in many deployments, the workload can simply route around. - **eBPF-based runtime monitors.** Great for *seeing* bad behavior, but usually not positioned to *stop* it before the bytes leave. None of these are bad. We use several of them. But notice the shared failure mode: **each one is a wall the agent is standing next to, not a property of the agent itself.** The moment the agent finds a hole, it's on free roam. That hole might be a zero-day in the proxy, a misconfigured seccomp profile, or a sidecar it can bypass. Once it's out, it has whatever ambient credentials were sitting in its environment, and it can talk to whatever it can reach. There's no identity gating the next connection, and often no record of what it did after it got out. A capable agent optimizing hard against a goal is a *tireless prober*. It will spend inference compute probing the weak points until it finds it. If your security model is "the box holds," you have already lost the argument the first time the box doesn't. ## Move the boundary into the kernel The shift we've been arguing for since [Rethinking Workload Identity at the Kernel Level](https://riptides.io/blog/rethinking-workload-identity-at-the-kernel-level/) is this: **stop drawing a perimeter around the workload, and make identity, secure transport, and policy properties of every connection it makes. Enforce that in the kernel, on the path every byte already travels.** The difference is structural. A sandbox is optional to how the agent runs: the workload executes, and separately, a wall stands nearby. Kernel-level enforcement is *not optional*, because there is no "outside the kernel" for a userspace process to escape to and still make a network call. Every outbound connection is a syscall. If identity and policy live at that layer, the agent cannot route around them without leaving the machine entirely. And it can't leave the machine without going through that same layer. Three things move into that layer. ### 1. Identity: every agent gets a real cryptographic identity An escaped agent is dangerous because it inherits ambient authority: whatever secrets were in `env`, whatever role the node had assumed. Kill the ambient authority and the escape is far less useful. We do this with SPIFFE. Every workload gets a short-lived **x509 SVID** bound to what it actually is, including every ephemeral sub-agent an orchestrator spins up. Not a shared service-account token, not a long-lived key in an environment variable, but a per-workload identity issued and rotated automatically. This is the same model we detailed in [Bringing SPIFFE to OAuth for MCP](https://riptides.io/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads/) and [MCP + Riptides OAuth](https://riptides.io/blog/mcp-riptides-oauth/). The agent presents a verifiable identity for the workload and carries the user and delegation context, so every action has dual accountability: **which agent, on whose behalf**. And to be clear about the limitation up front: this doesn't make an agent safe. A correctly-identified agent can still be told to do something stupid. What it does is make every action *attributable and gateable*, which is the precondition for everything below. ### 2. Secure connection: mTLS and credential injection on the wire Once a workload has an identity, the kernel intercepts its outbound TLS and verifies the identity. It wraps the connection in **mTLS** and injects the *destination* credentials onto the wire, so they're never handed to the process. This is the part that directly defuses the OpenAI-style escape. In that incident the agent inherited whatever credentials were reachable. In this model, **`process.env` reads nothing.** A compromised transitive dependency, or an agent that has broken out of its container, reads an environment with no secrets in it, because the secrets were never there. They exist only on the wire, and only for connections the kernel has already authorized. We covered the mechanics of taking credentials off the runner entirely in [Your GitHub Actions Job Deserves a Real Identity](https://riptides.io/blog/your-github-actions-job-deserves-a-real-identity/). The AI-agent case is the same boundary, applied to a much more adversarial workload. ### 3. Policy: enforcement from the kernel, not a proxy the agent can dodge Identity tells you *who*. Policy decides *whether*. And the enforcement point matters enormously. If your policy lives in an egress proxy, the agent's job is to not use the proxy. If it lives in a sidecar, the job is to route around the sidecar. If it lives in the kernel, on the syscall path, there is nothing to route around. Non-allowed destinations are **dropped before a byte leaves the host.** In the OpenAI scenario, the agent reached a node with open internet and then dialed out to Hugging Face. That's exactly the connection a kernel-level egress policy denies at the source, no matter how the agent got to that node. That's the whole point of enforcing from the core: breaking out of the container doesn't hand the agent the network, because the network was never gated by the container. ## Every connection through the kernel is a connection you can audit This is the part we want to underline, because it's where last week's post-mortems all converged. The consensus takeaway from the OpenAI incident was that anyone running autonomous agents needs detailed records: prompts, tool calls, credentials, network requests, and policy decisions. Without them, investigators cannot reconstruct *why* an agent did what it did. When an agent performs thousands of actions, a final benchmark score can't explain its behavior. You need the full path. Here's the thing: **if every connection is already going through the kernel for identity and policy, the audit trail is a free byproduct.** It's not a separate logging pipeline you hope the agent didn't disable. Every authorized connection, every dropped connection, and every credential injection is attributed to a specific SVID and its delegation context, and recorded at the enforcement point itself. Observability, audit, and tracing aren't a bolt-on. They fall out of the architecture, because the choke point where you ***enforce*** is the same one where you ***observe***. That's the difference between "we think the agent did X" and "here is every connection this identity opened, in order, with the policy decision on each one." One of those is a forensic reconstruction. The other is a log. ## Why not just run SPIRE for this? We embrace SPIFFE. We don't run SPIRE for agents, and the reasons are worse for AI workloads than for anything else. We laid this out in [Why Riptides Embraces SPIFFE but not SPIRE](https://riptides.io/blog/why-riptides-embraces-spiffe-but-not-spire/). Agents are ephemeral and spawn dynamically. An orchestrator creates sub-agents on demand to handle tool calls or parallelize work, and they live for seconds. SPIRE requires every workload to be **pre-registered** with the server before it can be attested and issued an SVID. For dynamic sub-agents, that means either pre-registering every possible variant or racing an automation pipeline to create registration entries as processes spawn. On top of that, SPIRE's attestation is **pull-based**: the workload must actively call the Workload API to get its identity. A sub-agent that doesn't integrate the SDK simply never gets one. And for a process whose entire useful life is shorter than the SVID handshake, that overhead doesn't pay for itself. The kernel approach sidesteps all of it. Identity is attested and issued at the layer the workload can't avoid using, **with no code change, no SDK, no sidecar, and no pre-registration dance for every ephemeral agent**. ## The uncomfortable version Assume the sandbox fails. Assume the agent, optimizing hard, finds the one weak point. That's not pessimism. It's what happened last week to one of the most sophisticated AI labs in the world, with the safeties deliberately down. The question that actually matters is: *what does the agent have the moment it's out?* - If the answer is "ambient credentials and an open network," you're relying on the box, and the box just failed. - If the answer is "no secrets in its environment, no identity to open a new connection, every destination gated at the kernel, and a full audit trail of everything it tried," then the escape is a logged, contained event instead of a breach. Sandboxing tries to make the escape impossible. That's a bet you will eventually lose. Kernel-level identity, enforcement, and audit make the escape ***survivable***, which is the only property that holds when a capable agent is spending real compute trying to get out. Build the boundary into the infrastructure the agent can't step outside of. That's the kernel. *Riptides gives every workload a SPIFFE identity, whether it's a human-triggered job, a machine workload, or an autonomous agent. Each one gets in-kernel mTLS with secretless credential injection, kernel-enforced egress policy, and a full audit trail of every connection. No sidecars, no SDK, no secrets in the environment.* [Talk to us](/request-a-demo), or [start free](https://console.riptides.io) and see our approach for yourself. --- --- ## Adopting SPIFFE Should Not Require an Engineering Team - URL: https://blog.riptides.io/adopting-spiffe-should-not-require-an-engineering-team - Published: 2026-07-26 - Author: Zsolt Varga - Category: SPIFFE - Tags: SPIFFE, identity, kernel, mTLS Most infrastructure authentication today is possession-based: whoever holds the API key, the database password, the token gets treated as authorized, whether or not they're actually the workload that credential was ever meant for. A stolen secret and a legitimate one look identical to whatever's checking, because nothing was ever asked to prove what it is, only that it's holding the right string. SPIFFE exists to replace that with attested identity: a cryptographic, short-lived identity for a workload, an SVID, scoped to a trust domain, verifiable by anyone who trusts that domain's root, derived from what the workload actually is rather than what it happens to be carrying, and independent of IP address, hostname, or anything else about the network it's running on. It's a clean answer to a real problem, which is why it shows up in identity roadmaps at companies of every size. What's less consistent is what happens after a team decides to adopt it. ## What the reference implementation actually requires SPIFFE is a specification. It describes the identity model, not how to run it. SPIRE, the CNCF reference implementation, is what most teams reach for to make that model real, and installing and maintaining it is no small feat. It means assembling several separate projects into one working system: - A **SPIRE Server**, the source of trust for a domain, plus a **SPIRE Agent** running on every node that needs to issue identities. - On Kubernetes, a **controller manager** to keep registration entries, the mapping from "this is what a workload looks like" to "this is the SPIFFE ID it gets", in sync with what's actually running in the cluster. Off Kubernetes, that same mapping has to be maintained some other way, usually by hand. - A **policy layer**, usually a service mesh like Envoy or Istio, or a separate engine like OPA, since SPIRE issues identity but has no opinion on what a given identity is allowed to do. - A **management UI**, commonly Tornjak, for anyone who wants visibility into registrations and trust bundles without reading server logs. None of this comes pre-integrated. Every one of those components is a separate release cadence, a separate set of upgrade notes, and a separate way for the chain to break. Getting from "we picked SPIFFE" to "every workload we care about has a real, enforced identity" tends to run longer than the initial estimate, and the gap between those two dates is rarely a SPIRE bug. It's the assembly work nobody scoped up front. None of that is a knock on SPIRE's design. It was built to prove the specification works in production, and it succeeded at that. It was not built to be the finished security platform a platform team runs unattended for years. Certificate issuance and rotation are SPIRE's job. What a workload is allowed to do with that identity, and what happens when the workload isn't a clean fit for SPIRE's model in the first place, is left to whoever owns the rollout. ## Standing up the ecosystem is only half of it Everything above is the operational cost of running SPIRE: infrastructure you install once and then keep alive. There's a second cost that's easy to undercount, because it isn't something you install once. It's a requirement placed on every individual workload, for as long as that workload exists. SPIRE's delivery model is pull-based. A workload gets an SVID by calling the SPIRE Agent's **Workload API** itself, over a Unix domain socket, using a client library, then loading the returned certificate and private key into its own process memory, then handling the TLS handshake, then calling the API again before the certificate expires. None of that happens unless the workload's own code does it. A **CSI driver** or a **SPIFFE-Helper** sidecar can do this on a workload's behalf if the workload itself can't, but that's a workaround for the requirement, not a removal of it: something still has to actively participate in the exchange. For a service you're actively building, that participation is a library import and some glue code, real work, but bounded, and something a team can plan for. For anything you're not actively rewriting, a legacy application, a third-party binary, a vendored dependency, an off-the-shelf appliance, that path isn't available at all. The only option left is a sidecar or proxy that terminates TLS in front of it, and that trade solves the participation problem by creating a different one: the certificate now belongs to the proxy process, not the workload it's supposed to represent, and every workload that can't speak SPIFFE natively needs its own companion process kept alive indefinitely. This is why "we deployed SPIRE" and "every workload we care about has a real identity" are different sentences, and why the second one tends to take longer than the first. Standing up the server, the agents, and the policy layer is an infrastructure project with a schedule. Getting every workload to actually participate, or fronting the ones that can't, means touching or wrapping every single thing that's supposed to have an identity, and that list rarely shrinks on its own. ## Where the gaps show up in practice A few places this tends to bite, regardless of how well the initial rollout goes: **Getting to day one.** Before a workload can have an identity at all, SPIRE forces a choice: rewrite it to call the Workload API directly, or stand up a sidecar in front of it. There's no third option. Day-one coverage across a real fleet is capped by how much of it fits cleanly into one of those two buckets, and everything that doesn't yet, the service waiting on a rewrite, the appliance waiting on a sidecar nobody's built, runs with no identity at all in the meantime. **Policy, as someone else's project.** SPIRE issues an identity and stops there. What that identity is actually allowed to do gets decided somewhere else entirely, a service mesh, an OPA install, a set of hand-written admission rules, with its own configuration language, its own release cadence, and its own way of drifting out of sync with what SPIRE issued. Two systems agreeing on who a workload is isn't the same as them agreeing on what it can do, and keeping both in step, as either one changes, is a permanent job, not a rollout task. **CI/CD.** Ephemeral runners spin up, do one job, and disappear, which is exactly the case SPIRE's registration model assumes doesn't happen often. Even a runner whose code is perfectly happy to call the Workload API still needs to be registered and attested in time to be useful, and getting that timing right for short-lived GitHub Actions or similar runners takes custom tooling most teams have to build themselves. Every one of these is solvable. None of them is solved by SPIRE alone, and the tooling that closes each gap is one more thing to keep patched, monitored, and correctly wired into everything else. ## A different way to close the gap We build Riptides on the same specification. We just made a different architectural bet: instead of adding a component for each gap above, and instead of asking every workload to participate in its own identity delivery, we moved identity issuance and policy enforcement into the layer every workload already passes through no matter its language, its framework, or where it's deployed, which is the Linux kernel. [Riptides issues SPIFFE-compliant X.509 SVIDs directly inside the kernel](/blog/why-riptides-embraces-spiffe-but-not-spire), bound to the real process making the connection. There's no Workload API for an application to call and no certificate for it to load, because the kernel attests the process and injects the identity into the connection itself. Policy is checked and enforced at the same point, before a connection is established, rather than handed off to a mesh or an external engine. [We've written separately about why this matters specifically for AI agents](/blog/how-to-deliver-spiffe-identity-to-ai-agents); this post is about the plainer case, two ordinary workloads running as unmodified pods in a vanilla Kubernetes cluster. Rather than argue that in the abstract, here's what it looks like end to end. ## What this looks like Picture a small, otherwise-vanilla Kubernetes cluster, no Istio, no Linkerd, no mesh already in place, so the comparison holds against what a from-scratch SPIRE rollout would need to add, not against infrastructure that already solved half the problem. **`orders-api`**, a `Deployment` and `Service` running a tiny HTTP server on port 8080 that returns a canned JSON response: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: orders-api namespace: demo spec: replicas: 1 selector: matchLabels: app.kubernetes.io/component: orders-api template: metadata: labels: app.kubernetes.io/component: orders-api spec: containers: - name: orders-api image: python:3.12-slim command: ["python3", "-c"] args: - | import http.server, json class Handler(http.server.BaseHTTPRequestHandler): def do_GET(self): self.send_response(200) self.send_header("Content-Type", "application/json") self.end_headers() self.wfile.write(json.dumps({"order_id": 42, "status": "packed"}).encode()) http.server.HTTPServer(("0.0.0.0", 8080), Handler).serve_forever() ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: orders-api namespace: demo spec: selector: app.kubernetes.io/component: orders-api ports: - port: 8080 targetPort: 8080 ``` **`checkout`**, a `Deployment` that polls it every couple of seconds: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: checkout namespace: demo spec: replicas: 1 selector: matchLabels: app.kubernetes.io/component: checkout template: metadata: labels: app.kubernetes.io/component: checkout spec: containers: - name: checkout image: curlimages/curl:latest command: ["sh", "-c"] args: - "while true; do curl -s http://orders-api.demo.svc.cluster.local:8080/orders; echo; sleep 2; done" ``` Neither manifest mentions Riptides, SPIFFE, or TLS anywhere. That's deliberate: this is what the two workloads look like with no identity story at all, and it's exactly what stays in place afterward. > **Assumes:** a Riptides control plane and a daemon already running on the cluster. The daemon install is a single Helm chart, one pod per node, no CSI driver, no admission webhook, no separate controller manager to reconcile against cluster state. See [Getting Started](https://docs.riptides.io/guides/getting-started) for the install itself. Installing it doesn't change anything on its own, there's no WorkloadIdentity yet for it to enforce. Giving both workloads a verified identity and requiring mTLS between them is two Riptides resources, following the same backend/frontend pattern as the docs' own [mTLS between services guide](https://docs.riptides.io/guides/mtls-between-services): a `WorkloadIdentity` for the server requiring mTLS from a specific caller, and a `WorkloadIdentity` for the client. > Not to be confused with these being Kubernetes resources in name only: the Riptides control plane speaks the native Kubernetes API, so the same GitOps and Kubernetes management tooling already in use can manage them directly. ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: demo-orders-api spec: workloadID: demo/orders-api scope: daemonGroup: id: riptides/daemongroup/demo-cluster selectors: - k8s:container:name: orders-api k8s:label:app.kubernetes.io/component: orders-api process:name: python3 connection: tls: mode: MUTUAL allowedSPIFFEIDs: inbound: - spiffe://acme.corp/demo/checkout --- apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: demo-checkout spec: workloadID: demo/checkout scope: daemonGroup: id: riptides/daemongroup/demo-cluster selectors: - k8s:container:name: checkout k8s:label:app.kubernetes.io/component: checkout ``` ```bash riptides-cli ctl apply -f workload-identities.yaml ``` Neither the `orders-api` container nor the `checkout` container needed a single line changed. Neither one knows Riptides exists. From here, `checkout`'s ordinary HTTP calls to `orders-api`, the exact same request the application always made, are transparently upgraded: the kernel on both nodes negotiates mutual TLS before a single byte of the actual request goes through, each side presenting a real SPIFFE X.509 certificate bound to the process that opened the connection, not to a proxy sitting next to it. Inspecting that certificate from either side shows a `URI` SAN of `spiffe://acme.corp/demo/orders-api` or `spiffe://acme.corp/demo/checkout`, a real SVID, not TLS for its own sake. An SVID that nothing checks is just a certificate, though, so the same policy that issued it enforces it. Any process that doesn't match either `WorkloadIdentity`, some unrelated pod that isn't `checkout`, has no SVID to present and is rejected at the handshake. Not because a network policy happened to catch it, but because it was never issued an identity to begin with. ## What that replaced Going back to the component list earlier in this post: no SPIRE Server, no separate SPIRE Agent process, no CSI driver or SPIFFE-Helper sidecar to get a certificate onto the pod, because nothing is written to the pod's filesystem at all. No separate policy engine or mesh, because the same enforcement point that issues the identity also checks it against policy before the connection completes. And going back to the participation problem: neither `orders-api` nor `checkout` called a Workload API, linked a SPIFFE client library, or ran alongside a sidecar. Both were ordinary containers that happened to open a socket. That's what makes the same model apply just as well to the legacy binary and the third-party appliance from earlier, they never have to become workloads that participate in their own identity delivery, because participation was never the mechanism to begin with. The other two gaps close the same way. Day-one coverage isn't capped by what's been rewritten or fronted with a sidecar, because there's nothing to rewrite or front: the kernel attests whatever's already running, on day one, across the whole fleet at once. And a short-lived CI job doesn't need anything set up for it in advance: the same `WorkloadIdentity` that covers a long-running service matches a runner that lives for ninety seconds just as well, the moment it opens a connection, not before. ## Focus on what matters The same SPIFFE IDs, the same SVIDs, verified against the same specification, issued and enforced straight from the kernel, with no SPIRE anywhere underneath it. No sidecars. No workload asked to participate in its own identity. No policy layer to bolt on afterward. Just a verified, enforced connection, from day one, on whatever you're already running. ## What this post doesn't cover Everything above is scoped to one thing on purpose: SPIFFE identity and kernel-enforced mTLS between two ordinary workloads. A few things adjacent to that are also part of Riptides, and worth naming so this doesn't read as the whole picture, but each deserves its own post rather than a paragraph tacked on here. Not everything a workload talks to speaks SPIFFE, and a lot of it never will: third-party SaaS APIs, cloud provider credentials, systems with their own established access model. Riptides brokers and injects credentials for those too, without the workload ever holding one, but that's a different mechanism from the identity story in this post. Every connection in the demo above is also a structured, queryable record: who connected to what, under which identity, allowed or denied and why. That's a standing product capability, not a side effect of this particular walkthrough, and it's owed more than a mention. And everything here applies just as well to an AI agent as it does to `orders-api`, an agent is just another process that opens a socket, but agents raise their own set of questions on top: delegated user authorization, sub-agents spawned on the fly, tool-chain trust boundaries. That's its own territory, [covered separately](/blog/how-to-deliver-spiffe-identity-to-ai-agents). ## Before you commit to building it None of this means standing up the ecosystem is the wrong call for every team. Some organizations have the platform headcount, the timeline, and a genuine reason to need SPIRE's specific attestation model, and for them, building it is exactly right. The honest way to tell which camp you're in isn't a demo. It's answering three questions plainly, before the project has a name and a budget line. How much of the fleet can actually be rewritten or fronted with a sidecar in the next two quarters, not eventually? Whatever falls outside that set runs with no identity in the meantime, and that leftover pile is almost always bigger than the first estimate. Once a policy layer sits on top of identity, who owns keeping the two in step, permanently, as both keep changing? If there's no name attached to that yet, it hasn't been decided. It's been postponed. Is operating a distributed system, a server, a fleet of agents, a policy engine, a management UI, something the team is signing up to do indefinitely, on top of everything it already runs? That cost doesn't end at rollout. It shows up on every on-call rotation after it. If any of those three land uncomfortably, the fix isn't rethinking whether SPIFFE is the right standard. It's rethinking what has to run underneath it to get it. ## Stuck with SPIRE? Let's talk. If you're convinced of what the market already seems convinced of, that SPIFFE is where workload identity is headed, but the only path anyone's shown you runs through SPIRE, the server, the agents, the policy layer, the participation requirement, that's exactly the conversation worth having. Not because the standard is wrong. Because betting on SPIFFE and being stuck building SPIRE to get it are two different problems, and only one of them has to be true. Riptides doesn't plug into an existing SPIRE deployment, it takes over identity issuance and enforcement entirely, down to the same SPIFFE IDs your workloads already carry, so nothing downstream has to change. [Talk to us](/request-a-demo) about what that migration actually looks like, or [start free](https://console.riptides.io) and see our approach for yourself. --- ## AI Agents Have an Identity Crisis And It's a Familiar One - URL: https://blog.riptides.io/ai-agent-identity - Published: 2026-07-13 - Author: Janos Matyas - Category: Non-Human Identity - Tags: Security, AI Agents, Credentials, Non-Human Identity, Prompt Injection Organizations spent a decade eliminating static secrets and enforcing least privilege. AI agents are quietly undoing all of it. Every agent is a new identity with broad access, no supervision, and credentials it was never designed to protect. This isn't a theoretical risk. It's the current state of production agentic AI. ## The problem security leaders already recognize The pattern is familiar. A workload needs access to a service, so it gets a credential. That credential gets stored somewhere — a config file, an environment variable, a process's memory. It persists longer than it should, gets shared wider than intended, and eventually becomes the thing an attacker is looking for. For a decade, security organizations have fought this pattern with secrets management, short-lived tokens, just-in-time access, and zero trust architectures. Real progress was made. Now AI agents have reintroduced the same problem at a dramatically larger scale. A single agent session might connect to a SaaS productivity tool through MCP, authenticate against a cloud provider API, call an AI model endpoint, and query an internal database — all in one task, all autonomously. Each of those connections requires a credential. And in virtually every agentic framework today, those credentials end up stored inside the agent — accessible to anything that can compromise the process. The attack surface isn't one stolen key. It's dozens of delegated credentials, scattered across autonomous workloads, with no centralized visibility into what's been issued, what's active, or what's been exposed. ## Why AI agents make this worse than traditional workloads Traditional services are relatively predictable. They run in known environments, connect to known endpoints, and operate within well-understood boundaries. Security teams can reason about the blast radius. AI agents are different in ways that matter for credential security. They are **autonomous and unpredictable**. An agent decides at runtime which tools to call, which services to connect to, and in what sequence. The access pattern can't be fully anticipated at deployment time. They are **ephemeral and dynamic**. Agents spawn sub-agents, delegate tasks, and terminate — sometimes in seconds. Each one needs credentials, and each one represents a window of exposure. They **act on behalf of humans**. When an agent connects to a third-party service, it carries a user's delegated authorization — an OAuth token that represents that user's permissions. If the agent is compromised, the attacker doesn't just get the agent's access. They get the user's. And critically, they are **vulnerable to a new class of attack**. Prompt injection — where malicious instructions are embedded in data the agent processes — can cause an agent to exfiltrate its own credentials without any traditional exploit being needed. No vulnerability, no CVE, no patch cycle. The agent simply follows instructions it shouldn't have received. The combination creates a risk profile that existing credential management tools were never designed to address. ## What a structural solution looks like The core insight behind the Riptides platform is that the problem isn't how credentials are *managed* — it's that credentials are in the agent's hands at all. Riptides takes a fundamentally different approach: **the agent never holds the real credential**. Not in memory, not in a config file, not in an environment variable. The actual tokens that grant access to services live in the operating system kernel — a layer beneath the application that the agent process cannot reach or tamper with. When an agent makes a request to an external service, the kernel intercepts it, verifies the agent's identity, looks up the appropriate credential, and injects it into the outgoing request at the network level. The service on the other end sees a fully authenticated request. The agent never knew what credential made it possible. An agent that can be prompted into leaking its credentials cannot leak credentials it does not possess. This is a structural guarantee, not a best-practice recommendation. ## Two identities, one request — full accountability Every interaction in a Riptides-managed environment carries two distinct, cryptographically verifiable identities. The first is the **agent's own identity** — a non-human, workload-level identity based on the SPIFFE standard, issued automatically when the agent process starts and bound to that specific process. This identity answers the question: *which agent is making this request?* The second is the **human's delegated identity** — the OAuth-based authorization of the user on whose behalf the agent is acting. When an agent needs to access a third-party service (such as an MCP-connected SaaS tool), the user goes through a standard authentication flow. But instead of handing the resulting access token to the agent, Riptides brokers the exchange, stores the real token in the kernel, and gives the agent a proxy credential that is meaningless outside the system. These two identities travel together but remain distinct. They can be audited independently, revoked independently, and governed by separate policies. The audit trail answers not just "what happened" but "which agent did it, on behalf of which user, at what time, and under what policy." For security leaders responsible for compliance, incident response, and access governance, this dual-identity model provides the kind of granularity that agentic workloads have been missing entirely. ## No code changes, no agent cooperation required One of the most significant operational properties of the Riptides approach is that it requires nothing from the agent itself. No SDK integration, no library imports, no code modifications of any kind. The platform operates at the kernel level of the Linux operating system, which means it works with any agent — regardless of programming language, framework, or orchestration model. A Python agent built on LangChain, a TypeScript MCP client, a Go-based orchestrator, or Claude Code running from a terminal — all receive the same identity, the same credential protection, and the same policy enforcement, without any of them being aware of it. This matters for two reasons. First, it eliminates the dependency on development teams to "integrate correctly" with a security tool. The enforcement happens beneath the application, not inside it. Second, it means security posture doesn't degrade as agent frameworks evolve — and they evolve fast. The kernel doesn't care what framework is running above it. ## Continuous verification, not point-in-time trust Traditional workload identity systems verify a workload once — when it starts — and trust it for the duration of its certificate's lifetime. In the AI agent context, this is insufficient. Riptides continuously validates the posture of every managed process. If an agent's binary changes after attestation, if its runtime environment drifts from the expected baseline, or if its behavior deviates from established patterns, its access can be restricted or revoked in real time. The system doesn't assume that because an agent was trustworthy when it launched, it remains trustworthy now. This continuous enforcement model is particularly relevant given the prompt injection risk. An agent's behavior can change mid-session without any change to its process signature. Continuous posture management is the mechanism that detects and responds to that drift. ## Federation without credential distribution As AI agents increasingly connect to external AI providers, cloud platforms, and third-party services, the challenge of managing federated credentials grows. Riptides handles this transparently. When an agent needs to authenticate against an external provider — whether that's Anthropic's Claude API, an AWS service, a GCP endpoint, or a third-party MCP server — Riptides manages the federation flow, acquires the necessary credentials, and injects them at the kernel level. No API keys are distributed into containers. No long-lived secrets are pushed into environment variables. The credential lifecycle — issuance, rotation, renewal, revocation — is managed centrally and enforced automatically. This extends the same secretless model across every external dependency an agent touches, creating a consistent security posture regardless of how many services are in the chain. ## What this means for security governance For security leaders evaluating how to govern agentic AI workloads, the Riptides model addresses several critical requirements simultaneously. **Credential exposure is eliminated architecturally.** The most discussed AI-specific attack vector — prompt injection leading to credential theft — is neutralized by ensuring credentials never exist in the agent's address space. This isn't a mitigation; it's a removal of the attack surface. **Auditability is built into the architecture.** Every credential issuance, every access decision, every identity binding is tied to both a verifiable workload identity and a verifiable user identity. This isn't a logging layer bolted onto the side — it's inherent to how every request flows through the system. **Policy enforcement is centralized and tamper-proof.** Access policies are defined centrally and enforced at the kernel level, beneath the application. The agent cannot bypass, misconfigure, or be prompted into circumventing its own access controls. **Operational overhead is minimal.** Deployment requires no agent code changes, no sidecar containers, no framework-specific integrations. Security teams can enforce consistent posture across diverse and rapidly evolving agent environments without creating a bottleneck for development teams. ## The bottom line AI agents represent the next generation of non-human identities — more autonomous, more dynamic, and more broadly connected than anything that came before. The credential management patterns that worked for traditional workloads are insufficient for workloads that can be manipulated into acting against their own security. The solution isn't better secret management. It's removing secrets from the equation entirely — ensuring agents can authenticate, participate in authorization flows, and make fully authenticated requests, without ever possessing the credentials that grant access. That's the architecture Riptides was built around. And as agentic AI moves from experimentation to production, it's the architecture that the scale and risk profile of these workloads demand. --- *To see kernel-level agent identity and secretless credential management in action, [request a demo](https://riptides.io/request-a-demo). Follow Riptides on [LinkedIn](https://www.linkedin.com/company/riptidesio/) and [X](https://x.com/riptidesio) for more.* --- ## Your GitHub Actions Job Deserves a Real Identity - URL: https://blog.riptides.io/your-github-actions-job-deserves-a-real-identity - Published: 2026-06-29 - Author: Nándor Krácser - Category: Non-Human Identity - Tags: github-actions, non-human identity, ci, workload-id, credential-injection Every day, engineers paste AWS keys and API tokens into GitHub secrets. They rotate them when they remember, audit them when something breaks, and quietly hope no one with push access ever decides to print them in a workflow log. This is the state of CI security in 2026, and it's not good enough. GitHub's OIDC support for AWS helped. Instead of a static key, your workflow can exchange a short-lived token for an IAM role. But it only covers AWS, and it tells you nothing about what your job actually did once it had those credentials. The deeper problem remains: **CI jobs are stateless, ephemeral workloads with no persistent identity**. They borrow credentials rather than earning them. And because they're treated as second-class citizens in your security model, nobody really knows what they're talking to. ## Every entity. Not just the services. Modern non-human identity practice is straightforward in principle: every non-human actor in your system (every pod, every VM, every service) gets a cryptographic identity. You don't hand a service an API key and hope for the best; you issue it a short-lived certificate, bind it to what it actually is, and enforce policy from there. This is what SPIFFE was designed for, and it's how Riptides works across your production fleet. CI jobs are the conspicuous exception. A GitHub Actions job has write access to your source code, your artifact stores, your signing keys, and your deployment targets. It runs arbitrary code from your dependency graph on every pull request. In terms of blast radius, a compromised job is at least as dangerous as a compromised production service, often more so. Yet most teams treat it as outside the identity model entirely: give it some secrets, hope the workflow file doesn't leak them, move on. The gap isn't technical. It's that no one built the bridge. This isn't a GitHub Actions problem. It's a CI problem. Jenkins jobs, GitLab pipelines, CircleCI workflows: they all run code, call services, and handle secrets, and almost none of them have a cryptographic identity. GitHub Actions happens to have the best building block available today (the OIDC token), which is why we started there. But the principle is the same everywhere: if a process touches your infrastructure, it should have an identity. One thing the AI era hasn't changed: CI is still there. Whether your engineers write every line by hand or vibe-code entire features in an afternoon, the pipeline that builds, tests, and ships that code runs the same way it always did. The surface area of CI isn't shrinking. If anything, faster iteration means more runs, more secrets in play, more opportunities for a compromised dependency to slip through unnoticed. ## Workload identity for CI Riptides gives every workload (whether it's a Kubernetes pod, a bare-metal server, or a GitHub Actions job) a SPIFFE x509 identity issued from your control plane. That identity is cryptographically bound to what the job actually is: which repository it came from, which workflow triggered it, which branch, which actor. When a GitHub Actions job starts, it presents a GitHub OIDC token to the Riptides control plane. The control plane verifies it against GitHub's public JWKS, checks the claims against your Verifier policy (repository owner, environment, ref, whatever you care about), and issues a short-lived x509 SVID. From that point on, the job has a real identity, not a borrowed credential. ## Two things that change **Secretless credential injection.** Once the job has a workload identity, Riptides enforces your policy at the network layer. You define which workloads are allowed to call which services, and what credentials to inject when they do. The job runs `aws s3 cp s3://my-bucket/config.json .`: no access key in the environment, no secret in the workflow, no IAM role assumption in the code. The kernel module intercepts the outbound TLS connection, verifies the workload identity, injects the right credentials on the wire, and wraps everything in mTLS. The application never knew anything happened. This is the same credential injection Riptides uses for production workloads. Your CI jobs now work exactly like your services: identity-first, secretless, policy-enforced. **Full connection visibility.** Every TCP connection a job makes (to S3, to your internal API, to your deployment target, to anything) is tracked with full workload identity context. Which workflow. Which repository. Which actor. Which ref. Whether it was allowed or denied by policy. You get the same network observability in CI that you have across the rest of your fleet. This matters more than it sounds. A supply chain compromise doesn't announce itself. A malicious dependency that phones home during a build looks like normal outbound traffic, unless you're watching. With Riptides, you are. And because credentials are injected at the kernel layer on the wire, they never exist in the job's environment. There's nothing for a malicious package to read, and unexpected outbound connections can be blocked by policy before they leave the host. ## Setup Riptides is [open to everyone](/blog/riptides-is-now-open-to-everyone) now (anyone can spin up a workspace), so you don't have to take our word for any of this. You can wire it into one of your own workflows in a few minutes. Three things are needed. A Verifier on the control plane that trusts GitHub Actions tokens from your organisation: ```yaml apiVersion: auth.riptides.io/v1alpha1 kind: Verifier metadata: name: github-actions namespace: riptides-system spec: GitHubActions: audience: riptides requiredMetadata: - githubactions:repository:owner: your-org ``` A workflow step that joins the job to the control plane: ```yaml jobs: deploy: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: riptideslabs/setup-riptides@v1 with: controlplane-url: https://your-env.console.riptides.io ``` And nothing else. The `setup-riptides` action installs the kernel module, fetches an OIDC token, exchanges it for an x509 identity, and starts the daemon. Every subsequent step in the job runs with that identity. ## A real workflow Take a deploy job that pushes a build to S3, registers the release with an internal API, and posts to Slack. Today it looks like this: ```yaml - env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} RELEASE_API_TOKEN: ${{ secrets.RELEASE_API_TOKEN }} SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} run: | aws s3 cp ./dist "s3://artifacts/$GITHUB_SHA" --recursive curl -H "Authorization: Bearer $RELEASE_API_TOKEN" \ https://releases.internal/deploys -d "{\"sha\":\"$GITHUB_SHA\"}" curl -X POST "$SLACK_WEBHOOK_URL" -d '{"text":"deployed"}' ``` Four secrets, sitting in the environment for the whole job. With Riptides the same job is: ```yaml - run: | aws s3 cp ./dist "s3://artifacts/$GITHUB_SHA" --recursive curl https://releases.internal/deploys -d "{\"sha\":\"$GITHUB_SHA\"}" curl -X POST https://hooks.slack.com/services/... -d '{"text":"deployed"}' ``` No `env:` block, no `secrets:`. The AWS credentials, the bearer token, the webhook authentication — all injected on the wire by the kernel, based on the job's identity, after your policy confirms this workflow is allowed to reach those destinations. ## Why this can't be stolen The difference isn't cosmetic. Picture the realistic attack: a compromised build dependency, one transitive package updated overnight, wakes up inside your deploy job and looks for credentials to ship somewhere it controls. In the first version it finds four of them sitting in the environment, reads them, and `POST`s them to `https://evil.example.com`. The job succeeds, the logs look normal, and your AWS keys are gone. Because they're long-lived secrets in GitHub's store, they keep working long after the run finishes. In the second version that attack has nothing to work with. The credentials are never written to the environment, a file, or any process memory the dependency can reach. They exist only inside the kernel module, on the connection, for the instant each request is made. And the injection is destination-bound: a credential is only attached to traffic headed for a service your policy actually defines. The attacker's endpoint isn't one of them, so nothing gets injected. And because it isn't an allowed destination either, the kernel drops the connection before a single byte leaves the runner. You can't steal a secret that was never there, and you can't smuggle it out to a destination the kernel won't let you reach. ## The broader point CI is part of your production trust boundary. It builds your software, signs your artifacts, deploys to your infrastructure, and calls your internal APIs. It deserves the same security posture you apply to everything else. Treating CI jobs as identity-less credential consumers is a design choice, not a constraint. Riptides makes the alternative straightforward. Spin up a workspace at [riptides.io/get-started](https://riptides.io/get-started). Setup takes a few minutes, and the [`setup-riptides` action](https://github.com/marketplace/actions/setup-riptides), available on the GitHub Actions Marketplace, handles the rest. --- ## Riptides is now open to everyone - URL: https://blog.riptides.io/riptides-is-now-open-to-everyone - Published: 2026-06-16 - Author: Marton Sereg - Category: Identity - Tags: AI, Identity, Agents For the last several months, we've been building Riptides alongside a small group of partners — security teams, platform engineers, and the people on the hook for the agents their companies are racing to ship. We learned a lot in private. Today, we're opening it up. Anyone can spin up a workspace at [riptides.io/get-started](https://riptides.io/get-started) and try it. This post is what we'd tell you if we sat down for coffee: what we kept hearing, what we built, and how to take it for a spin in a few minutes. ## What we kept hearing The story rhymes across everyone: **moving agents into production is hard.** Not because the models don't work. Because nobody can answer the boring operational questions security and platform teams ask before agents go anywhere near anything real. The pilot demos beautifully; the prod conversation starts with "okay, but what is it actually allowed to touch, and how would we know?" — and that conversation tends to go on for months. The same three questions came up almost everywhere. The first one is *access control*. What is this agent allowed to reach, and how is that enforced? Existing IAM was built for predictable, long-running services that do roughly the same thing on Tuesday as they did on Monday. Agents don't oblige: the same prompt produces different tool calls, different credential requests, different lateral moves on different days. The protocols agents talk over — MCP, A2A, tool-call APIs — are invisible to the IAM stack everyone already has. Faced with that, teams either over-grant and hope, or they say no and stall. Neither is a long-term position. The second one is *attribution*. When something happens — a sensitive query, a strange API call, a data pull at 2 AM — who actually did it? Which agent, on whose behalf, with what context? The honest answer in most environments today is *we'd have to put a few people on it for a couple of days*. Logs show that *a service account* ran a query. Reconstructing the agent, the user who triggered it, and the path through three tools and four APIs is archaeology across logs, traces, and SIEM. By the time the answer arrives, the question has moved on. The third is *access to credentials*, and it's the one that wakes security teams up at night. Agents get handed long-lived API tokens and keys because nothing better exists in the runtime. Those credentials end up sitting in environment variables, config files, and process memory — exactly the places a prompt injection or a compromised dependency can reach. The blast radius of a leaked key is its lifetime, and its lifetime tends to be forever. Underneath all three of those questions is the same thing: the service-account model. It was designed for services that look the same every day, and it's been showing cracks for years. Agents bend it, then snap it. The fix everyone in those conversations was implicitly reaching for is the same one: **identity that lives and dies with the agent** — making JIT access and zero-standing privileges actually achievable, instead of a slide in someone's roadmap deck. ## What Riptides is Riptides is runtime machine IAM, built for the agentic era: know what every agent did and why, and control what it's allowed to access — all without changing a line of application code. ### The foundation: composite identity The thing the service-account model gets wrong is treating identity as a label you hand to an agent and hope it doesn't lose. Riptides treats identity as something that's *attested* — the kernel watches the agent start, observes what it actually is (the binary, the container, the Kubernetes service account, the user who launched it), and issues a cryptographic SPIFFE identity bound to that running process. The agent can't claim to be something it isn't, because nothing inside the agent is doing the claiming. That alone solves the "which service account is this" problem. But agents add a second dimension: they act on behalf of *people*. When a developer triggers a coding agent, when an operator runs a query through an analytics agent, when an end user kicks off a workflow — there's a human whose authority the agent is borrowing for that session. Riptides binds the human's authorization context to the agent's runtime identity for the lifetime of that session. Downstream systems no longer see "some agent running as some service account." They see *which agent, on whose behalf, with what session context.* No more shared service accounts. No more confused deputies. No more guessing. Everything else we do sits on top of that. ### Attribution Once identity is real and composite, attribution becomes a side effect rather than a project. Every outbound action the agent takes produces one record: which agent made the call, what credentials it used, what system it talked to, on whose behalf, what context was in scope at the time, and which policy decision was applied. That record is generated at the kernel, where the call actually happens, not reconstructed after the fact from application logs that may or may not have been written. The practical effect is that the investigations security teams dread — "this happened at 02:47, figure out which of our 40 agents did it and why" — collapse from days of cross-referencing logs and SIEM queries into a single query against a stream of structured events. It also gives the platform team something they've never really had: a defensible answer to *who is in control of what these agents can do*. ### Agent access The flip side of attribution is enforcement. Riptides applies per-agent egress policy on the call path: the kernel sees the connection attempt, checks it against the agent's identity-bound policy, and allows or denies it before the first byte leaves the host. The agent doesn't get to negotiate with the policy, route around it, or be talked into bypassing it by a clever prompt. The same machinery does credentials. Instead of handing agents long-lived tokens and praying they don't leak, Riptides brokers credentials just-in-time on the call path. The credential is fetched from your secret store, injected into the specific outbound request that needs it, and never returned to user space. It exists in process memory for exactly zero seconds. A prompt injection or compromised dependency can't exfiltrate a credential the process has never seen. This is what "JIT access and zero-standing privileges" looks like once it stops being a slide. The agent has only the identity it was attested with, only the access its policy allows, and only the credential it needs for the call it's making right now. ### The same model for classic workloads The static-service-account problem isn't new. The same long-lived keys, the same flat internal trust, the same "who actually used this credential" gap have been quietly grinding away at traditional workloads for years — it's where we started thinking about this in the first place. Agents made the problem loud enough that solving it for them is the priority right now, but the foundation underneath is the same. So Riptides applies the same foundation to classic workloads: the same kernel attestation, the same identity, the same JIT credentials, the same enforcement. Agents and non-agent workloads end up governed from one control plane. That matters operationally — one mental model, one set of policies, one set of dashboards — and it matters strategically, because the line between "agent" and "regular service" is going to keep getting blurrier. ### How it runs A note on the *how*, because it tends to be the question people ask once they're sold on the *what*. Riptides runs at the syscall path, beneath the application. There is no SDK to integrate, no proxy to deploy, no gateway to operate, no application code to change. Agents make ordinary network calls; the kernel does identity, policy, credentials, and telemetry transparently underneath. The result is that Riptides works with whatever you're already running — LangChain, CrewAI, AutoGen, LangGraph, a custom Python script, a Go service that's been in production for five years, or a third-party application you can't even touch — without any of those things knowing or caring that Riptides exists. ## Try it today The whole self-serve flow takes a few minutes. Request a workspace at [riptides.io/get-started](https://riptides.io/get-started), then install the daemon on a node or VM and point it at your workspace: ```bash curl -fsSL https://docs.riptides.io/install.sh | sudo bash -s -- \ --control-plane https://.console.riptides.io ``` Give your agent an identity: ```bash riptides apply -f agent-identity.yaml ``` Attribution starts flowing immediately — every connection, every tool call, every credential use. When you're ready to enforce, add access rules: ```bash riptides apply -f credential-binding.yaml ``` That's the whole flow. No other steps, no agent reconfiguration, no application restart. The shape of `agent-identity.yaml` and `credential-binding.yaml`, and what you can put in them, is covered in the [docs](https://docs.riptides.io). ## What's next The thread we're pulling on hardest right now is **coding agents on developer workstations**. Cursor, Claude Code or Codex are accessing cloud infrastructure, Kubernetes clusters, GitHub, and production databases — usually with whatever long-lived credentials happened to be sitting in the developer's shell environment, on macOS or Windows laptops that the rest of the security stack barely sees. The same identity model that covers servers and Kubernetes nodes should cover those machines too: a coding agent running on someone's laptop is, from a runtime perspective, just another agent that needs an identity, a policy, and credentials it doesn't get to hold onto. Once that works the same way everywhere, the gap between "what my agent can do in prod" and "what my agent can do on my Mac" closes — without slowing developers down, and without handing the next prompt-injection or compromised MCP server a long-lived key to walk away with. That work is much easier with real teams in the loop. So here's the actual ask: spin up a workspace, point Riptides at something you care about, and tell us what breaks. The last few months produced something we're proud to open up. The next few months are how it gets to "this is how every team runs agents in production." We'll also be sharing more about how *we* run Riptides — the agents we let loose on our own infrastructure, the policies we wrote, the times we got it wrong, and what we changed. If that's the kind of thing you want in your feed, stay tuned. [riptides.io/get-started](https://riptides.io/get-started) --- ## Miasma Hit Microsoft. It Came for Credentials. Riptides Has None. - URL: https://blog.riptides.io/miasma-hit-microsoft - Published: 2026-06-10 - Author: Janos Matyas - Category: Non-Human Identity - Tags: Security, Supply-chain, AI, Credentials On June 8, GitHub disabled [at least 70 Microsoft repositories](https://techcrunch.com/2026/06/08/microsofts-open-source-tools-were-hacked-to-steal-passwords-of-ai-developers/) after a credential-stealing worm called **Miasma** spread through them. The hit list reads like a directory of modern AI engineering: `Azure/azure-functions-host`, the entire `durabletask` ecosystem, and tooling consumed daily by Claude Code, Gemini CLI, Cursor, and VS Code. It's the second compromise of the same Microsoft ecosystem in under a month. The question that hit a lot of CISO inboxes this morning is straightforward: *would Riptides have prevented this?* The honest answer is **NO and YES**. Riptides would not have stopped the repos from being compromised. That's a code-integrity problem at a different layer. What Riptides *would* have done is make the malware come up empty. Miasma exists to harvest credentials. On a Riptides protected workload, service or AI agent there are no credentials to harvest. That's the entire thesis. The rest of this post is the work behind it. ## What's actually new about Miasma Three details from [Cloudsmith's analysis](https://cloudsmith.com/blog/miasma-worms-path-of-destruction) matter for the threat model: **The attack used legitimate identity, not a software bug.** Attackers compromised a Red Hat maintainer's GitHub account and used valid OIDC tokens to publish malicious packages with valid SLSA provenance. Every conventional integrity control — maintainer signing, build attestation, registry scanning — passed. **The trigger is opening a project in an AI coding tool.** Miasma planted droppers directly in source repos for high-value targets. The payload fires when an engineer clones the repo and opens it in Claude Code, Gemini CLI, Cursor, or VS Code. Not at `npm install`. Not in CI. At the moment an AI coding agent reads the project. **The target is cloud identity.** Earlier Shai-Hulud variants scraped local secrets. Miasma ships dedicated harvesters for Azure and GCP credentials on both developer workstations and CI runners. The intent is to leave the code and get into the cloud. The pattern itself isn't new, we wrote about it months ago in [*Shai-Hulud 2.0: Why Secrets Need to Die*](https://riptides.io/blog/shai-hulud-2-0-a-technical-breakdown-and-why-secrets-need-to-die) and demoed it live in [*Growing Threat of npm Supply Chain Attacks*](https://riptides.io/blog/growing-threat-of-npm-supply-chain-attacks). The payload gets sharper, the assumption it exploits never changes: **credentials exist as files and environment variables on the machines that use them.** That is the assumption to break. ## What Riptides does not do I'll be direct, because this is where most vendor blog posts overclaim. Riptides would not have prevented Miasma from compromising the Microsoft repos. That's upstream of us — branch protection, account hardening, and artifact governance are the right controls there. Riptides would not have prevented an engineer from cloning a poisoned repo into VS Code. Code review and dependency hygiene still matter. That's the honest scope. Now the part that matters. ## What Riptides does: make the payload worthless On workloads we cover - production servers, workloads, AI agents, K8s, bare metal or VM based clusters, and crucially **CI/CD runners** — there is no static credential surface for Miasma to scrape. Static API keys for OpenAI, Anthropic, Azure, AWS, GCP, GitHub, databases, Vault, etc they don't exist as files, environment variables, or keyring entries. Credentials are issued just-in-time, bound to the specific process that needs them, and injected on the wire from kernel space at the moment of the connection. The credential never lands in user space at all. The implication is the part that should interest a CISO: - **The dropper's first move returns nothing.** Miasma enumerates `~/.aws/credentials`, `~/.config/gcloud`, `.npmrc`, the OS keyring, environment variables in `/proc/*/environ`. On a Riptides workload these are empty. The collector runs, the exfil POST still fires, the body is blank. - **Credentials captured in flight can't be replayed.** Each credential is cryptographically bound to an attested workload identity. A token harvested on one machine cannot be used from anywhere else. - **The dropper can't exfiltrate to wherever it wants.** Per-process egress policy is evaluated below user space. A poisoned extension can't open a connection to an unapproved destination, regardless of what credentials it managed to scrape. - **Investigation is a single query.** Every connection emits an attributable record: which process, which identity, which destination, which policy. Microsoft is reconstructing the Durable Task timeline by hand. On a Riptides workload, you'd have it in minutes. The economic logic of credential-stealer campaigns is straightforward — the attacker invests in delivery because the credentials are valuable. Remove the credentials and the campaign stops being profitable. ## The CI angle: partial prevention is real prevention Cloudsmith is explicit that CI/CD runners are an explicit Miasma targets and this is where the prevention is strongest. Normal Miasma flow on a build agent: dropper finds the runner's GitHub token, AWS deploy credentials, Azure service principal secret, signing keys, npm publish tokens. Cloud pivot follows within hours. On a Riptides-protected runner the dropper finds none of them. The runner authenticates to Azure, AWS, GCP, and GitHub through federated workload identity exchanged in the kernel at request time. The credential bridge between the runner and the cloud doesn't exist for the malware to cross. This is the part of the breach that gets prevented at the cascade level, not just blunted. ## The takeaway For the next 48 hours, the right response to Miasma is the one Cloudsmith laid out: assume exposure, rotate every credential the malware can touch, audit your GitHub for new repos and unfamiliar workflows. The strategic posture is different. Miasma is a template, not an endpoint. Every assumption it exploits is still true across the industry: maintainer credentials remain the porous perimeter, AI coding agents are now code-execution environments rather than editors, and credentials on disk are the actual asset attackers want. The first two are hard to fix and will keep failing. The third one is fixable today. Static analysis checks code before it runs. Runtime identity controls what it can do once it's running. Miasma slipped past every static check on the planet. Only the second layer would have stopped it from paying off. We believe that **Riptides is that layer**. --- If your workloads, CI runners or AI agents touch anything you'd rather not see in a Miasma postmortem, [request a demo](https://riptides.io/request-a-demo). --- ## The Hidden Cost of Stored Credentials - URL: https://blog.riptides.io/the-hidden-cost-of-stored-credentials - Published: 2026-06-02 - Author: Zsolt Varga - Category: Security - Tags: credentials, zero-trust, compliance, aws ## The Credentials Are Already Compromised. You Just Don't Know It Yet. Your applications store AWS credentials somewhere. In environment variables. In config files. In secrets management systems. In CI/CD pipelines. Each of those locations is an attack surface. A compromised container image now contains live AWS credentials. A developer's laptop with a cached `.aws/credentials` file gets stolen. A CI/CD log is accidentally exposed. A cloud instance with mounted secrets gets accessed by an insider. A supply chain compromise leaks your entire secrets repository. These aren't hypotheticals. They happen regularly. The industry response has been incremental: better secrets management, shorter rotation periods, more audit logging. All helpful. All incomplete. Because the fundamental problem remains: **credentials exist in plaintext, somewhere, stored at rest.** The only way to truly eliminate this attack surface is to eliminate stored credentials entirely. ## The Business Cost of Credential Compromise When AWS credentials leak, the timeline is ruthless: **Immediate impact:** - Attackers spin up expensive compute resources (EC2, Lambda, data processing jobs) - They exfiltrate data from accessible S3 buckets and databases - They create backdoor IAM users and roles for persistent access - Detection is often delayed — you're charged before you notice **Investigation and response:** - Invalidate all affected credentials (hours of work) - Audit CloudTrail logs (days to weeks) - Rotate secrets across dependent systems (weeks) - Regulatory notification requirements (compliance deadlines) **Long-term impact:** - Reputational damage - Regulatory fines (GDPR, CCPA, industry-specific requirements) - Customer trust erosion - Future audits and scrutiny The actual financial impact is hard to quantify, but real: AWS's own customers have reported bills exceeding $100,000 from cryptomining attacks following credential compromise. That's before investigation costs, downtime, and regulatory response. ## Compliance and Audit Pressure Your compliance and audit teams are already asking: where are your credentials stored? How often are they rotated? Who can access them? What's your incident response plan if they leak? The industry consensus (driven by SOC 2, FedRAMP, and zero trust frameworks) is increasingly clear: **long-lived credentials are a liability, not an acceptable control.** Standards like SPIFFE (Secure Production Identity Framework For Everyone) and zero trust architecture explicitly recommend ephemeral, workload-bound identities over stored secrets. If you're pursuing SOC 2 Type II certification or FedRAMP authorization, your auditors will scrutinize how you handle non-human identity credentials. A secretless approach directly addresses this. Your applications never see AWS credentials. They never store them. They never transmit them unsecured. Auditors see this and move on to the next control. Compliance becomes demonstrable, not theoretical. ## The Operational Burden of Credential Rotation Stored credentials require active management: - **Rotation schedules**: Every 30/60/90 days, credentials must be replaced - **Rollover windows**: New credentials deployed, old ones kept active for switchover - **Failed rotations**: A delayed rotation cascades through dependent systems - **Emergency revocation**: Leaked credentials must be invalidated immediately, often in the middle of an incident - **Audit trails**: Every rotation creates records to maintain For a typical organization with dozens of services and cloud accounts, this is a perpetual operational tax. Multiple teams rotate different credentials on different schedules. Automation helps but adds complexity and failure modes. Secretless authentication eliminates this entirely. Credentials are generated dynamically, used for a single request, and discarded. No rotation, no schedules, no emergency revocations, no manual management. ## How Secretless Authentication Works Instead of storing credentials, your applications receive short-lived credentials **just-in-time** for each request. The workflow is simple: 1. Application sends a request to an external service (AWS, GCP, etc.) 2. Riptides intercepts the request at the kernel level 3. Your workload's identity is verified against a trusted CA 4. Temporary credentials are obtained (using your workload's proven identity) 5. Those credentials are injected into the request 6. The request is authenticated and sent 7. Credentials are discarded after the request completes From the application's perspective, nothing changes. It makes the same API call it always did. Credentials are never visible to the application code, never stored on disk, never leaked in logs. ## Real-World Impact Consider a customer workload that calls AWS Bedrock. Traditionally: - AWS credentials are stored in environment variables or config files - Those credentials are valid for hours/days - If the container is compromised, an attacker has access to AWS - Credential rotation requires redeploying the container - Compliance audits ask uncomfortable questions about credential lifecycle With secretless authentication: - The workload's identity is verified by the kernel - Temporary credentials are generated automatically - The request is authenticated at the point of egress - If the container is compromised, there are no credentials to steal - Compliance audits see automatic, ephemeral credentials with no manual management The vulnerability still exists (it's the same application). But the impact is contained. ## The Path Forward Secretless authentication isn't a new concept, but it's finally becoming operationally practical. Major cloud providers now support OIDC federation, allowing workloads to prove their identity without storing credentials. Zero trust frameworks explicitly recommend this pattern. If your organization is serious about reducing credential risk, the question isn't whether to adopt secretless authentication. It's when, and which services to prioritize. Start with your highest-risk workloads: - Services that call external APIs frequently - Workloads running on shared infrastructure - Applications handling sensitive data - Services subject to compliance audits Riptides eliminates stored credentials transparently, without code changes. Your applications keep working exactly as they do today. But they're no longer carrying the liability of stored secrets. **Ready to explore secretless authentication?** [Schedule a demo](/request-a-demo) to see how Riptides eliminates credential storage, or read our technical guide on [on-the-wire credential injection](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example). --- ## On-Demand Driver Builds: How We Replaced Speculative Kernel Builds with a Demand-Driven Pipeline - URL: https://blog.riptides.io/on-demand-driver-builds-demand-driven-kernel-build-pipeline - Published: 2026-05-26 - Author: Peter Balogh - Category: Kernel - Tags: kernel, linux, build, automation ## From Speculative Builds to Just-in-Time Kernel Coverage In our previous posts — [Building Linux Driver at Scale](/blog/building-linux-driver-at-scale-our-automated-multi-distro-multi-arch-build-pipeline) and [Beyond the Limits](/blog/beyond-the-limits-scaling-our-kernel-module-build-pipeline-even-further) — we described how we compile kernel drivers nightly for a matrix of distributions, architectures, and kernel versions. The current batch covers roughly 540 variants across Ubuntu, Amazon Linux, Fedora, Debian, and CentOS, on both x86_64 and aarch64. That covers the common case well. The nightly run picks up new kernels from Falco's [`kernel-crawler`](https://github.com/falcosecurity/kernel-crawler), generates a diff against the previously built set, and rebuilds only what changed. But coverage has a natural ceiling. Cloud providers push live-patched kernels on their own schedule. Customers run specific distribution versions that diverge from our pinned defaults. New distribution releases land between our index updates. The long tail of kernel variants a real fleet encounters is essentially unbounded. When a Riptides node boots and its kernel driver is not in the prebuilt set, the Riptides daemon needs to work now, not at the next nightly run. We needed a way to trigger a build for exactly the right kernel, on demand, and have the result ready within minutes. ## The Build Service The solution is a lightweight Go HTTP service — the `build-service` — that sits between the Riptides driver-loader and the GitHub Actions build pipeline. The API is intentionally minimal: | Endpoint | Description | |---|---| | `POST /build` | Trigger a driver build | | `GET /status/{id}` | Poll build status | | `GET /healthz` | Health check | A build request carries exactly what the compiler needs: ```json { "kernel_version": "6.12.58-82.121.amzn2023.x86_64", "architecture": "x86_64", "distribution": "amazonlinux", "distro_version": "2023", "driver_version": "v0.5.16" } ``` Before doing anything, the service checks whether the package already exists in GitHub Releases. If it does, the response is immediate with no workflow dispatch needed. If it does not, the service validates the request — checking the kernel version format, architecture, distribution, and distro version — then fires a `workflow_dispatch` event to the `driver-build.yml` GitHub Actions workflow with the exact inputs needed. The caller gets back a build ID and polls `/status/{id}` until the state transitions to `done` or `error`. ```json {"message": "workflow dispatched", "build_id": "a3f92b1c"} ``` ```bash $ curl /status/a3f92b1c {"status": "in_progress"} $ curl /status/a3f92b1c {"status": "done"} ``` Under the hood, the service polls the GitHub Actions API every 15 seconds to find the workflow run that was dispatched after a given timestamp, then monitors it until completion. The maximum wait is 15 minutes before the build is marked as timed out. The service also deduplicates concurrent requests — if two nodes boot on the same new kernel simultaneously, only one workflow is dispatched and both callers receive the same build ID to poll. ![Build service flow](../../assets/on-demand-driver-build/flow-build-service.jpg) ## Driver Distribution: RPM and DEB Packages Every build produces a native package for the target distribution — a `.deb` for Debian-based distros and an `.rpm` for RPM-based ones — published to GitHub Releases. The package name encodes everything needed to locate the right artifact: ``` riptides-driver-ubuntu-6.14.0-1021-gcp_v0.1.2_amd64.deb riptides-driver-amazonlinux-6.12.58-82.121.amzn2023.x86_64_v0.1.2_amd64.rpm ``` In both deployment environments the driver-loader follows the same flow: read `/etc/os-release` and `uname -r` to identify the exact distribution, version, and kernel, construct the package name, and attempt to download it from GitHub Releases. If the download succeeds, it proceeds to load the driver. If the response is a 404 — the package has not been built yet — it calls the build-service, waits for the build to complete, then downloads and loads the result. How the package is loaded once downloaded differs by environment. **VM and bare metal** — the driver-loader runs as a systemd service on the host and installs the Riptides driver package directly with the system package manager. **Kubernetes** — the driver-loader runs as a privileged container but the kernel module must be loaded into the host kernel. Rather than installing the package on the host and risking package manager conflicts, the driver-loader extracts the `.ko` files from the package inside the container and uses `insmod` to load them directly into the host kernel. This unified package-based approach means the same build artifact works across all deployment models — no separate paths for binary blobs, no per-environment build variants. ![Driver loader flow](../../assets/on-demand-driver-build/flow-driver-loader.jpg) ## Filling the Gaps: Koji Fallbacks and Livepatch Discovery Whether a build is triggered on demand or by the nightly batch, two specific gaps keep showing up in practice when compiling drivers for certain distributions. ### Fedora and CentOS Stream: Koji Fallbacks RPM-based distribution mirrors are deliberately shallow. Older kernel-devel packages age out quickly, and the standard DNF repositories often do not carry the version actually running on a customer's node. For Fedora, the answer is [Fedora Koji](https://koji.fedoraproject.org/) — the official Fedora build system that retains packages long after they leave the standard mirrors. CentOS Stream has its own equivalent: the [CentOS Build Service (CBS)](https://cbs.centos.org/). The build script detects which distribution it is running on and selects the right Koji profile automatically: ```bash KOJI_PROFILE="koji" # Fedora Koji by default if [[ "$ID" == "centos" ]]; then KOJI_PROFILE="cbs" cat > /etc/koji.conf.d/cbs.conf <<'EOF' [cbs] server = https://kojihub.stream.centos.org/kojihub weburl = https://kojihub.stream.centos.org/koji topurl = https://kojihub.stream.centos.org/kojifiles EOF fi koji --profile="$KOJI_PROFILE" download-build \ --arch="$ARCH" --arch=noarch --rpm \ "${KERNEL_DEVEL_PACKAGE}-${KERNEL_VERSION}" ``` The fallback chain for any RPM-based distribution is: DNF repo → explicit kernel URLs → Koji/CBS. This means we can build for Fedora and CentOS Stream kernel versions that have rolled off the standard mirrors without maintaining our own package mirror. ### Amazon Linux: Livepatch Kernels Amazon Linux 2023 ships live-patched kernels through a separate CDN repository (`kernel-livepatch-repo-cdn`). These kernels never appear in Falco's [`kernel-crawler`](https://github.com/falcosecurity/kernel-crawler) data because they are not published through the standard package mirrors that kernel-crawler scrapes. In practice, livepatch kernels are handled naturally by the on-demand path — when a node running a livepatch kernel boots and requests a build, the driver-loader triggers the build-service and the result lands in `kernels.json` like any other kernel. No special discovery step is needed for the default flow. For the optional kernel-crawler mode however, livepatch kernels would be silently missed. To close that gap, we added a discovery step that queries the livepatch CDN repository directly and merges the results into the kernel-crawler data before generating the build matrix. On the build side, the Amazon Linux Dockerfile also handles the livepatch install path — if the standard kernel-devel RPM is not available, the build script falls back to installing the matching `kernel-livepatch` package instead. ## Distro Versioning Done Right "Ubuntu" is not enough information to build a kernel module. You can technically compile a driver for an Ubuntu 22.04 kernel inside an Ubuntu 24.04 container — the build may succeed — but the default toolchain version differs between distro releases. That mismatch can cause subtle driver load errors at runtime because the compiler embeds version-specific metadata into the module. Pinning an exact toolchain version for every distro version combination is fragile and hard to maintain. The simpler solution is to just build on the same distro and version as the target. A driver destined for an Ubuntu 22.04 host gets compiled in an Ubuntu 22.04 container. The toolchain matches by construction. To make that work end to end, we pass `distro_version` through the entire pipeline: from the build-service request, through the GitHub Actions workflow inputs, into the Docker build arguments, and into the artifact cache key. This way the build environment is always correctly aligned with the target, and different versions of the same distribution are treated as distinct build variants rather than being conflated. The `distro_version` field is optional in the request. If omitted, the service uses the pinned default per distribution (e.g., `24.04` for Ubuntu). The build container then pulls the matching prebuilt base image from our GHCR registry: ```bash BASE_IMAGE=ghcr.io/riptideslabs/${distribution}:${version} ``` The build cache key also includes the distro version so that concurrent builds for `ubuntu:22.04` and `ubuntu:24.04` with the same kernel version are treated as distinct builds rather than deduplicated: ```go func buildCacheKey(req BuildRequest) string { return fmt.Sprintf("%s:%s:%s:%s:%s", req.KernelVersion, req.Architecture, req.Distribution, req.DistroVersion, req.DriverVersion) } ``` In the default `kernels.json` mode this is straightforward — the `distroversion` field is already recorded in the file by the on-demand build, so the matrix generator reads it directly. When `kernel-crawler` mode is used instead, the raw kernel-crawler data does not carry an explicit distro version. In that case `matrix-gen` derives it from the kernel release string itself, since the version is encoded there by convention: | Distribution | Pattern in kernel release | Example | |---|---|---| | Fedora | `.fcXX.` | `6.14.0-100.fc42.x86_64` → `42` | | CentOS | `.elXX` | `6.12.0-212.el10.x86_64` → `stream10` | | Debian | `~bpoXX+` | `6.1.0-28~bpo12+1` → `12` | | Amazon Linux | target name suffix | `amazonlinux2023` → `2023` | This way the kernel-crawler path produces correctly versioned build entries without any manual annotation, consistent with what the on-demand path records explicitly in `kernels.json`. ## Custom Base Images The nightly batch was already using prebuilt base image tarballs to avoid registry rate limits during large parallel runs, as described in our [previous post](/blog/beyond-the-limits-scaling-our-kernel-module-build-pipeline-even-further). On-demand builds raised the same issue in a different form: sporadic single-kernel builds were still pulling from public registries, and those pulls were slow and occasionally throttled. We extended the base image strategy to cover all distributions and versions used in on-demand builds. Each base image is built from a versioned `Dockerfile.base.{distro}` and pushed to GHCR. The base images are rebuilt on a weekly cron (`0 2 * * 1`) and on manual trigger. Every distro/version combination we support has a corresponding base image: | Distribution | Versions | |---|---| | Ubuntu | 22.04, 24.04, 25.10, 26.04 | | Debian | 12, 13 | | Fedora | 42, 43 | | CentOS | stream9, stream10 | | Amazon Linux | 2023 | The base images include all compiler toolchain dependencies pre-installed — build-essential or gcc/make equivalents, kernel header tools, OpenSSL, elfutils, TPM tools, and anything else the driver compilation requires. The per-kernel Dockerfiles now just pull from the prebuilt base and install the target kernel headers on top: ```dockerfile ARG DISTRO_VERSION=24.04 FROM ghcr.io/riptideslabs/ubuntu:${DISTRO_VERSION} ARG KVERSION ENV KVERSION=${KVERSION} # kernel header install + driver build only ``` This eliminates the per-build package installation that previously happened inside each worker container, cutting several minutes off individual build startup time. For on-demand builds that matters directly: the driver-loader is blocked waiting for the build to complete before it can load the driver and the node becomes fully operational. A faster build means a faster node start. ## kernels.json as the Source of Truth The original batch build pipeline used Falco's [`kernel-crawler`](https://github.com/falcosecurity/kernel-crawler) as its primary input. It scrapes distribution mirrors and produces a comprehensive list of every kernel version that exists across all supported distros — hundreds of entries per run. That gave us broad speculative coverage: we would precompile for kernels before any of our customers encountered them. The problem is that most of those kernels never appear in any real fleet. Building for all of them up front costs significant CI time and runner capacity for variants that may never be needed. We flipped the default: the nightly batch now reads from `kernels.json` instead of kernel-crawler. When `use_kernel_crawler` is `false`, the matrix generator reads `.github/kernels.json` directly and builds only the kernels recorded there. The `kernels.json` is not written by hand. Every successful on-demand build appends the kernel it just compiled to the file via an automated PR. The result is a kernel list that reflects the kernels Riptides nodes have actually booted on — demand-driven rather than speculatively broad. ```json { "x86_64": [ { "target": "amazonlinux", "kernelrelease": "6.12.58-82.121.amzn2023.x86_64", "distroversion": "2023" }, { "target": "ubuntu", "kernelrelease": "6.14.0-1021-gcp", "distroversion": "24.04" } ] } ``` The PR is created with a predictable branch name so concurrent builds for the same kernel do not open duplicates: ```bash BRANCH="add-kernel/${DISTRIBUTION}/${ARCHITECTURE}/$(echo "${KERNELRELEASE}" | tr '.' '-')" if git ls-remote --exit-code --heads origin "$BRANCH" > /dev/null 2>&1; then echo "Branch $BRANCH already exists, skipping PR creation." exit 0 fi ``` The flow is: 1. A node boots on a kernel not in `kernels.json` → on-demand build triggers 2. Build completes → PR opens adding the kernel to `kernels.json` 3. PR merges → nightly batch includes this kernel going forward 4. Next node booting on the same kernel finds its driver prebuilt in GitHub Releases The first node to encounter a new kernel pays the on-demand build latency. Every node after that finds the driver already available. `kernels.json` converges toward the actual kernel distribution of the fleet automatically. [`kernel-crawler`](https://github.com/falcosecurity/kernel-crawler) remains available as an opt-in (`use_kernel_crawler: true`), but we are not actively using it. The on-demand path combined with `kernels.json` covers everything the fleet actually needs. ## Conclusion The nightly batch and the on-demand path form a closed loop. On-demand handles kernels we have never seen; the batch rebuilds kernels we have, keeping drivers fresh as new driver versions ship. Together they ensure that a Riptides node can always load the right driver regardless of which kernel it finds at boot. **Key takeaways:** - Check for an existing artifact before dispatching any build — most on-demand requests will be served instantly from the existing prebuilt set. - Use atomic check-and-reserve under a write lock to prevent duplicate workflow dispatches for concurrent requests. - Build a Koji fallback chain (DNF → explicit URLs → Koji/CBS) so that kernels that have rolled off distribution mirrors remain buildable. - Amazon Linux livepatch kernels are handled naturally by the on-demand path — no special discovery needed. If using kernel-crawler mode, extend its data with the livepatch CDN repository, as those kernels never appear in standard mirrors. - Build on the same distro version as the target — cross-version builds may succeed but toolchain differences embed version-specific metadata that causes driver load errors at runtime. - Prebuilt base images in GHCR eliminate public registry rate limits and cut per-build startup time, which directly reduces the time a node waits before its driver is ready. - Default to a demand-driven kernel list over a speculative crawl — build what the fleet actually needs, not everything that exists. Interested in how the batch build pipeline works? See our earlier posts: - [Building Linux Driver at Scale: Our Automated Multi-Distro, Multi-Arch Build Pipeline](/blog/building-linux-driver-at-scale-our-automated-multi-distro-multi-arch-build-pipeline) - [Beyond the Limits: Scaling Our Kernel Module Build Pipeline Even Further](/blog/beyond-the-limits-scaling-our-kernel-module-build-pipeline-even-further) If you enjoyed this post, follow us on [LinkedIn](https://www.linkedin.com/company/riptidesio/) and [X](https://x.com/riptidesio) for more updates. If you'd like to see Riptides in action, [get in touch with us for a demo](https://riptides.io/request-a-demo). --- ## Why Deep Kernel Security Matters for Enterprises — When eBPF Falls Short - URL: https://blog.riptides.io/why-deep-kernel-security-matters-for-enterprises - Published: 2026-05-26 - Author: Balint Molnar - Category: Security - Tags: kernel, security, workload-identity, zero-trust ## The Observability Trap Your security team has great visibility. You see network flows. You monitor connections. Your eBPF-based tools flag suspicious activity in real time. But visibility without enforcement is a spectator sport. When an attacker compromises a workload, the sequence is predictable: they scan the network, discover internal services, and authenticate using ambient credentials or stolen tokens. Your monitoring shows every step — but it doesn't stop it. This is the fundamental challenge enterprises face today. Tools that observe threats are table stakes. What separates contained incidents from breaches is whether you can *enforce* trust at the point of connection. ## The Hidden Risk: Credentials in Motion Most enterprise workloads authenticate using one of three mechanisms: - **Embedded credentials**: API keys, database passwords stored in config or environment - **Short-lived tokens**: OAuth tokens, JWT tokens from cloud providers - **Certificate-based identity**: X.509 certificates managed by a central CA Observability tools are designed to see that this is happening. They're not designed to prevent unauthorized access when attackers get those credentials. And they do. Regularly. In a 2024 security incident, attackers didn't exploit a software vulnerability. They found a staging database password in a CircleCI environment variable. Detection took weeks. The damage was extensive. No amount of network telemetry would have stopped it — the credentials were legitimate, even if the user holding them wasn't. This is where kernel-level enforcement changes the game. ## From Visibility to Control When identity is enforced at the kernel level, the attack chain breaks at a different point. An attacker can still achieve code execution inside a workload. But that process — the attacker's shell, their injected binary — has no identity of its own. It cannot authenticate to internal services, databases, or APIs, even if it finds the credentials. The kernel authenticates based on process identity, not the tokens or keys a process happens to possess. This transforms the risk profile. RCE becomes a localized incident, not a cascading breach. ### What This Means Operationally **Reduced dwell time**: Threats are contained before lateral movement occurs. Your MTTR (mean time to recover) collapses because the blast radius is inherently limited. **Lower credential risk**: Short-lived, process-bound identities replace long-lived secrets. Even if an attacker finds a credential, it's useless outside the process that generated it. **Compliance acceleration**: Zero trust security models now have a native enforcement layer. SOC 2, FedRAMP, and other compliance frameworks increasingly require cryptographic identity and mutual authentication — kernel-level enforcement delivers this natively. **Reduced security operations burden**: Fewer false positives from visibility-only tools. Fewer incident response escalations. Fewer lateral movement attempts to investigate. ## Why eBPF Alone Falls Short eBPF is purpose-built for observability. It excels at: - Tracing network flows in real time - Detecting anomalous behavior patterns - Logging system calls and context switches - Filtering and sampling high-volume events But eBPF runs in a sandbox by design. It cannot: - Generate and manage cryptographic keys - Perform TLS handshakes - Inject or replace authentication headers transparently - Enforce filesystem-level access control with process binding - Create and manage scoped authentication paths When you need to *prevent* an unauthorized process from connecting to a database, observability is insufficient. You need enforcement. And enforcement requires kernel integration. ## The Enterprise Decision Organizations typically face this decision after an incident. A workload gets compromised. Attackers move laterally. The post-mortem shows that all the suspicious activity was visible in logs — it just wasn't blocked. The question then becomes: how do we change this architecture so the next incident is contained before it becomes a breach? Kernel-level identity solves this. Riptides brings that enforcement to any Linux environment without requiring code changes. Your applications keep working exactly as they do today. But internally, every connection is authenticated, every service proves its identity before communicating, and any process without legitimate identity simply cannot reach protected resources. It's the difference between watching an attack happen and stopping it from succeeding. ## What Comes Next If your organization is serious about zero trust, you're already investing in workload identity. The question is whether that identity is enforced or merely declared. Observability and enforcement both matter — but enforcement is what stops breaches. The economics are straightforward: the cost of a contained incident is your response time. The cost of a cascading breach is your customer data, regulatory fines, and reputation. Kernel-level enforcement shifts the outcome dramatically in your favor. **Interested in how kernel-level identity enforcement works in practice?** [Reach out for a demo](/request-a-demo) to see Riptides in action, or read our technical deep dive on [how we implement transparent kernel-based identity](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). --- ## Introducing KeyLedger: Because You Probably Don't Know How Many AI Keys Your Org Has - URL: https://blog.riptides.io/keyledger-ai-api-keys - Published: 2026-05-18 - Author: Balint Molnar - Category: Security - Tags: AI, Agentic, Go Try this thought experiment. Open every AI provider dashboard your organization uses. Count the keys. OpenAI: a handful of active keys scattered across several projects. Anthropic: a dozen or more across a few workspaces. Google Cloud: service accounts with IAM keys for Vertex AI that nobody's audited in months. AWS Bedrock: access keys attached to IAM users whose owners you'd have to look up. You'll find dozens of credentials, issued by multiple providers, managed through multiple dashboards, with different API shapes and different definitions of what metadata to expose. **Now ask yourself:** is one of those OpenAI keys over a year old and never been used? Is there an Anthropic key in the default workspace, created by someone who left the company two quarters ago? Has an AWS access key gone unrotated since before your last performance review? In most organizations, the answer to all three is yes. Nobody notices, because there's no single place to look. That's why we built **KeyLedger**, and we're open-sourcing it today. --- ## What KeyLedger Does KeyLedger is a Go TUI tool that connects to AI provider admin APIs and gives you a unified inventory of every API key issued across your organization. One command, all providers, one table. ![KeyLedger Demo](/images/keyledger-blog/keyledger.gif) No database server. No background process. No infrastructure. One binary. --- ## What You Get Out of the Box KeyLedger ships with six capabilities designed to fit how infrastructure teams actually work, from interactive exploration to fully automated CI pipelines. **Interactive TUI.** A full-screen terminal dashboard built with [Bubble Tea](https://github.com/charmbracelet/bubbletea). Browse, filter, and sort keys across every provider from one place. If you've used `k9s` for Kubernetes or `lazygit` for git, you know the workflow. KeyLedger brings the same experience to AI key management. No browser required. **Health scoring.** Every key gets an automatic risk score based on configurable thresholds. Keys older than 90 days get flagged as stale. Keys unused for 30 days get flagged as idle. Keys that were created and never used at all get flagged as critical. You configure the thresholds; KeyLedger does the math across every provider in every workspace and project. **Snapshots and diffs.** KeyLedger stores point-in-time snapshots in a local SQLite database. Diff any two snapshots — or a snapshot against the current live inventory — to see exactly what changed: new keys issued, keys revoked, status changes, scope changes. This gives you rotation history that no provider dashboard offers natively, since their APIs only return the current state. **Watch mode.** The interactive TUI is great at a terminal, but pipelines need non-interactive commands. KeyLedger ships with `watch` as standalone CLI commands. The `watch` mode runs continuously, polling providers on a schedule and alerting when things change. **Encrypted credential storage.** The admin keys you configure (your OpenAI admin key, your Anthropic admin key) are stored in an AES-256-GCM encrypted SQLite database, unlocked with a password at the start of each session. No OS keyring dependency, no plaintext config files with credentials sitting on disk. The encryption is handled by KeyLedger itself, one less external dependency to manage. **Docker-ready.** A pre-built Docker image runs KeyLedger in `watch` mode with a built-in unseal API, so credentials can be supplied at runtime without a terminal. This makes it straightforward to deploy as a long-running service in Kubernetes or Docker Compose, polling your providers continuously and exposing health status. --- ## Why AI Key Hygiene Matters Now If you're running AI agents in production — and at this point, most of us are — your API key surface area has grown faster than your processes to manage it. Here's why that's a problem: **Financial exposure.** An OpenAI key with no spend limits can rack up tens of thousands of dollars overnight if it lands in the wrong hands. Unlike a leaked database password that requires further exploitation, a leaked LLM API key is immediately monetizable. Copy, paste, run inference. That's it. **Key sprawl is accelerating.** Every new agent, every new environment, every PoC, every integration spins up new keys. Developers create them in the provider's dashboard, drop them in an `.env` file, and move on. Six months later nobody remembers they exist. The key is still active, still has full permissions, and probably still sitting in someone's shell history. **Rotation isn't happening.** Be honest: when was the last time you rotated your AI provider keys? Most teams treat them as set-and-forget. Meanwhile, the compliance frameworks you're bound to (SOC 2, ISO 27001, PCI DSS) all require key rotation policies. You can't enforce a rotation policy if you don't have an inventory. **No unified visibility exists.** Every provider has its own dashboard. OpenAI organizes keys by projects. Anthropic uses workspaces. Google Cloud nests them under service accounts inside projects. AWS ties them to IAM users. If you want to answer "how many active AI keys do we have?" you're clicking through 4+ dashboards and correlating manually. That last point is the one that really drove this project. The open source ecosystem has excellent tools for *finding* leaked secrets — TruffleHog, Gitleaks, Betterleaks — but **nothing** that answers the other half of the question: "what keys are legitimately issued?" We looked. Extensively. It doesn't exist. AI gateways like LiteLLM route requests but don't query provider admin APIs. Secret managers store credentials but can't inventory what a third party has issued. The provider inventory side is a gap. KeyLedger fills it. --- ## Provider Breakdown ### Tier 1: Full Admin API Coverage These are the providers where we can give you the richest picture, because they expose comprehensive admin APIs with key listing, owner information, and (in some cases) usage tracking. #### OpenAI OpenAI organizes keys in a two-level hierarchy: **Organization → Projects → Keys**, with admin keys living at the org level. KeyLedger enumerates all projects in your organization (via `GET /v1/organization/projects` with pagination), then queries keys within each project, plus org-level admin keys separately. The metadata you get back is the most complete of any provider we support: - **Key ID and name** — the identifier and human-readable label - **Created timestamp** — when the key was issued - **Last used timestamp** — when the key was last used (this is big — most providers don't expose this) - **Owner** — full name, email, role, and whether it's a user or service account - **Redacted key value** — partial hint for identification - **Project** — which project the key belongs to The `last_used_at` field is genuinely valuable. Combined with `created_at`, it lets KeyLedger flag keys that were created but never used, or keys that have gone idle — both of which are rotation candidates or revocation targets. **A note on reliability (a telling one):** OpenAI exposes two separate listing endpoints — one for [project-scoped API keys](https://developers.openai.com/api/reference/resources/admin/subresources/organization/subresources/projects/subresources/api_keys/methods/list) and one for [org-level admin API keys](https://developers.openai.com/api/reference/resources/admin/subresources/organization/subresources/admin_api_keys/methods/list). During development we hit persistent 404s on the admin keys endpoint. It was just broken. So we [opened a thread on the OpenAI community forum](https://community.openai.com/t/persistent-404-not-found-for-v1-organization-admin-api-keys/1379606), and OpenAI support eventually confirmed and fixed it. But here's the part that stuck with us: once the thread was live, other users started chiming in — "same here," "also getting 404s," "oh, it works now." The endpoint had been failing intermittently, and nobody had reported it. In a world where we talk endlessly about AI security, the endpoint that lets you *list your own admin keys* was quietly returning 404s and nobody had noticed, because nobody was calling it. If that doesn't tell you something about how few organizations are actually tracking their AI credentials, we don't know what does. *KeyLedger handles this with retry logic and graceful degradation: if the admin keys endpoint returns 404, the project-level inventory still completes and a warning is logged.* #### Anthropic Anthropic organizes keys under **Organization → Workspaces → Keys**. Keys in the default workspace have a null `workspace_id`. KeyLedger queries the Anthropic admin API without a workspace filter to get keys across all workspaces in a single call. It then makes secondary calls to resolve workspace names (from `GET /v1/organizations/workspaces`) and user names and emails (from `GET /v1/organizations/users`, since the keys API only returns user IDs in the `created_by` field). What you get: - **Key ID, name, and partial hint** - **Created timestamp** - **Status** — active, inactive, or archived - **Created by** — user ID (resolved to name/email via secondary call) - **Workspace** — which workspace the key belongs to Notable limitation: **Anthropic does not expose a `last_used_at` field.** KeyLedger fetches a 30-day usage report (/v1/organizations/usage_report/messages) and matches usage to keys by ID. Keys with no usage in the past 30 days show ‘>31 days ago’ as the Last Used value, since the usage window only covers 31 days of history — the exact last-used date beyond that window is not retrievable. This usage-report workaround gives you *some* signal — you can tell whether a key was used in the last 30 days — but it has real gaps: you can't tell the exact last-used date for keys idle longer than a month, and the matching relies on key IDs appearing in usage records, which can miss edge cases. A native `last_used_at` field on the keys API would eliminate all of this. If you're an Anthropic PM reading this — please consider adding per-key usage timestamps to your admin API. It's a small change on your end and would make idle-key detection dramatically more reliable. #### Google Cloud (Vertex AI / Gemini) Google Cloud has the deepest hierarchy of any provider: **Organization → Projects → Service Accounts → Keys**. A single GCP organization can have dozens of projects, each with multiple service accounts, each with multiple keys. KeyLedger enumerates all projects accessible to the service account you configure, then discovers all service accounts within each project, then lists keys per service account. This three-level traversal ensures nothing is missed. What you get: - **Key ID and algorithm** - **Key origin** — whether the key was created by GCP or user-provided - **Key type** — user-managed vs. system-managed - **Valid after / valid before** — the key's validity window - **Disabled status** - **Project and service account** — the full path in the hierarchy By default, KeyLedger filters out GCP-managed keys (which GCP auto-creates and rotates for service accounts), since they're typically noise. You can include them via config if you need a complete picture. Notable limitation: **No `last_used_at` in the keys API.** Google offers this data separately through the IAM Activity Analyzer, which we plan to integrate in a future release. #### AWS (Bedrock / IAM) AWS doesn't organize keys by project or workspace — it ties them to **IAM Users**. Every IAM user can have up to two access keys. KeyLedger calls `ListUsers` to enumerate all IAM users in the account, then `ListAccessKeys` for each user, then `GetAccessKeyLastUsed` for each key. This gives you the most complete usage picture of any provider: - **Access Key ID** - **IAM user** — who owns the key - **Created timestamp** - **Status** — active or inactive - **Last used timestamp** — when the key was last used - **Last used service** — which AWS service was called (e.g., `bedrock`, `s3`) - **Last used region** — from where The `last_used_service` field is uniquely valuable — it tells you not just *when* a key was used but *what for*. A key that last called S3 two years ago probably isn't being used for Bedrock. ### Tier 2: Work in Progress We've built provider stubs for Mistral, Cohere, Pinecone, Groq, Together AI, Fireworks AI, ElevenLabs, DeepSeek, and Replicate. The honest reality is that almost none of them have a management API that supports programmatic key listing today. We're in active conversations with **Mistral**, where there's a real possibility of proper admin API support. For the rest, key management is dashboard-only — you log into the web console, and that's your inventory tool. *Mistral is now working with the help of session tokens. Please check the documentation.* We're keeping these in the codebase because the landscape is evolving quickly. As these providers mature and introduce admin APIs (and we believe they will, as enterprise adoption demands it), we'll be ready to integrate. If you have contacts at any of these providers or know of undocumented APIs, we'd love to hear from you. In the meantime, KeyLedger's pluggable architecture means you can write a provider in under 100 lines of Go. --- ## Riptides: Where KeyLedger Meets Zero-Trust Key Management KeyLedger is open source and always will be. It's a read-only audit tool — it tells you what keys exist, flags what's stale, and diffs what changed. It does not manage, rotate, or deliver keys. For teams that need the full lifecycle — not just visibility, but credential protection, rotation, and enforcement — **[Riptides](https://riptides.io)** is our commercial platform, and KeyLedger is embedded in it. ### KeyLedger Runs Automatically Inside Riptides With Riptides, provider enumeration runs **automatically and continuously** for every customer. The Riptides control plane polls your AI provider admin APIs on a schedule, tracks key inventories over time, diffs snapshots, and alerts when something changes — a new key appears, an old key goes stale, a key gets revoked. You don't run a CLI; it's already running. The admin credentials needed to query provider APIs (your OpenAI admin key, your Anthropic admin key, your GCP service account) are sourced from **configurable secure backends** — HashiCorp Vault, Kubernetes Secrets, AWS Secrets Manager, or the Riptides native credential store. From the control plane, you can rotate, revoke, and set rotation schedules for every provider credential, with full audit logging and policy enforcement. ### Credentials Never Meet User Space Here's the part that changes the security model fundamentally. Riptides deploys as a **kernel module** — not a sidecar, not a proxy, not an SDK. When your AI agent makes an outbound API call to OpenAI, Anthropic, or any other provider, the Riptides kernel module intercepts the request, matches the agent's SPIFFE identity to a credential binding, and **injects the API key into the request on the wire, in kernel space**. The credential never enters process memory. There's no `.env` file, no config variable, no `Authorization` header your code constructs. The key exists only in kernel memory, only for the duration of that specific request, only for the process that was authorized by policy to use it, and with the shortest possible lifespan. From the developer's perspective nothing changes — the API call works as if the key were configured normally — but from a security perspective, there's nothing to leak because the credential never touches user space. This is how Riptides handles all credential types — AI provider API keys, SigV4 signatures, bearer tokens, mTLS certificates — and it works with any agent framework (LangChain, CrewAI, OpenAI Agents SDK, MCP-based agents), any language, any runtime. No code changes required. Agents cannot bypass, disable, or route around it. ### Active Key Scanning and Lateral Movement Prevention Visibility into what's issued is only half the picture. Riptides also actively scans for exposed AI provider keys across the places they shouldn't be: - **Code repositories** — full git history scanning, not just the current tree, including every commit, branch, and stash - **GitHub organizations** — org-wide scanning across all repos - **File systems** — `.env` files, config files, dotfiles, shell histories, IDE configs - **Environment variables** — runtime inspection of deployed environments - **Communication channels** — Slack messages, where developers routinely paste keys during debugging This scanning layer is critical for preventing **lateral movement**. If an attacker compromises a developer workstation, one of the first things they look for is API keys — in shell history, config files, browser storage, and Slack. Leaked AI keys aren't just a billing risk; they're a pivot point for broader access if the key's provider account connects to other systems. Riptides cross-references what the scanners find against what KeyLedger's provider inventory reports. The result: you know not just that a key leaked, but whether it's still active, who issued it, which workspace it belongs to, and whether it should be auto-revoked. And because Riptides injects credentials at the kernel rather than exposing them to applications, the attack surface for future leaks collapses — there are no keys in user space to find. --- ## Getting Started ```bash # Install go install github.com/riptideslabs/keyledger/cmd/keyledger@latest # Run keyledger ``` Full configuration options, provider setup guides, and the JSON config reference are in the [project README and documentation](https://github.com/riptideslabs/keyledger). --- ## What's Next and How You Can Help KeyLedger is at v0.1. It does what it says — lists keys across providers, scores their health, diffs between runs — and it does it well. But there's a lot more we want to build: - **Tier 2 provider support** as their admin APIs mature - **OAuth grant auditing** — querying GitHub, Slack, and Google for third-party app authorizations granted to AI platforms (the "deposited keys" problem) - **Cost attribution** — linking keys to usage spend where providers expose billing APIs - **RBAC auditing** — showing what permissions each key has, not just that it exists - **Webhook notifications** — automated alerts when critical changes are detected between snapshots We built KeyLedger because we needed it. If you're managing AI infrastructure at any scale, you probably need something like it too. We'd rather build it together. **Try it, break it, tell us what's missing.** Open an issue, submit a PR, or just star the repo if the problem resonates with you. [**https://github.com/riptideslabs/keyledger**](https://github.com/riptideslabs/keyledger) --- *KeyLedger is maintained by the team at [Riptides](https://riptides.io). KeyLedger is MIT licensed and free forever.* --- ## Anthropic Workload Identity Federation with Riptides - URL: https://blog.riptides.io/anthropic-workload-identity-federation-support - Published: 2026-05-13 - Author: Zsolt Varga - Category: Federation - Tags: federation, non-human identity, cloud Anthropic recently added workload identity federation support for Claude, and our reaction was: *This is exactly the direction the industry should be moving toward.* Because the alternative was getting increasingly awkward. Modern AI workloads need access to everything. Claude APIs, MCP servers, GitHub, Slack, cloud providers, internal services, databases, customer systems. Once you start wiring agents and tools together, credentials begin spreading through runtime environments surprisingly fast. Most systems still handle this the old way: generate another API key, push it into a secret store, inject it into a container, and hope it does not accidentally leak somewhere along the way. This works until you start dealing with highly dynamic workloads, transient agents, orchestration systems, MCP servers, or tool chains where credentials move constantly between components. At that point, operational overhead alone becomes painful, let alone the security implications. This is why Anthropic’s workload identity federation support is genuinely useful. You can read the documentation here: https://platform.claude.com/docs/en/manage-claude/workload-identity-federation Instead of distributing permanent Anthropic credentials into workloads, Claude can now trust workload identities coming from systems the workload already runs under. That model maps directly onto how Riptides already works. A workload running under Riptides can securely communicate with Claude without developers manually managing Anthropic API keys at all. The workload itself does not need to implement token refresh logic, credential rotation, revocation handling, or federation flows. Riptides handles the entire lifecycle transparently underneath using runtime workload identity. The actual configuration is intentionally very small. First, you define the Anthropic federation configuration itself as a `CredentialSource`: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialSource metadata: name: anthropic-wif namespace: riptides-system spec: anthropic: organizationId: 6bf...........................b3b9 workspaceId: default serviceAccountId: svac_012Vh.........QsxofNQzN3 federationRuleId: fdrl_01X3a.........5iLuiRdjvs ``` Then you bind it to workloads using a `CredentialBinding`: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialBinding metadata: name: anthropic-wif namespace: riptides-system spec: credentialSource: anthropic-wif propagation: injection: selectors: - svc:name: anthropic-api workloadID: riptides-demo/app/foo ``` That is it. The workload simply talks to Claude normally. Riptides intercepts the connection, handles workload identity federation with Anthropic, injects short-lived credentials transparently, and continuously manages the credential lifecycle underneath. Rotation, expiration, and revocation are all handled automatically. More importantly, the application itself never needs to permanently hold reusable Anthropic credentials in process space. That becomes increasingly important once you start dealing with real-world AI infrastructure instead of isolated demos. Agents, MCP servers, orchestration systems, and temporary workloads create environments where distributing static credentials everywhere becomes both operationally difficult and fragile from a security perspective. The healthier model is letting workloads authenticate using runtime identity while the infrastructure handles the federation and credential lifecycle automatically. That is exactly what Riptides was already built around, which is why Anthropic’s federation model fits into the platform naturally. --- ## Bearer Token Meets Runtime Enforcement - URL: https://blog.riptides.io/bearer-token-meets-runtime-enforcement - Published: 2026-05-06 - Author: Zsolt Varga - Category: Security - Tags: zero-trust, credentials, security Modern infrastructure has become very good at issuing credentials. JWTs, SPIFFE identities, federated trust, workload identity providers, and short-lived certificates have significantly improved how systems authenticate workloads and exchange trust information across increasingly dynamic environments. These are important advances, but they also introduced an assumption that deserves closer examination. Many systems implicitly treat possession of a valid credential as sufficient proof for access. If a token is correctly signed, issued by a trusted authority, unexpired, and presented successfully, the infrastructure often assumes the current actor presenting it should be trusted. That assumption is increasingly breaking down. A valid token proves that somebody authenticated successfully at some earlier point in time. It does not necessarily prove that the current actor presenting it is still legitimate, that the workload context remains unchanged, or that the current action should actually be allowed. This distinction matters far more in modern infrastructure than it did in earlier generations of systems. Historically, possession-based trust models worked reasonably well because environments were more static. Workloads lived longer, infrastructure changed more slowly, and credentials moved through systems less pervasively. Authentication and runtime behavior were often closely coupled simply because the environments themselves were comparatively predictable. Modern infrastructure looks very different. Workloads are ephemeral by default, APIs continuously communicate with other APIs, identities cross organizational and cloud boundaries, and credentials move automatically between systems at machine speed. Runtime context changes constantly, yet many systems still make authorization decisions primarily based on possession of transferable artifacts. This is one of the reasons credential theft remains so effective even in highly modern environments. The problem is usually not that the credential itself is invalid. In many incidents, the credential remains perfectly legitimate from a cryptographic perspective. It may be correctly signed, issued by a trusted authority, short-lived, and fully compliant with modern identity standards. The failure occurs because token validity alone says very little about the legitimacy of the current actor presenting it. The recent [Vercel security incident](https://vercel.com/kb/bulletin/vercel-april-2026-security-incident) is a useful example of this broader pattern. Although the specific technical details vary across environments, the underlying issue remains familiar: once valid credentials are exposed, replayed, or reused outside their original runtime context, infrastructure built primarily around possession-based trust tends to treat the actor presenting them as legitimate. At that point, the infrastructure gradually loses the ability to distinguish between the original workload, a compromised process, malware operating within the workload, or an attacker replaying credentials elsewhere. The infrastructure validates the credential itself, but not necessarily the runtime legitimacy of the current actor using it. This distinction becomes increasingly important as the industry shifts from workload identity toward workload access management. The realization that authentication alone is insufficient is directionally correct, but many modern access systems still fundamentally rely on transferable credentials underneath increasingly dynamic policy systems. The architecture typically evolves into something resembling: - authenticate workload - issue credential - attach policies - evaluate access dynamically While this improves flexibility and enables richer authorization decisions, it does not fully solve the core problem if access decisions remain detached from runtime execution itself. The critical question is no longer whether a workload authenticated successfully at some earlier point in time. The more important question is whether the current runtime context still justifies access at the exact moment an action occurs. Those are very different guarantees. Many authorization systems evaluate policies in components that operate adjacent to execution rather than directly observing it. Proxies, SDKs, service meshes, and centralized policy engines are all capable of coordinating sophisticated authorization workflows, but they often do not directly observe the thing that matters most: which process is actually establishing the connection right now, under what runtime conditions, and whether the execution context still matches the original trust assumptions. Once runtime context becomes detached from credential validation, authorization gradually degrades back into possession-based trust again. The infrastructure is no longer continuously validating the actual workload, execution state, or legitimacy of the current action. Instead, it validates possession of something issued earlier and assumes the original trust relationship still holds. This problem becomes even more pronounced in federated environments. As discussed previously in [SPIFFE Identity Federation: Extending Trust Across Boundaries](https://blog.riptides.io/spiffe-identity-federation-extending-trust-across-boundaries/), federation solves trust propagation but does not inherently solve runtime enforcement. Extending trust across boundaries without continuously validating runtime legitimacy can significantly increase the blast radius of transferable credentials. That observation led directly to another earlier post, [Federation Is Easy. Runtime Enforcement Is Hard.](https://blog.riptides.io/federation-is-easy-runtime-enforcement-is-hard/), which focused on the growing gap between establishing trust and enforcing trust during actual runtime communication. This distinction is ultimately what shaped the architectural direction behind Riptides. Instead of treating token validity alone as sufficient proof for access, Riptides continuously binds identity and authorization decisions to the actual running workload itself. Enforcement does not rely solely on portable artifacts copied into memory or assumptions made by adjacent infrastructure layers. Instead, authorization decisions are evaluated directly at the point where communication occurs, at the boundary between the process and the network. This changes the security model substantially. The system can continuously evaluate which process is initiating the connection, what identity is attached to it, where it is attempting to connect, and whether the current runtime context still satisfies the required trust assumptions. Most importantly, connections can be blocked immediately when those assumptions no longer hold. This is fundamentally different from validating a credential once and assuming the resulting trust relationship remains valid indefinitely. As infrastructure becomes increasingly dynamic, identity alone is no longer enough. Valid credentials still matter, but runtime legitimacy matters more. Modern systems need to continuously validate not only whether a credential is valid, but whether the current actor presenting it should still be trusted in the current execution context. A valid token should not automatically mean access. If these problems resonate with how you think about workload security, non-human identity, and runtime enforcement, give Riptides a try. We would love to show you what runtime-bound identity and real-time enforcement look like in practice. --- ## The Quantum Threat to Workload Identity — And Why It Starts Today - URL: https://blog.riptides.io/the-quantum-threat-to-workload-identity-and-why-it-starts-today - Published: 2026-04-22 - Author: Nandor Kracser - Category: Security - Tags: pqc, cryptography, tls, quantum *The follow-up we promised: the harvest-now, decrypt-later attack pattern, what the NIST PQC timeline means for infrastructure teams, and where Riptides stands.* --- Someone may be recording your mTLS traffic right now. Not to read it today. To read it later. No exploit. No alert. No active interference. Just a passive tap: a compromised hypervisor, a rogue switch port, a network device at an internet exchange, silently capturing TLS handshakes and storing them cheaply until a cryptographically relevant quantum computer exists. At that point, the math changes, and every session recorded today becomes readable retroactively. This is Harvest Now, Decrypt Later, and it is the most underappreciated threat in workload identity security. Not because it is subtle (it is not), but because it creates urgency before any quantum computer capable of executing it actually exists. ## What a Quantum Computer Actually Is The short version: a classical computer solves a problem by trying possibilities sequentially. A quantum computer can, for certain specific types of problems, explore many possibilities at once and collapse to the right answer. This is not useful for most computing tasks — but for the math underlying encryption, it is a structural shortcut that breaks the assumptions everything else is built on. Quantum computers are not faster classical machines, and they are not related to AI. They are a fundamentally different category of hardware, based on different physics. A classical computer stores information as bits: transistors that are either on or off. A quantum computer stores information as qubits, which exploit superposition: a qubit exists as 0 and 1 simultaneously until measured. Carefully designed algorithms can use this property to cancel out wrong answers and amplify correct ones, a process called interference. This is not brute force. It is a structural shortcut through specific classes of mathematical problems. The algorithm that matters here is Shor's algorithm, published in 1994. It can factor large integers and solve discrete logarithms, the mathematical foundations of RSA and elliptic curve cryptography, exponentially faster than any classical machine. A cryptographically relevant quantum computer running Shor's algorithm could recover a private key from its public key in hours. That machine does not exist yet. Google's Willow chip, released in late 2024, has 105 physical qubits. Breaking RSA-2048 requires approximately 20 million physical qubits running with error correction. We are roughly five orders of magnitude away from a working attack. The Global Risk Institute's 2025 survey of leading quantum researchers places a 28–49% probability on a cryptographically relevant machine arriving within 10 years, and 51–70% within 15. In practical terms: we are still a very long way from a working attack, but credible researchers place it in the range of a decade or two — well within the lifetime of infrastructure decisions made today. The timeline is uncertain. The direction is not. ## Why Identity Infrastructure Is the High-Value Target Generic HTTPS traffic, when decrypted retroactively, leaks user sessions. Harvested mTLS traffic between internal workloads leaks something more durable: the shape of your infrastructure itself. A captured mTLS handshake reveals the SPIFFE IDs of both communicating workloads: their identities, their roles, who talks to whom. It exposes the content of internal API calls: authentication flows, secrets requests, RPC payloads. It reveals your CA hierarchy and certificate issuance patterns. This is not session data. It is architectural intelligence about how your systems are built, and it remains useful long after the specific sessions have ended. The sensitivity lifetime of that information is measured in years. If a cryptographically relevant quantum computer arrives in eight years, and your internal service topology today reflects your service topology in eight years (which for most organizations it largely will), then the traffic being captured now is genuinely useful to whoever holds it when Q-Day arrives. This is the calculation that changes how urgency should be framed: **the deadline for acting is not when a quantum computer can decrypt your traffic. It is today, because every unprotected handshake is a permanent record.** ## Two Separate Problems in TLS It helps to be precise about what, specifically, is vulnerable. A TLS 1.3 handshake involves two distinct asymmetric operations, and they carry different risk profiles. **Key exchange** establishes the session key, the symmetric key used to encrypt the actual connection. In standard TLS 1.3, this is ECDHE: an ephemeral elliptic curve Diffie-Hellman exchange. The session key is derived from public keys visible in the handshake transcript. A quantum attacker with a recording of that handshake can run Shor's algorithm against the captured ECDHE public key, recover the session key, and decrypt every record. This is the HNDL-critical layer. **Certificate signatures** authenticate the parties by proving that the certificate presented matches the private key held by the peer. Breaking this requires running Shor's algorithm in real time during an active connection, or forging certificates after compromising a CA key. This is a real threat, but it does not enable retroactive decryption of stored traffic. The urgency is lower. The practical implication: fixing key exchange first is the right priority. That protects against the attack that is executable today: passive collection for future decryption. Certificate signature migration can follow as the ecosystem matures, and it will need to. ## Where the Ecosystem Stands The good news is that the critical layer, key exchange, is moving fast. Go 1.24, released in February 2025, enabled X25519MLKEM768 hybrid key exchange by default in `crypto/tls`. Any Go service running 1.24 or later, without explicit overrides to `CurvePreferences`, is already negotiating post-quantum key exchange with compatible peers. Chrome, Firefox, and OpenSSL 3.5 all ship ML-KEM hybrid support by default. A meaningful fraction of internet traffic is already post-quantum protected at the key exchange layer. The certificate signature layer is further behind. ML-DSA, the NIST-standardized post-quantum signature algorithm, landed as an internal implementation in Go 1.26. A public API is proposed for Go 1.27, but X.509 and `crypto/tls` integration is a further step, targeted at Go 1.28 or later. Production CA support for ML-DSA certificates is expected from the CA/Browser Forum in 2026–2027. SPIRE, the reference SPIFFE implementation, has no public roadmap for ML-DSA SVID issuance yet. **Update (September 2026):** the language side moved faster than this paragraph predicted. Go 1.27 ships a public `crypto/mldsa` package, and `crypto/tls` now advertises ML-DSA-44/65/87 among its default signature algorithms — the integration expected in 1.28 or later arrived early. What has *not* moved is issuance: getting post-quantum signatures onto real workload certificates still waits on the CA side, including SPIRE. The migration will happen in two phases, whether organizations plan for it or not. The question is whether you are ahead of the collection window or behind it. ## Where Riptides Stands The short version: the part of the connection that protects data in transit is already upgraded. The part that verifies identities is still catching up across the whole industry, and we are tracking it. In our [previous post](/blog/tls-13-for-internal-connections/), we upgraded the Riptides kernel module to TLS 1.3 and shipped opt-in PQC hybrid mode (X25519 combined with ML-KEM-768) for internal mTLS handshakes. That was the key exchange layer. **Update (September 2026):** the hybrid handshake is no longer opt-in. The kernel module offers `X25519MLKEM768` as its first key exchange group on every handshake it performs — internal mTLS and intercepted egress toward external servers alike — with no flag to set and no per-service rollout. The fuller picture across our stack: **The record layer is not a quantum problem.** Riptides uses kTLS for record encryption, with AES-256-GCM. Symmetric ciphers are not broken by quantum computers in any practical sense. Grover's algorithm provides only a quadratic speedup against brute force search, leaving AES-256 with an effective 128-bit security level that is computationally infeasible to attack. kTLS does not need to change. **The Go daemon and control plane negotiate PQC key exchange automatically** on Go 1.24+, as long as `CurvePreferences` is not explicitly overridden. We have audited our gRPC credential setup to verify this. Go 1.26 adds SecP256r1MLKEM768 to the defaults as well. **The kernel module handshake layer is where active work continues.** The TLS 1.3 upgrade replaced BearSSL (TLS 1.2 only, with no PQC path) with a library that supports both TLS 1.3 and ML-KEM hybrid key exchange. The hybrid handshake ships today, on by default. Since this post was published we also replaced the ML-KEM implementation underneath it. The TLS library's own post-quantum support is a shim over a userspace library, which cannot link or run in kernel context and was never production-grade for our purposes. In its place the module now carries [mlkem-native](https://github.com/pq-code-package/mlkem-native), the Post-Quantum Cryptography Alliance's C implementation of FIPS 203, whose C sources are machine-proved memory-safe and type-safe with CBMC. It runs entirely in kernel space, draws its randomness from the same kernel-seeded PRNG as the rest of the module, and keeps ML-KEM's large intermediate buffers off the 16 KB kernel stack. So the "experimental library" caveat from the original version of this post no longer applies to the key exchange. What remains is authentication, and the gap is narrower than it was. ML-DSA has landed in mainstream TLS stacks, but not in one that runs in kernel space: the in-kernel TLS library has no ML-DSA path, so the module cannot present or verify a post-quantum certificate signature today. Issuance has not moved either — nothing is signing post-quantum workload certificates yet, SPIRE included. We are tracking both, and will extend as they land. This is not a solved problem. It is work in progress, done in the open, with an honest assessment of what is covered and what is not. ## Check Your TLS Posture Right Now Before planning any migration, you need to know where you actually stand. That starts with understanding what your clients are advertising in their ClientHello. We built [safetls.riptides.io](https://safetls.riptides.io) to answer that question directly. Visit it from any browser or HTTP client and it inspects the TLS handshake your client just performed, returning a full analysis of the TLS versions offered, cipher suites advertised, key exchange groups including whether any post-quantum groups were offered, and a security grade. No configuration, no agent, no signup. For a browser check, just open the URL. If you're not a command-line user, that's all you need — you'll see a letter grade and a plain-English summary of your client's TLS posture. For programmatic inspection from any HTTP client: ```bash curl -s https://safetls.riptides.io/api/inspect | jq .observation.grade.rating ``` The response tells you your client's rating (A+ through F). For the full breakdown including score, findings, and recommendations: ```bash curl -s https://safetls.riptides.io/api/inspect | jq .observation.grade ``` ```json { "score": 85, "rating": "B", "summary": "Modern TLS client with a strong baseline, but no post-quantum hybrid key share was observed.", "post_quantum_offered": false, "findings": [ { "severity": "warning", "code": "static-rsa-key-exchange", "message": "Static RSA cipher suites are offered, which means those connections lack forward secrecy." }, { "severity": "info", "code": "cbc-cipher-suite-compat", "message": "CBC-based cipher suites are offered for compatibility, but TLS 1.3 is also present." }, { "severity": "info", "code": "pqc-hybrid-missing", "message": "No post-quantum hybrid key share was observed." } ], "recommendations": [ "Prefer ECDHE-based TLS 1.2 suites and TLS 1.3 suites only.", "Favor AEAD suites such as AES-GCM or ChaCha20-Poly1305.", "Add a hybrid key exchange such as X25519MLKEM768 where your TLS stack supports it." ] } ``` One important caveat: if you are behind a corporate TLS proxy that intercepts and re-establishes connections, you will see the proxy's ClientHello, not your own. That is itself useful information about your network posture. ## What Infrastructure Teams Should Do Now Three concrete actions that do not require waiting for anything. **Audit where TLS 1.2 still runs.** Older TLS versions cannot be made quantum-safe — there is no upgrade path for them, only replacement. TLS 1.2 has no post-quantum migration path. The IETF will not define PQC cipher suites for it. Any service speaking TLS 1.2 today is categorically excluded from a quantum-safe future, and every handshake it performs is harvestable. **Verify your Go TLS defaults are intact.** Go 1.24 shipped post-quantum protection on by default, but certain configuration patterns silently disable it — and most teams don't realize. If you are on Go 1.24+ and `CurvePreferences` is nil in your TLS configuration, you are already negotiating hybrid PQC key exchange with compatible peers. If `CurvePreferences` is set explicitly, or if you are using gRPC convenience helpers like `NewClientTLSFromCert`, that protection may be silently disabled. Use the API above to verify before assuming. **Start with the traffic where you control both ends.** If you own both sides of a connection — internal services talking to each other — you don't need to wait for industry standards to catch up. Where Riptides is in the path, those handshakes already come up hybrid; there is no mode to enable. The overhead is measurable only at very high connection rates with no reuse; for normal internal RPC patterns it is negligible. The quantum computer that can decrypt your traffic does not exist today. The adversary collecting your traffic in case it eventually does is already operational. --- *Questions about post-quantum key exchange or the TLS 1.3 rollout? Reach out at [riptides.io](https://riptides.io) or [get in touch for a demo](https://riptides.io/request-a-demo).* *If you found this useful, follow us on [LinkedIn](https://www.linkedin.com/company/riptidesio/) and [X](https://x.com/riptidesio) for more updates.* --- ## SPIFFE Is What AI Agents Need for Identity, The Question Is How to Deliver It - URL: https://blog.riptides.io/how-to-deliver-spiffe-identity-to-ai-agents - Published: 2026-04-20 - Author: Janos Matyas - Category: SPIFFE - Tags: SPIFFE, Oauth2, AI, OIDC, MCP, Agentic ## Why SPIFFE Is Non-Negotiable for Agent Identity — and Why SPIRE Was Never Built to Deliver It If you're building or operating AI agents in production, you already know the identity problem is real. Your agent connects to MCP servers, calls external APIs, handles OAuth tokens, and chains tools together, all autonomously. Every one of those interactions needs authentication. Every one of those interactions is a credential that can be stolen. SPIFFE solves the identity part. It gives your agent a cryptographically verifiable identity — an SVID — that's ephemeral, rotatable, and doesn't rely on static secrets. That's exactly what agents need. But _how_ you deliver that SPIFFE identity, and what you do with it after issuance, matters more than most people realize. And if you're reaching for SPIRE — SPIFFE's reference implementation — you're about to hit walls on both fronts. >Note: There are two distinct identity problems when you run AI agents in production. The first is the agent's own identity — a *non-human, workload-level identity* that proves which process is making a request. That's what SPIFFE solves. The second is the user's identity — the human on whose behalf the agent is acting. When your agent calls an MCP server or a cloud API, it needs a credential that represents the user's authorization, not just its own. These are separate trust relationships and they need separate mechanisms: SPIFFE SVIDs for the agent, OAuth/delegated credentials for the user, **both managed without secrets ever touching the agent's memory**. For further reference check out or latest post on the topic, [Securing Agentic OAuth Flows with Riptides](https://riptides.io/blog/mcp-riptides-oauth/). ## SPIRE: A Reference Implementation, Not a Security Platform Before we talk about agents, let's be precise about what SPIRE is and isn't. SPIFFE is a specification. It defines the identity model — SPIFFE IDs, SVIDs, trust domains — but says nothing about how those identities should be issued, enforced, or operationalized. SPIRE is the CNCF reference implementation of that spec. It was built to demonstrate that the SPIFFE model works: run a server, deploy node agents, attest workloads, issue SVIDs. And for that purpose, it succeeded. SPIRE brought SPIFFE to the mainstream and proved that workload identity at scale is achievable. But a reference implementation is designed to prove the spec works — not to be the production security platform you build on. The next question is: what does a purpose-built delivery mechanism look like, one that handles not just identity issuance, but policy, rotation, posture, and credential lifecycle as a single system? Even in traditional infrastructure — long before AI agents entered the picture — SPIRE leaves significant operational gaps that teams have to fill themselves: **Certificate lifecycle is the operator's problem.** SPIRE issues certificates, but what happens next is on you. Rotation logic has to be handled by the workload or its sidecar. If a certificate expires because the renewal failed or the agent was unreachable, the workload loses its identity. There's no built-in fallback, no automatic recovery, no centralized lifecycle management. At scale, this turns into an operational burden that grows with every workload you onboard. **No access policy enforcement.** SPIRE tells you _who_ a workload is. It doesn't tell you _what_ that workload is allowed to do. There is no policy engine, no authorization layer, no way to express "workload A can talk to service B on port 443 but not service C." Policy enforcement is left entirely to external systems — service meshes, OPA sidecars, application-level checks. The gap between "identity issued" and "access controlled" is filled by other tools, other configurations, and other failure modes. **No posture management.** SPIRE attests a workload at registration time. But what happens if the workload's binary changes after attestation? If its environment drifts? If its runtime characteristics no longer match the security posture you expect? SPIRE doesn't continuously verify. It doesn't re-attest. Once the SVID is issued, the workload is trusted for the duration of that certificate's validity, regardless of what it becomes. **No credential management beyond SVIDs.** Workloads don't just need SPIFFE identities. They need OAuth tokens, cloud provider credentials, API keys, database passwords. SPIRE has no mechanism for managing, injecting, or rotating these credentials. That entire surface area — which is where most real-world authentication happens — falls outside SPIRE's scope. **Secrets live in user space.** SPIRE delivers SVIDs to workloads through a gRPC Workload API, typically via a Unix domain socket. The workload (or its proxy) loads the certificate and private key into its own process memory. From that point forward, the private key is in user space — accessible to the process, to any exploit that achieves code execution in that context. These aren't edge cases. They're the everyday reality of operating SPIRE at scale. And they exist _before_ you introduce the complexity of AI agents. ## Now Add AI Agents — and SPIRE's Model Breaks Down Further If SPIRE already struggles as a complete security solution for traditional workloads, the AI agent world makes every limitation worse. **Agents are polyglot and framework-diverse.** Your agent might be a Python script using LangChain, a TypeScript process running on Vercel, a Go binary orchestrating sub-agents, or Claude Code running from a terminal. SPIRE's Workload API requires the workload to _participate_ — to call the API, load the certificate, and manage the TLS connection. That means code changes. That means language-specific libraries. In the agent ecosystem, where frameworks evolve weekly and there is no stable integration point, this is a non-starter. **Agents are ephemeral and spawn dynamically.** An agent orchestrator might spin up sub-agents on demand to handle tool calls, delegate context expansion, or parallelize tasks. SPIRE requires that every workload be pre-registered with the SPIRE server — a registration entry mapping a SPIFFE ID to a set of selectors — before the workload can be attested and issued an SVID. For dynamic sub-agents that are created on the fly, this means either pre-registering every possible agent variant ahead of time or building an external automation pipeline that races to create registration entries as processes spawn. On top of that, SPIRE's attestation is pull-based: the workload itself must actively call the Workload API over a Unix domain socket to request its SVID. A sub-agent that doesn't integrate the SPIRE SDK or client library simply never gets an identity. For short-lived processes, the overhead of calling the API, receiving the SVID, configuring TLS, and then performing actual work may outweigh the process's entire useful lifetime. **Agents don't live exclusively in Kubernetes.** Many agent deployments run on VMs, bare-metal dev boxes, edge nodes, or as local CLI tools. SPIRE's operational model leans heavily on Kubernetes primitives for workload registration, sidecar injection, and service account-based attestation. If your agent runs outside Kubernetes — and many do — you're fighting the deployment model, not leveraging it. **Agents chain tools across trust boundaries.** A single agent session might hit a Cloudflare MCP server, call an AWS Bedrock endpoint, query a GCP Vertex API, and write to an internal database. Each hop requires a different credential type. SPIRE issues SVIDs, but the agent still needs to exchange those SVIDs for OAuth tokens, AWS SigV4 signatures, or GCP access tokens. That exchange logic has to live somewhere, and in SPIRE's model, it lives in your code or in yet another sidecar. ## The Sidecar Trap If you can't embed the SPIFFE logic in the agent, the next move is usually a sidecar or proxy — Envoy with SDS, a custom mTLS proxy, or a service mesh that handles certificate rotation and TLS termination on the agent's behalf. Here's why it falls apart for agents: **Identity confusion.** When a sidecar proxy handles TLS, the SVID belongs to the proxy process, not the agent process. The MCP server on the other end sees the proxy's identity, not the agent's. If two agents share a node and a proxy, the identity fidelity degrades further. You've introduced a layer of indirection between "who is making this request" and "who authenticated this request." In the agent world, where processes are dynamic and potentially untrusted — think prompt injection leading to rogue tool calls — this ambiguity is a security gap. **Operational weight.** Every agent process now needs a companion proxy. That's double the processes, double the memory, double the failure modes. For a team running hundreds of agent instances across mixed infrastructure, this overhead is a real operational and cost burden. **Deployment coupling.** Sidecar injection in Kubernetes is well-understood. Sidecar injection on a developer's laptop, a VM running Claude Code, or an edge device running a local agent? That's a custom problem every time. **The credential problem doesn't go away.** Even with a sidecar handling mTLS, the agent still needs to manage OAuth access tokens, API keys, and session credentials for third-party services. The sidecar terminates TLS but it doesn't solve the credential lifecycle problem. Tokens still end up in the agent's memory, in environment variables, in config files — plaintext, replayable, and one prompt injection away from exfiltration. ## What Agents Actually Need Let's reframe the requirements from the ground up. An AI agent needs a cryptographic identity that is bound to its process — not to a pod, not to a node, not to a proxy sitting next to it. That identity needs to be issued automatically when the process starts, rotated transparently, and revoked when the process ends. The agent code should not need to know about SPIFFE, SVIDs, certificate rotation, or TLS handshakes. But identity alone is not enough. The agent also needs access policies that govern which services it can reach and under what conditions. It needs continuous posture verification — not just a one-time attestation at boot, but ongoing validation that the agent process is what it claims to be. It needs credential management for every external service it touches — OAuth tokens, cloud credentials, API keys — without those credentials ever landing in the agent's memory. And it needs all of this managed centrally, rotated automatically, and enforced at a layer the agent can't tamper with. SPIRE delivers the first piece — identity issuance — and leaves the rest to you. That's not a security solution. That's a starting point. ## The Kernel as the Identity and Enforcement Plane There's one layer that every agent process touches, regardless of language, framework, deployment model, or orchestrator: the Linux kernel. Every `connect()` syscall, every socket opened, every TCP handshake — it all goes through the kernel. That makes the kernel the natural enforcement point for workload identity, access policy, and credential management. This is the approach Riptides takes. But it's important to understand what Riptides is and isn't: **Riptides is not just a SPIFFE provider.** We use and love SPIFFE as the identity standard — it's the right abstraction, and we issue SPIFFE-compliant SVIDs. But identity issuance is the foundation, not the product. What Riptides delivers is a complete security platform built on top of that foundation. ### Identity: SPIFFE in the Kernel Riptides issues SPIFFE-compliant X.509 certificates directly inside the Linux kernel, bound to the actual process initiating communication. Using kernel TLS (kTLS), the SVID is injected into the TLS handshake at the record layer. The application — your agent — doesn't participate. It doesn't load certificates. It doesn't call a Workload API. It just opens a connection, and the kernel handles the rest. **Per-process identity binding.** The identity is tied to the specific process, identified at the syscall level. Not to a pod, not to a node, not to a proxy. If an agent spawns a sub-process, that sub-process gets its own identity based on its own attestation. If a rogue process tries to impersonate an agent, it can't — the kernel module verifies process attributes before injecting any credential. **Zero code changes.** Your Python LangChain agent, your TypeScript MCP client, your Go orchestrator, Claude Code — none of them need to know about SPIFFE. There are no SDKs to embed, no libraries to import, no gRPC calls to make. **No secrets in user space.** Private keys never leave kernel memory. SVIDs are not written to disk, not exposed via filesystem mounts, not passed through environment variables. **Works everywhere Linux runs.** Kubernetes, VMs, bare metal, developer workstations, edge devices. The deployment model is the same everywhere. ### Automatic Certificate Lifecycle Unlike SPIRE, where rotation is the workload's responsibility, Riptides manages the entire certificate lifecycle from the control plane. Certificates are issued, rotated, and revoked automatically. The agent process never participates in renewal. There are no expiration windows to race against, no renewal failures to debug, no zombie workloads running with stale certificates. The control plane tracks certificate state, and the kernel module receives updated credentials through a secure channel — transparently, continuously, without downtime. ### Access Policy Enforcement Riptides doesn't just tell you who a workload is. It controls what that workload can do. Access policies are defined centrally and enforced at the kernel level, at connection time. When an agent process opens a socket to a destination, the kernel module checks the agent's identity against the configured policy before the connection is established. If the policy says this agent can't reach that service, the connection is denied — not by a proxy, not by a sidecar, not by an application-level check, but by the kernel itself. This is enforcement that the agent process cannot bypass, cannot misconfigure, and cannot be prompted into circumventing. ### Continuous Posture Management Riptides doesn't attest a workload once and trust it forever. The attestation pipeline continuously collects process-level evidence — binary hashes, environment metadata, runtime characteristics, node posture — and validates it against expected baselines. If an agent's posture drifts — if its binary changes, if its environment is tampered with, if its runtime behavior deviates — its identity can be revoked or its access restricted in real time. In the age of AI agents, where prompt injection can alter an agent's behavior without changing its process signature, continuous posture verification is not optional. It's the difference between "this process was trustworthy when it started" and "this process is trustworthy right now." ### Secretless Credential Injection Here's where the kernel-based model goes beyond anything SPIRE or sidecar architectures can offer. When your agent connects to a remote MCP server that requires OAuth, it goes through a standard OAuth2 authorization code flow. But with Riptides, the agent never holds the real access token. Instead, Riptides acts as an intermediate authorization server, brokers the real OAuth flow on the agent's behalf, stores the actual access token in the kernel, and issues the agent a Riptides JWT — a transient credential that is meaningless outside the system. When the agent sends a request to the MCP server with that JWT in the Authorization header, the kernel module intercepts the outgoing request, verifies the process identity matches, looks up the real credential, and swaps the JWT for the actual access token before the packet leaves the machine. The MCP server sees a fully authenticated request. The agent never possessed the credential that made it possible. This same pattern applies to AWS credentials, GCP tokens, Azure service principal secrets, and any other credential type. The kernel module does the swap on the wire. The agent operates in a secretless model — it authenticates, it participates in authorization flows, but it never holds the keys to the kingdom. An agent that can be prompted into exfiltrating its own credentials cannot exfiltrate credentials it does not have. ## The Full Picture: SPIRE vs. Riptides for AI Agents | Capability | SPIRE | Riptides | |---|---|---| | SPIFFE identity issuance | Yes (user space) | Yes (kernel space) | | Zero code changes | No (Workload API or sidecar required) | Yes | | Per-process identity binding | Limited (at SVID issuance time) | Yes (continuous, syscall-level) | | Automatic certificate rotation | Partial (workload must participate) | Full (kernel + control plane) | | Access policy enforcement | No (external tools required) | Yes (kernel-level, per-connection) | | Continuous posture management | No | Yes | | Credential injection (OAuth, cloud, API keys) | No | Yes (on-the-wire, kernel-level) | | Secrets in user space | Yes | No | | Works outside Kubernetes | Limited | Yes (anywhere Linux runs) | | Agent framework agnostic | No | Yes | SPIRE is a reference implementation that demonstrates SPIFFE identity issuance. Riptides is a security platform that uses SPIFFE as its identity foundation and builds everything else — policy, posture, credential management, and enforcement — on top of it, at the kernel level. ## What This Looks Like in Practice From an operator’s perspective, this is what’s happening: - The agent starts. No code changes. - Riptides assigns a verified runtime identity to the process. - The agent connects to an external service (for example an MCP server). - OAuth happens on behalf of the user, without exposing tokens to the agent. - Every connection is verified and enforced at runtime. - Certificates rotate without intervention. - Posture is verified continuously. No sidecars. No SDKs. No credentials in application space. ## The Bottom Line SPIFFE is the right identity standard for AI agents. It's ephemeral, cryptographic, and doesn't rely on static secrets — exactly what you need for autonomous workloads that spawn dynamically and chain tools across trust boundaries. But SPIRE — the reference implementation — was never designed to be a complete security solution, even for traditional workloads. It issues identities and leaves everything else — policy, rotation, posture, credential management, enforcement — to the operator. For AI agents, where the workload is polyglot, ephemeral, framework-diverse, and operating across trust boundaries, SPIRE's operational model doesn't just fall short. It doesn't apply. What agents need is not a better SPIFFE provider. They need a security platform that uses SPIFFE as the identity foundation and builds policy enforcement, continuous attestation, automatic credential lifecycle, and secretless credential injection on top of it — at a layer the agent cannot bypass and does not need to know about. That layer is the kernel. And that platform is Riptides. --- *If you want to see kernel-level agent identity in action, [get in touch for a demo](https://riptides.io/request-a-demo). Follow us on [LinkedIn](https://www.linkedin.com/company/riptidesio/) and [X](https://x.com/riptidesio) for more.* --- ## Upgrading Riptides to TLS 1.3: Forward Secrecy and a Path to Post-Quantum mTLS - URL: https://blog.riptides.io/tls-13-for-internal-connections - Published: 2026-04-15 - Author: Nandor Kracser - Category: Security - Tags: spiffe, mtls, tls13, pqc, identity, zero-trust, security *How we upgraded the Riptides kernel module from TLS 1.2 to TLS 1.3, and why that change is the foundation for everything we are building toward.* Most internal mTLS deployments were not designed to be wrong. They were designed for the constraints of the time: TLS 1.2, a negotiated cipher suite, certificates from an internal CA with comfortable lifetimes. That is a defensible baseline, and for years it was good enough. It is no longer good enough — and the problem is not dramatic. It is structural. TLS 1.2 does not mandate forward secrecy. It allows cipher suite combinations that have known weaknesses. It carries a negotiation surface large enough that misconfiguration is common and often undetected. And its handshake structure was not designed with the cryptographic landscape of 2026 in mind. We have upgraded the Riptides kernel module to TLS 1.3. This post explains what changed, why each change matters for internal workload traffic specifically, and what this upgrade unlocks going forward. ## Why Internal mTLS Needed This There is a reasonable assumption that internal east-west traffic is lower risk than public-facing TLS. It never leaves the cluster, the VPC, the datacenter. But this assumption breaks down in ways that matter. **Compromised infrastructure components.** A rogue node, a misconfigured network tap, a compromised CNI plugin — internal traffic can be captured at the hypervisor or switch layer, not at the perimeter. The capture happens silently, long before anyone raises an alert. **Supply chain persistence.** An adversary who lands in your environment via a compromised build artifact does not need to exfiltrate data immediately. A low-volume, low-frequency tap on internal handshake transcripts is operationally quiet and extremely valuable. The services most worth targeting — authentication, secrets management, inter-service RPC — are precisely the ones running on long-lived connections with rarely rotated credentials. **Long-lived credential exposure.** TLS 1.2 does not enforce forward secrecy. A session key derived from a static RSA certificate is retroactively recoverable if the certificate is later compromised. For internal services with multi-year certificate lifetimes — which describes most environments that have not adopted automated rotation — every recorded session is a liability. The question is not whether internal traffic is harder to reach than perimeter traffic. It is whether the information in that traffic is worth the effort for an adversary who is patient and well-resourced. For authentication and RPC traffic, the answer is clearly yes. ## What Changed in the Kernel Module Riptides performs TLS interception and enforcement at the kernel level via kTLS. The kernel module handles the handshake; the userspace daemon validates the SPIFFE SVID presented by the peer; the connection is accepted or torn down before any application bytes flow. No changes are required to application code. Until this release, the kernel module's TLS implementation was limited to TLS 1.2. That constraint was a function of the underlying TLS library, which had no TLS 1.3 support and no active development path toward it. We replaced that library. Without going into implementation specifics, the selection came down to a concrete set of requirements: active maintenance posture, production-grade TLS 1.3 support, and a memory footprint compatible with kernel context. **We have upgraded the kernel module to TLS 1.3.** TLS 1.3 is not an incremental improvement over 1.2. The protocol was rearchitected. The changes that matter most for internal workload traffic: **Forward secrecy is mandatory.** Every session uses an ephemeral key exchange. There is no code path in TLS 1.3 that allows a static certificate key to decrypt recorded traffic. This eliminates an entire class of retroactive exposure that TLS 1.2 leaves open. **The cipher suite surface is dramatically reduced.** TLS 1.3 ships with five cipher suites, all of which provide authenticated encryption (AEAD). The negotiation attack surface that plagued TLS 1.2 deployments — BEAST, POODLE, FREAK, and the long tail of downgrade attacks — structurally does not exist. **Fewer round trips.** The 1-RTT handshake reduces latency for short-lived connections. For high-frequency internal RPC patterns with connection reuse, the reduction per connection is small but accumulates at scale. **A clean extension model.** The `key_share` extension in TLS 1.3 is the designed integration point for new key exchange algorithms. This is not incidental — it is what makes the next step possible. ## Post-Quantum mTLS: The First Use of the New Foundation The TLS 1.3 upgrade is a prerequisite, not a destination. The first capability it enables is automatic post-quantum mTLS for internal workloads. The kernel module negotiates a hybrid key exchange on every mTLS handshake: X25519 for classical security, ML-KEM-768 for post-quantum security. The session key is derived from both; neither alone is sufficient to reconstruct it. **Update (September 2026):** this post originally described the hybrid handshake as an opt-in `pqc_hybrid` setting. It is no longer a setting — the module offers `X25519MLKEM768` as its first key exchange group on every handshake it performs, and there is nothing to configure. No application changes are required. SPIFFE SVID validation, workload attestation, and rotation policy are all unchanged. If a peer does not support the hybrid group — a legacy endpoint, a third-party service — Riptides falls back to X25519 only and logs the event. The threat model this addresses, and the full reasoning behind the hybrid construction and algorithm selection, deserves its own treatment. We will cover that in a follow-up post — including the harvest-now, decrypt-later attack pattern and what the NIST PQC standardization timeline means for infrastructure teams making decisions today. ## Deployment Considerations **TLS 1.3 is not a setting either.** For transparent mTLS between Riptides workloads the internal connection is always TLS 1.3 — the module's server side accepts nothing older. Where Riptides intercepts an outbound TLS session toward an external server, the remote end still decides, and the telemetry pipeline reports the negotiated version per connection — a useful audit of which destinations are holding you on TLS 1.2. **The hybrid key share costs bytes, not configuration.** ML-KEM-768 adds roughly 1.1 KB to the handshake versus ~32 bytes for X25519, which is measurable for very high connection-rate workloads with no connection reuse. For most internal RPC patterns the impact is negligible, and it is the price of closing the harvest window — the exchange is on for every handshake. **Certificate rotation is independent.** The TLS 1.3 upgrade and the hybrid key exchange both protect session keys. They do not change SVID validity windows. If you are running 90-day or longer SVID lifetimes, that is a separate and worthwhile conversation — short lifetimes matter regardless of what the key exchange is doing. ## The Right Layer to Fix This The alternative to solving TLS hygiene at the kernel module layer is asking every service team to instrument their TLS stack independently. That produces patchy coverage, inconsistent cipher suite enforcement, and no visibility into which workloads have and have not been upgraded. Riptides covers every managed workload from one place, with no per-service TLS configuration at all. TLS 1.3 is the baseline. Post-quantum key exchange builds on it. Both are available now. *Questions about the TLS 1.3 rollout or post-quantum key exchange? Reach out at [riptides.io](https://riptides.io) or [get in touch for a demo](/request-a-demo).* --- ## Securing Agentic OAuth Flows with Riptides - URL: https://blog.riptides.io/mcp-riptides-oauth - Published: 2026-04-08 - Author: Zsolt Rappi - Category: MCP - Tags: oauth, credentials, mcp, agentic In our [previous post on SPIFFE-backed OAuth for MCP](/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads), we showed how workloads running inside a Riptides environment can use their SPIFFE identity to self-register and authenticate with OAuth, eliminating client secrets entirely. That story assumed a world where both the agent and the MCP server are managed by Riptides. The reality is that most remote MCP servers today are third-party services: productivity tools, data providers, SaaS platforms. They run their own OAuth2 authorization servers, and agents connecting to them must go through a standard OAuth2 authorization code flow, authenticate the user, and store the resulting access token. That last part is the problem. Access tokens are sensitive credentials. They grant delegated access to a user's resources for as long as they remain valid. Yet in virtually every agentic framework today, these tokens end up stored as plaintext: in memory, in configuration files, in environment variables passed to subprocesses. The agent that holds the token is an attractive target. This post describes how Riptides solves this. We transparently broker the authentication on behalf of the agent and inject the real credential at the kernel level at request time. The agent participates in a fully standard OAuth2 flow, gets back a token it can store, and sends that token with every request, but the token it holds is a Riptides-issued JWT that is worthless outside our system. The real access token never leaves the kernel. **TL;DR:** When an agent connects to a remote MCP server, Riptides acts as an intermediate OAuth2 authorization server. The agent completes a standard OAuth flow and stores a Riptides-issued JWT. The real access token is brokered by Riptides, stored in the kernel, and injected into outgoing requests at the network level. The agent never holds the credential that actually grants access. ## How Remote MCP Servers Use OAuth2 [RFC 9728 — OAuth 2.0 Protected Resource Metadata](https://datatracker.ietf.org/doc/html/rfc9728/) defines a discovery mechanism that any OAuth-protected resource server can implement. A client discovers how to authenticate by fetching a well-known document: ``` GET /.well-known/oauth-protected-resource ``` The response tells the client which authorization server(s) protect this resource. The client then fetches the authorization server's own metadata from a second well-known endpoint ([RFC 8414](https://datatracker.ietf.org/doc/html/rfc8414)), discovers the authorization and token endpoints, and proceeds with the OAuth2 flow. For remote MCP servers, the [MCP specification](https://modelcontextprotocol.io/specification/2025-03-26/basic/authorization) requires this exact mechanism. An MCP client: 1. Fetches the protected resource metadata to discover the authorization server 2. Optionally registers itself as a client via [Dynamic Client Registration (RFC 7591)](https://datatracker.ietf.org/doc/html/rfc7591) 3. Runs the authorization code flow, redirecting the user to log in 4. Receives an access token and stores it for subsequent requests Steps 1–2 happen once per client registration. Step 3 happens once per user. Step 4 is the problem. The agent now has a token. ## The Problem with Tokens in Agent Memory Access tokens represent a user's delegated trust. If an agent is compromised (through prompt injection, a vulnerability in the framework, or a misconfigured process), the attacker inherits that trust. They can replay the token against the MCP server directly. Nothing in the token itself proves it came from the legitimate agent process. This is the same credential sprawl problem we see everywhere in non-human identity, in a new form. Long-lived API keys got replaced with OAuth tokens, but the storage pattern is identical: plaintext, in the workload's memory space, accessible to anything running as that process. The standard mitigations (short TTLs, refresh token rotation) reduce the window of exposure but don't eliminate it. A token that lives for an hour is still a token that can be stolen and replayed for an hour. ## Our Approach: Riptides as an Intermediate Authorization Server All the building blocks to solve this already existed in Riptides. We have [on-the-wire credential injection](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example), which replaces credentials in outgoing requests at the kernel level. We have a [control plane that is a fully-fledged OIDC server](/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads). We have process-level workload identity that knows exactly which agent process is making each request. The new piece is an intermediate OAuth2 flow that connects all three together. At a high level: the agent thinks it is talking to the MCP server's authorization server, but it is actually talking to Riptides. It intercepts the agent's OAuth discovery requests and rewrites the authorization server endpoints to point to the Riptides control plane. No changes to the agent or its configuration are required. Riptides brokers the real authorization flow on the agent's behalf, acquires the real access token, and stores it in the kernel. The agent receives a Riptides-issued JWT, a proxy credential that is only valid inside our system, which it stores and sends with every request. At request time, the kernel recognizes the JWT, selects the corresponding real credential, and replaces it before the traffic leaves the machine. The agent is a standard OAuth2 client throughout. It follows the spec. It just never sees the sensitive token. ![Auth flow diagram showing Claude authenticating via Riptides, which brokers the real OAuth flow to Cloudflare MCP and stores the access token in the kernel](../../assets/mcp-riptides-oauth/auth-flow.jpg) ## The Double Auth Flow, Step by Step Here is the full sequence in detail. ### 1. The agent initiates the OAuth flow When the agent begins the OAuth2 authorization code flow for the remote MCP server, it interacts with the Riptides control plane, which acts as the authorization server. The agent performs [Dynamic Client Registration (RFC 7591)](https://datatracker.ietf.org/doc/html/rfc7591) and, as a Riptides-managed workload, can use its SPIFFE identity as the registration credential via [SPIFFE-backed registration](/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads). No pre-shared client secrets are needed. ### 2. The user logs in to Riptides (first authentication) The user is redirected to Riptides' authorization endpoint and logs in. At this point, Riptides has assembled all the context it needs: - **Workload identity**: the SPIFFE ID of the agent process, established by the [kernel module at process start](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) - **User identity**: the authenticated user who just logged in - **Target resource**: the MCP server this agent is trying to reach ### 3. The real OAuth flow toward the MCP server (second authentication) Now that Riptides knows who the user is and which MCP server they want to reach, it starts the actual OAuth2 flow toward the MCP server's real authorization server, the one configured as a `CredentialSource` in the Riptides control plane. The control plane registers itself as a client with the real authorization server and redirects the user to authenticate there. The user logs in a second time, this time to the MCP server's own authorization server. Riptides does not silently impersonate the user. The MCP server's authorization server issues tokens for its own users, and the user must consent there directly. After the user authorizes, the MCP server's authorization server returns the access token to the Riptides control plane. This double login happens once per user. The agent receives a refresh token alongside the Riptides JWT and handles renewal through the standard OAuth2 refresh flow. The double login only recurs if the refresh token itself expires. ### 4. Credential storage and JWT issuance The control plane stores the access token as a credential and propagates it to the kernel module on the agent's host. This is the same credential lifecycle used for any other `CredentialSource` type in Riptides, whether that source is [AWS](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example), [GCP](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp), or [OCI](/blog/secretless-oci-authentication-with-spiffe-based-workload-identity). The `CredentialSource` tells the control plane how to retrieve the credential; the `CredentialBinding` tells the kernel where to inject it. If the MCP server's authorization server also issued a refresh token, Riptides stores that too and handles renewal transparently. When the access token expires, the control plane refreshes it and propagates the updated credential to the kernel without any action required from the agent or the user. The control plane then issues a short-lived Riptides JWT and returns it to the agent as the result of the authorization code flow. Here is what that token looks like at runtime: ```json { "sub": "spiffe://acme.corp/claude", "act": "spiffe://acme.corp/employee/acme.corp/alice", "aud": ["https://controlplane.riptides.io"], "client_id": "0ec59979-20cd-44a5-a3e4-715457539172", "iss": "https://controlplane.riptides.io/oauth2/.../oidc", "iat": 1775637390, "exp": 1775723790, "nbf": 1775637390, "jti": "85bc11fe-7a24-4542-bbdf-8a4a80c86719" } ``` The `sub` claim is the SPIFFE identity of the agent process. The `act` claim ([RFC 8693](https://datatracker.ietf.org/doc/html/rfc8693)) carries the SPIFFE identity of the user on whose behalf the agent is acting. The audience is scoped to the Riptides control plane, not the MCP server. This token is meaningless anywhere else. The agent stores this JWT. It looks like a normal access token. It is not. ### 5. Credential injection at request time When the agent sends a request to the MCP server, it includes the Riptides JWT in the `Authorization` header, exactly as it would with a real access token. The kernel module intercepts the outgoing request. It inspects the JWT, verifies the workload identity matches the process making the call, looks up the stored credential for that workload and user combination, and replaces the JWT with the real access token. The request arrives at the MCP server fully authenticated. The agent process never held the real token. The real token never appeared in user space. ![Token exchange diagram showing Claude presenting the Riptides JWT, which the kernel exchanges for the real user access token before the request reaches Cloudflare MCP](../../assets/mcp-riptides-oauth/token-exchange.jpg) ## Why the JWT Is Safe to Store A Riptides JWT has no value outside of the Riptides system. There is no endpoint at the MCP server or its authorization server that will accept it. An attacker who extracts the JWT from agent memory gains nothing; they cannot replay it against the MCP server directly. Inside the Riptides system, the JWT is still constrained. It is bound to a specific workload identity: only requests from the process with the matching SPIFFE ID will trigger credential injection. A JWT presented by a different process will not be honored. It is bound to a specific user: the credential lookup requires both the workload and the user to match. And it is short-lived: the JWT expires, and the agent must use its refresh token to obtain a new one. A full authorization flow is only required again if the refresh token itself has expired. This is the same security model that applies to every credential type Riptides manages. The `CredentialSource` retrieves the real credential; the `CredentialBinding` scopes where and for whom it gets injected. The workload participates in the authenticated flow, but it never possesses the credential that grants access. ## Configuration From an operator's perspective, three resources need to be defined. First, register the remote MCP server as a `Service`: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: cloudflare-mcp spec: addresses: - address: mcp.cloudflare.com port: 443 labels: app: cloudflare-mcp external: true ``` Second, configure the MCP server's authorization server as a `CredentialSource`. This tells the Riptides control plane where to run the real OAuth flow: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialSource metadata: name: cloudflare-mcp namespace: riptides-system spec: oa2ac: authorizationEndpointUrl: https://mcp.cloudflare.com/authorize tokenEndpointUrl: https://mcp.cloudflare.com/token registrationEndpointUrl: https://mcp.cloudflare.com/register usePkce: true ``` Third, create a `CredentialBinding` that ties a specific workload and user together. This tells Riptides which agent should have its outgoing requests injected with the credential, and on behalf of which user: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialBinding metadata: name: claude-cloudflare-mcp-alice spec: credentialSource: cloudflare-mcp workloadID: claude humanID: spiffe://acme.corp/employee/acme.corp/alice propagation: injection: selectors: - app: cloudflare-mcp ``` The `humanID` field is a SPIFFE ID representing the user who authenticated during the flow. It maps directly to the `act` claim in the Riptides JWT, while `workloadID` maps to the `sub` claim. Together they form the key the kernel uses to look up and inject the right credential. The binding ensures that only requests from the `claude` workload, acting on behalf of that specific user, will have the real credential injected. For larger deployments, bindings can be managed by user group or by arbitrary claims from the user's ID token rather than per individual user. The intermediate authorization flow, the JWT issuance, and the credential injection all happen automatically once these resources are in place. ## Demo The video below shows the complete double auth flow against Cloudflare MCP. Claude initiates the OAuth flow, the user logs in to Riptides, and is then redirected to Cloudflare to authorize Riptides as a client toward the MCP server. After both authentications complete, Claude is connected. At the end we inspect Claude's credentials file to verify the outcome: ```shell cat ~/.claude/.credentials.json | jq ``` The file contains the Riptides JWT, not a Cloudflare access token. ## Conclusion Remote MCP servers introduced a new class of credential into the agentic workload stack: OAuth access tokens representing delegated user permissions. The standard handling pattern (store the token, attach it to requests) exposes sensitive credentials in agent memory and creates a target for attackers. By acting as an intermediate authorization server and brokering the authentication on the agent's behalf, Riptides removes the real credential from the agent entirely. The agent follows a fully standard OAuth2 flow. The kernel handles the rest. This extends the same secretless model we apply to [cloud provider credentials](/blog/federating-non-human-identities-with-external-idps-using-id-tokens-in-aws-gcp-and-azure) to the OAuth world: workloads authenticate, participate in authorization, and make authenticated requests, without ever possessing the tokens that grant access. The credential lives in the kernel, bound to the workload that earned it, and disappears when it expires. As agentic systems grow in autonomy and begin chaining tools across multiple remote MCP servers, this property becomes critical. An agent that can be prompted into exfiltrating its own credentials cannot exfiltrate credentials it does not have. --- ## The 200-Day TLS Era Has Begun — Is Your Infrastructure Ready? - URL: https://blog.riptides.io/the-200-day-tls-era - Published: 2026-03-17 - Author: Nandor Kracser - Category: Security - Tags: spiffe, mTLS, tls, identity, zero-trust, security As of March 15, 2026, the maximum validity period for publicly trusted TLS/SSL certificates officially dropped from 398 days to 200 days. If you haven't felt the impact yet, you will soon — the next renewal cycle is coming faster than you think. This isn't a decision by any single certificate authority. It's an industry-wide mandate from the CA/Browser Forum (CABF), the governing body made up of certificate authorities, browser vendors, and OS providers. The goal is clear: make the internet more secure by forcing faster rotation, better validation hygiene, and — critically — automation. **Further reading:** - [DigiCert — TLS Certificate Lifetimes Will Officially Reduce to 47 Days](https://www.digicert.com/blog/tls-certificate-lifetimes-will-officially-reduce-to-47-days) - [Sectigo — The 200-Day SSL/TLS Certificate Change](https://www.sectigo.com/blog/200-day-ssl-tls-certificate-lifespans-are-less-than-a-year-away) - [SSL.com — The 200-Day Certificate Deadline](https://www.ssl.com/article/the-200-day-certificate-deadline-is-coming-are-you-ready/) --- ## What Actually Changed Any certificate issued on or after March 15, 2026 must comply with the new 200-day maximum. It's the issue date, not the renewal request date, that determines compliance. If you ordered a renewal before the deadline but it wasn't issued in time, it's still subject to the new limit. This affects more than just public websites. APIs, load balancers, application services, edge environments — anything using publicly trusted TLS is now on a faster clock. In practical terms: you're renewing certificates more than twice as often as before. What was already a demanding process has become genuinely unmanageable without the right automation in place. --- ## This Is Just the Beginning The 200-day limit isn't the destination — it's the first step on a much shorter road: - **March 15, 2027** — maximum validity drops to 100 days - **March 15, 2029** — maximum validity drops to 47 days At 47 days, manual certificate management doesn't just become difficult — it becomes effectively impossible at any meaningful scale. The industry is sending a clear message: automation is no longer optional, it's the baseline. --- ## Why Shorter Certificates Are Actually Good The security rationale is straightforward. Long-lived certificates are a silent risk. A compromised or misissued certificate valid for 398 days gives attackers nearly 13 months of potential exposure. Shortening that window reduces the blast radius significantly. The real-world consequences of long-lived certificates are well documented. In 2023, the Chinese APT group Bronze Starlight was caught signing malware with a stolen code-signing certificate belonging to Ivacy VPN — a certificate that remained valid and trusted for months before DigiCert eventually revoked it. The attackers used it to bypass security tools and blend in with legitimate software traffic, completely undetected. A short-lived, automatically rotated certificate would have rendered the stolen key useless far sooner. There's also a cryptographic agility argument. As post-quantum cryptography standards continue to mature, the ability to rotate certificates quickly — without waiting for year-long validity windows to expire — becomes increasingly critical. Shorter lifetimes mean the ecosystem can adapt faster when trust requirements or algorithms change. More frequent renewals also mean more frequent domain and organization validation, keeping certificate holders accountable and ensuring that who controls a domain today is still who controls it tomorrow. --- ## The Operational Problem No One Talks About Enough Here's what 200-day certificate management actually looks like without automation: a constant cycle of CSR generation, domain validation, installation, and testing — running in parallel across every endpoint in your infrastructure. APIs, internal services, load balancers, Kubernetes ingress controllers, legacy apps. Each one with its own expiry date, each one a potential outage if someone misses it. At 200 days you might still get away with careful spreadsheet tracking and calendar reminders. At 100 days that gets painful. At 47 days it breaks entirely. The teams that are struggling today are the ones who will be in crisis in 2027. The real problem isn't the renewal itself — it's the lack of visibility. Most organizations don't have a clear inventory of every certificate they own, where it's deployed, and when it expires. Without that foundation, automation has nothing to build on. --- ## The Better Model: Identity-Native TLS with SPIFFE and SVIDs One of the most important shifts happening alongside the cert lifecycle changes is a move away from managing certificates as static artifacts and toward treating them as expressions of workload identity. This is the architectural direction the industry is heading — and it's the model Riptides is built around. The SPIFFE standard (Secure Production Identity Framework for Everyone) addresses exactly this. Instead of issuing long-lived TLS certificates tied to a hostname or IP, SPIFFE assigns every workload a cryptographically verifiable identity — called an SVID (SPIFFE Verifiable Identity Document). SVIDs are short-lived by design, automatically rotated, and scoped to the workload rather than the network location. This is a fundamentally better model for the world we're moving into. Rather than asking "is this certificate still valid?", you're asking "is this the workload it claims to be?" Trust is tied to identity, not to infrastructure topology. Because SVIDs are short-lived and automatically rotated, the operational burden of the 200-day — and eventually 47-day — world largely disappears. The platform handles rotation continuously, invisibly, and correctly. Riptides implements this model natively. Every workload and AI agent in your infrastructure gets a SPIFFE identity, and every connection is mutually authenticated using that identity. The CA that signs those identities can live inside the Riptides control plane or be delegated to an external system like HashiCorp Vault or AWS Private CA — whatever fits your existing PKI. Trust extends naturally across cloud providers, clusters, and environments through federated SPIFFE trust domains, with no manual certificate tracking required. --- ## This Trend Isn't Just About Public Certs It's worth being precise about scope: the CA/Browser Forum changes apply specifically to publicly trusted TLS certificates — the ones securing your public websites, external APIs, and customer-facing endpoints. For those, the tooling answer is a solid CLM platform with ACME automation. But the same underlying philosophy — short-lived credentials, continuous rotation, identity over infrastructure — is spreading well beyond public certs. Internal service-to-service communication, workload-to-workload trust, AI agent authentication: these are all areas where the industry is moving toward the same model, driven by security rather than regulatory mandate. This is the space [Riptides](https://riptides.io) is built for. It's not a replacement for your CA or cert manager — it's an identity fabric for the east-west traffic that traditional cert management doesn't reach. Riptides assigns every workload and AI agent a short-lived SPIFFE identity, automates mTLS between services at the kernel level without touching application code, and maintains a real-time inventory of all non-human credentials across your environment. The CA/Browser Forum is forcing short-lived cert rotation onto public infrastructure. Riptides applies that same model to internal workloads — where the tooling has historically lagged furthest behind. If you're rethinking your cert strategy for 2027 and beyond, it's worth thinking about both halves of the problem. [See how Riptides handles the internal side](https://riptides.io/request-a-demo). --- ## What To Do Right Now Even if your current certificates are still valid under the old 398-day limit, the clock is ticking. Here's where to start: **1. Audit your certificate estate.** Identify every public TLS certificate across your infrastructure — load balancers, APIs, ingress controllers, internal services. Record expiry dates, validation types, and who owns each renewal. **2. Find your manual gaps.** Which certificates are renewed by email, spreadsheet, or ticket? Those are your highest-risk assets under the new regime. **3. Move toward automation.** ACME-based workflows and certificate lifecycle management (CLM) platforms are the minimum. Identity-native platforms that eliminate long-lived certs entirely are the longer-term answer. **4. Plan for 2027, not just today.** The 200-day limit is manageable. The 100-day limit arriving in March 2027 is where manual processes start to break. Build for that, not for the deadline you just passed. The 398-day certificate era is over. The infrastructure that thrives in what comes next is the infrastructure that was already treating rotation as a continuous, automated process — not a calendar event. --- ## Our AI Is Helpful. Also Slightly Overprivileged. - URL: https://blog.riptides.io/out-ai-is-helpful-also-slightly-overprivileged - Published: 2026-03-10 - Author: Zsolt Varga - Category: MCP - Tags: mcp, ai, security, agentic This post is the first in a short series about MCP servers and the security questions that come with them. There is a lot of excitement around MCP right now, and that excitement is well deserved. MCP turns AI assistants into something much more practical than a simple chat interface. Instead of just generating text, they can interact with real systems by opening pull requests, updating tickets, querying internal services, or automating workflows that normally require jumping between tools. For many teams this feels like the natural next step in automation. APIs made software programmable, and MCP provides a common interface that allows AI agents to interact with tools and services in a consistent way. If you have not looked at the protocol itself yet, we previously wrote a short introduction that walks through the basics of MCP and how it works: - [MCP: A Quickstart Guide](/blog/mcp-a-quickstart-guide) One of the reasons MCP adoption is moving quickly is that the protocol intentionally stays simple. It focuses on how agents and tools communicate and deliberately avoids prescribing how identity, authorization, or governance should be implemented around it. That flexibility makes MCP easy to adopt, but it also means those concerns are left to the surrounding platform. And that is where things start to get interesting. ## The pragmatic way MCP gets deployed Most MCP deployments do not begin as a carefully designed security architecture. They usually start as experiments. A team sets up a server and connects it to a few useful systems. GitHub is a common starting point, followed by something like a ticketing system or a cloud API. Once those connections are in place, the AI assistant suddenly becomes capable of performing real tasks. It can open pull requests, fetch logs, create tickets, and interact with internal systems that previously required manual steps. In order for that to work, the MCP server needs credentials. Someone therefore provides them, usually in the most straightforward way possible. Sometimes this is a service account, sometimes a personal access token, and sometimes an API key stored in configuration. At this stage the goal is not to design the perfect identity model but simply to make the workflow functional. And once the system works, it tends to stay that way. The AI becomes more useful, developers save time, and the integration remains in place without much further thought. Nothing about this feels controversial. ## The questions that appear later The friction usually shows up later, and it tends to appear in fairly ordinary situations. Someone notices a change in a repository or a cloud environment and wants to understand where it came from. The logs show that the action was performed by the identity attached to the MCP server, which is technically correct. The next question, however, becomes harder to answer clearly. Which human actually initiated the action? Was it Alice using the assistant? Was it Bob running a different workflow? Was it triggered indirectly by an earlier request? It is usually possible to reconstruct the answer by examining several different logs. The AI client might show who asked the original question, the repository history might reveal the sequence of changes, and infrastructure logs may provide additional context. What often does not exist is a single place where the delegation chain is explicitly represented. Instead, there is a trail of events that can be pieced together after the fact, but not necessarily a clear identity relationship that was enforced at the time the action occurred. That gap is subtle, yet organizations tend to notice it quickly. ## The shared credential problem Another issue appears around permissions. In most mature organizations human identities are tightly governed. People authenticate through SSO, access is tied to roles, and there may be approval flows or temporary elevation mechanisms when higher privileges are required. Service accounts usually behave differently. They exist to automate tasks, and automation tends to break when permissions are too restrictive. As a result, service accounts often accumulate capabilities over time. When an MCP server relies on a single credential for multiple users, its authority gradually becomes the union of everyone’s needs. One developer may require repository access, another may need infrastructure visibility, and someone else might rely on the ability to create tickets or query internal services. Nobody intentionally designs an overpowered identity. Instead, the permissions grow gradually because removing them risks breaking workflows that people have come to depend on. Security teams have been dealing with this pattern for years in the broader world of non-human identity. MCP does not introduce the problem, but it adds another layer where the same pattern can appear. ## Identity becomes harder to explain Lifecycle management is another place where things become slightly uncomfortable. Human identities typically follow clear lifecycle controls. When someone joins a company they receive access, when they change roles their permissions are adjusted, and when they leave the organization their account is disabled. Machine credentials rarely follow those same patterns. If an MCP server’s permissions were expanded over time to support different workflows, those privileges tend to remain in place unless someone deliberately reviews and reduces them. The relationship between human lifecycle and machine authority therefore becomes indirect. Some teams try to improve attribution by passing user tokens through the MCP layer instead of relying on a shared service account. This approach can help with visibility because downstream systems see the human identity rather than the machine identity. At the same time it raises a different set of questions. What exactly is being delegated to the MCP server? Is the server simply forwarding the request, or is it combining its own authority with the user’s authority? Is the delegation limited to a specific session or scope, or is it effectively open ended? In many real deployments these questions do not have clearly defined answers. The system functions correctly, but the delegation model remains more implicit than explicit. ## Communication security is only part of the story Another part of the puzzle is the connection between agents and MCP servers themselves. As soon as MCP servers begin interacting across services or environments, the system starts to resemble workload-to-workload communication in distributed systems. At that point questions about authentication, encryption, and trust boundaries become important as well. We explored that aspect earlier when discussing how MCP communication can be secured using strong workload identity and mutual authentication: - [Securing MCP Communication with Riptides](/blog/securing-mcp-communication-with-riptides) Establishing trust between agents and MCP servers is an important step. However, transport security only addresses part of the challenge. Even if every connection is encrypted and every machine proves its identity, the system still needs to represent who the machine is acting for. The identity of the workload and the identity of the human behind the request are two different pieces of information, and both of them matter. ## Machines acting for people This is where things become particularly interesting. In traditional systems two identity models have evolved somewhat independently. OAuth based systems are good at representing delegated user authorization and answering questions such as who granted access and what scope was approved. Workload identity systems such as SPIFFE focus on machines and provide strong cryptographic identity for services communicating with each other. In earlier posts we explored how these two worlds are starting to converge, especially in environments where AI agents and automation interact with real systems: - [SPIFFE Meets OAuth2: Current Landscape for Secure Workload Identity in the Agentic AI Era](/blog/spiffe-meets-oauth2-current-landscape-for-secure-workload-identity-in-the-agentic-ai-era) - [Bringing SPIFFE to OAuth for MCP: Secure Identity for Agentic Workloads](/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads) As agent based workflows become more common, machines increasingly act on behalf of people. Systems therefore need to represent both identities at the same time: the machine identity that executes the request and the human identity whose authority is being delegated. When those two pieces are not clearly defined together, governance becomes much harder to reason about. ## Why this matters now None of these issues feel dramatic during day-to-day use. The AI assistant works, tasks are completed faster, and engineers spend less time switching between tools. The tension usually appears during audits, security reviews, or incident investigations, when someone needs to explain how authority actually flowed from a human, through an AI assistant, into a production system. At that point the architecture often turns out to be more informal than expected. This is not because teams are careless, but because MCP adoption is moving faster than the identity models around it. ## What this series will explore This series will look more closely at that identity layer. In the next posts we will examine how delegation actually works in MCP based systems, why traditional non-human identity controls only partially address the problem, and how stronger patterns might emerge as AI agents become a normal part of software systems. For now, the key observation is simple. MCP introduces a new layer between people and the systems they interact with. Whenever a new layer appears in the identity chain, it is worth treating it as a governance boundary rather than just an implementation detail. --- ## Federation is easy. Runtime enforcement is hard. - URL: https://blog.riptides.io/federation-is-easy-runtime-enforcement-is-hard - Published: 2026-03-02 - Author: Zsolt Varga - Category: SPIFFE - Tags: spiffe, identity, federation In our previous article (*[SPIFFE Identity Federation: Extending Trust Across Boundaries](/blog/spiffe-identity-federation-extending-trust-across-boundaries)*), SPIFFE federation was examined as a mechanism for extending identity trust across independent trust domains. Bundles are exchanged, trust anchors are verified, and workloads authenticate each other using X.509 SVIDs over mTLS. From a cryptographic standpoint, federation is elegant. Two workloads from different organizations can establish mutual trust without sharing a root CA. Identity is represented as a SPIFFE ID, and trust is derived from verified bundles. The protocol is minimal and precise. At that level, federation is straightforward. However, meaningful SPIFFE based communication in production environments requires more than **bundle exchange and certificate validation**. It requires a system in which identity, certificate lifecycle, and communication control remain coherent as complexity increases. ## mTLS Is Only One Layer For SPIFFE based mTLS communication to operate reliably, multiple layers must function together. Workloads must be identified accurately. Certificates need to be issued, distributed, rotated, and eventually revoked. Workloads must be able to use those credentials without disruption during rotation. And there must be a clear, enforceable model that defines which identities are allowed to communicate. Federation extends identity across trust domains. It does not eliminate the need to solve the other layers. When these responsibilities are handled by separate subsystems with different configuration models and runtime assumptions, operational complexity grows. The friction usually appears not in cryptography, but in coordination. ## Federation Expands the Identity Graph Introducing federation expands the identity graph. External SVIDs are now present. Remote bundle endpoints are configured. Cross domain communication paths become possible. The mechanics of verification remain consistent. The surface area that must be understood increases. As trust domains multiply, so do the relationships between identities. Questions emerge that are less about certificate validity and more about system behavior. Which internal workloads can communicate with which external identities? How are those decisions enforced? Where is that enforcement happening? Federation does not introduce these concerns. It amplifies their impact. ## Fragmented Enforcement Increases Complexity In many deployments, identity and enforcement are layered. An identity system issues and rotates SVIDs. Another component terminates TLS. A separate policy engine evaluates authorization. Network rules may apply at IP. Observability is handled elsewhere. Each layer may function correctly in isolation. The difficulty is that enforcement logic becomes distributed across boundaries that do not share the same identity model. Identity is validated in one place. Authorization is decided in another. The actual network connection is controlled somewhere else. As more trust domains are introduced, the cost of aligning these layers increases. The result is not necessarily misconfiguration. It is growing difficulty in reasoning about how identity translates into communication control. ## Trust Scoping in Federation Federation introduces a subtle but important boundary. A federated trust domain should not implicitly become a generally trusted certificate authority within the environment. In [Riptides](https://riptides.io/request-a-demo), a federated bundle is associated explicitly with a declared SPIFFE trust domain and is used only to validate X.509 SVIDs that belong to that domain and conform to SPIFFE validation rules. For example: ``` yaml apiVersion: core.riptides.io/v1alpha1 kind: FederatedTrustDomain metadata: name: payments.acme.corp spec: trustDomain: payments.acme.corp bundleEndpoint: url: "https://payments.acme.corp/.well-known/spiffe-trust-domain-federation" profile: HTTPS_WEB caBundle: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` The `trustDomain` defines the SPIFFE namespace being extended. The associated bundle endpoint is used exclusively for validating SVID chains that assert identities within that namespace. The CA material is not merged into a general purpose trust store, and it cannot validate unrelated TLS certificates. This explicit scoping keeps trust boundaries clear as additional domains are federated. ## Where the Assumption Breaks In many service mesh deployments, the workload itself does not directly hold the SVID. The sidecar proxy terminates mTLS and presents the identity on behalf of the workload. From the remote service's perspective, the connection originates from a verified SPIFFE ID associated with that proxy. This abstraction simplifies application integration, but introduces a critical vulnerability: [identity is attached to the proxy, not to the individual process that initiated the connection](/blog/the-hidden-risk-in-service-mesh-mtls-when-your-sidecar-becomes-a-trojan-horse). If a rogue or compromised process can route traffic through the proxy, it inherits the proxy's valid identity. mTLS succeeds. Authorization policy evaluates the expected SPIFFE ID. The connection is allowed because the enforcement layer sees a legitimate identity. Nothing is misconfigured. The system behaves exactly as designed. The distinction is that identity validation and enforcement [happen in user space at the proxy boundary, not at the point where a specific process creates a socket](/blog/zero-trust-from-perimeter-to-kernel---how-riptides-pushes-the-boundary). The runtime origin of the connection is abstracted away. When federation expands the set of reachable identities across trust domains, this becomes more consequential. A valid federated identity presented by the proxy is sufficient to authenticate externally, even if the initiating process was never intended to establish that relationship. ## From Identity Validation to Runtime Enforcement Riptides approaches workload identities and federation differently. Once an SVID, local or federated, is validated against its corresponding trust domain bundle, the verified SPIFFE ID is bound directly to the originating process at socket communication time. Identity is attached at the connection boundary in the kernel rather than at a user space proxy. Certificate distribution and rotation are handled transparently. Short lived credentials can rotate without application awareness. Revocation or bundle updates affect subsequent connection attempts immediately. Communication policy is evaluated at the moment a connection is created. Enforcement happens before application logic or proxy layers interpret the traffic. A process cannot inherit identity simply by routing traffic through a shared proxy, because identity is bound to the runtime context of that process itself. This alignment of identity validation, certificate lifecycle, and connection level enforcement reduces the number of independent subsystems that must agree for a policy decision to hold. Federation introduces additional identities into the model, but it does not introduce a new enforcement boundary or another user space layer. [No application modification is required. No additional user space interception is necessary. Enforcement remains anchored to the runtime network boundary.](/blog/rethinking-workload-identity-at-the-kernel-level) ## Keeping the Model Coherent as Trust Expands As additional trust domains are federated, the identity graph grows. In architectures where identity issuance, TLS termination, and policy enforcement are handled by distinct layers with separate control planes, each new trust relationship increases coordination overhead. In a unified runtime model, federation extends the set of valid identities while enforcement remains centralized at the connection boundary. The same mechanism governs local and federated identities. The same policy model applies regardless of which trust domain issued the SVID. The cryptographic mechanics of SPIFFE federation remain intentionally simple. The more complex challenge is maintaining a coherent operational model as trust relationships expand. Federation determines which identities can authenticate across boundaries. Runtime enforcement determines which of those authenticated identities can actually communicate. When these concerns are aligned, federation remains an extension of identity rather than an additional source of operational fragmentation. If federation is already part of the architecture, it is worth examining where identity is actually bound and where enforcement truly happens. Riptides makes that boundary explicit at the runtime level. Explore the architecture or reach out if a deeper technical discussion would be useful. --- ## Secretless Azure access with tokenex: Federated Identity via User-Assigned Managed Identity - URL: https://blog.riptides.io/secretless-az-access-with-tokenex - Published: 2026-02-23 - Author: Sebastian Toader - Category: Credentials - Tags: federation, credentials, azure Non-human identities (services, agents, CI/CD pipelines, workloads, etc.) are now the primary actors in modern cloud systems. Yet many systems still rely on: - Client secrets stored in CI systems - Long-lived service principal credentials - Manually rotated keys This is operationally expensive and security-fragile. [tokenex](https://github.com/riptideslabs/tokenex) is an **open-source Go library** that simplifies the process of providing short-lived credentials from various providers. Instead of embedding long-lived secrets or tightly coupling to a specific cloud SDK authentication flow, [tokenex](https://github.com/riptideslabs/tokenex) allows you to exchange identity tokens from external identity providers for short-lived cloud-native access tokens. In other words: > Your workload proves *who it is* using an external identity provider. > [tokenex](https://github.com/riptideslabs/tokenex) exchanges that identity for native short-lived cloud credentials. > The target platform, whether Azure, AWS, GCP, OCI, or any other supported provider, then decides what the workload can do using its own identity primitive (Managed Identity, IAM Role, Service Account, etc.) and its native authorization model (RBAC, IAM policies, resource policies, and so on). This makes [tokenex](https://github.com/riptideslabs/tokenex) ideal for modern zero-trust, federated, multi-cloud environments. ## Why secretless matters ### What does "secretless" really mean? "Secretless" does not mean there are no credentials involved. It means: - No long-lived client secrets - No stored access keys, api keys, tokens, etc - No credentials written to disk - No static environment variables containing secrets Instead, credentials are **derived dynamically** based on identity, are **short-lived**, and exist **only in memory** for the minimum time required. ### The security advantage Traditional approaches often rely on: - Service principal secrets stored in CI/CD systems - Static cloud access keys baked into container images - Credentials written to configuration files - Long-lived tokens injected as environment variables These become high-value targets. If a zero-day vulnerability, dependency compromise, or supply chain attack allows arbitrary code execution inside a workload, the attacker's first move is almost always: > Search the filesystem and environment for credentials. If secrets are stored at rest in files, configs, or environment variables, they can be harvested and reused elsewhere. With a secretless model: - No static cloud credentials exist on disk - No reusable long-lived secrets are embedded in the workload - Access tokens are short-lived and scoped - Credentials are exchanged just-in-time - Tokens expire quickly and cannot be reused indefinitely Even if an attacker gains runtime execution, there are no persistent secrets to extract and exfiltrate for long-term abuse. ### Why short-lived credentials change the threat model Short-lived credentials dramatically reduce blast radius: - Tokens expire automatically - Compromised credentials lose value quickly - Replay windows are narrow - There is no static secret to rotate after an incident This is especially important in the context of: - Zero-day exploits - Dependency confusion attacks - Malicious container base images - CI/CD pipeline compromises ### Security through ephemerality The combination of: - External identity assertion - Token exchange - Short-lived native cloud credentials - No stored secrets creates a model where authentication is dynamic and authorization is enforced natively without leaving reusable artifacts behind. This is not just an operational improvement; it is a fundamental shift in security posture. *Secretless is not about convenience. It is about eliminating credential persistence as an attack surface.* ## Sample Go application using [tokenex](https://github.com/riptideslabs/tokenex) In the following section, we’ll walk through a concrete example: a simple Go application that uses [tokenex](https://github.com/riptideslabs/tokenex) to obtain an Azure access token and then invokes an Azure API using that token for authentication. To get there, we will: - Configure Azure to trust an external identity using **federated credentials** - Bind that trust to a **User-Assigned Managed Identity (UAMI)** - Use **tokenex’s Azure credentials provider** to exchange an external ID token for an Azure access token - Use the returned access token inside a Go application to authenticate and call an Azure API The goal is to demonstrate an end-to-end, secretless flow where: - Identity is asserted externally - Credentials are exchanged dynamically - Authorization is enforced by Azure RBAC By the end, you’ll have a minimal but production-relevant example showing how to invoke Azure APIs securely without storing Azure secrets in your application. ### Azure setup: Federated Identity with User-Assigned Managed Identity (UAMI) Below are the required Azure configuration steps to enable federation. 1️⃣ Create a User-Assigned Managed Identity ``` bash az identity create --name demo-uami --resource-group demo-rg --location ``` Capture the output values: - `clientId` - `principalId` - `id` 2️⃣ Assign a Role to the Managed Identity Grant the identity permission to access resources in the `demo-rg` resource group. ``` bash az role assignment create --assignee --role Reader --scope /subscriptions//resourceGroups/demo-rg ``` Adjust the role and scope as needed for your demo. 3️⃣ Create Federated Identity Credential Now configure Azure to trust your external identity provider's ID token. ``` bash az identity federated-credential create --name demo-fic --identity-name demo-uami --resource-group demo-rg --issuer https://your-idp.example.com --subject --audience api://AzureADTokenExchange ``` Important fields: - `issuer` → must match the `iss` claim of your ID token - `subject` → must match the `sub` claim of your ID token - `audience` → must be `api://AzureADTokenExchange` Once configured, Azure will accept valid ID tokens from your external IdP and exchange them for Azure access tokens scoped to the user assigned managed identity. ### Demo application ``` go package main import ( "context" "log/slog" "os" "os/signal" "sync" "syscall" "time" "github.com/Azure/azure-sdk-for-go/sdk/azcore" "github.com/Azure/azure-sdk-for-go/sdk/azcore/policy" "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources" "github.com/go-logr/logr" "go.riptides.io/tokenex/pkg/azure" "go.riptides.io/tokenex/pkg/credential" "go.riptides.io/tokenex/pkg/token" ) // accessTokenStore is a thread-safe store for an Azure access token. type accessTokenStore struct { azcore.TokenCredential mu sync.RWMutex accessToken azcore.AccessToken } func (s *accessTokenStore) Set(token *credential.Oauth2Creds) { s.mu.Lock() defer s.mu.Unlock() s.accessToken.Token = token.AccessToken s.accessToken.ExpiresOn = token.Expiry } func (s *accessTokenStore) GetToken(ctx context.Context, _ policy.TokenRequestOptions) (azcore.AccessToken, error) { s.mu.RLock() defer s.mu.RUnlock() return s.accessToken, nil } func main() { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() // clean up signal handler logger := logr.FromSlogHandler(slog.Default().Handler()) logger.Info("Press Ctrl+C to stop...") // setup credential provider to receive Azure credentials credProvider, err := azure.NewCredentialsProvider(ctx, logger) if err != nil { logger.Error(err, "failed to create Azure credentials provider") return } // under the hood, the credential provider uses Microsoft Entra ID workload identity federation to fetch user principal session tokens from Microsoft Entra ID service // the credential provider exchanges an input ID token for an Azure user principal session token // the input ID token can be obtained from any OIDC compliant IDP (e.g. Google, Microsoft, Auth0, Okta, etc.) // for this example, we use a static ID token provider that returns a hardcoded ID token issued by an OIDC compliant IDP // in a real application, you would implement the `token.IdentityTokenProvider` interface to create a dynamic ID token provider that fetches the ID token from an OIDC compliant IDP idTokenJwt := os.Getenv("ID_TOKEN_JWT") if idTokenJwt == "" { logger.Error(nil, "ID_TOKEN_JWT environment variable is not set") return } azSubscriptionId := os.Getenv("AZURE_SUBSCRIPTION_ID") if azSubscriptionId == "" { logger.Error(nil, "AZURE_SUBSCRIPTION_ID environment variable is not set") return } azClientId := os.Getenv("AZURE_CLIENT_ID") if azClientId == "" { logger.Error(nil, "AZURE_CLIENT_ID environment variable is not set") return } azTenantId := os.Getenv("AZURE_TENANT_ID") if azTenantId == "" { logger.Error(nil, "AZURE_TENANT_ID environment variable is not set") return } resourseGroupName := os.Getenv("AZURE_RESOURCE_GROUP_NAME") if resourseGroupName == "" { logger.Error(nil, "AZURE_RESOURCE_GROUP_NAME environment variable is not set") return } idTokenProvider := token.NewStaticIdentityTokenProvider(idTokenJwt) creds, err := credProvider.GetCredentials(ctx, idTokenProvider, // supplies the ID token issued by an OIDC compliant IDP for the application(workload) that is going to use the Azure service principal session tokens for authentication. azure.WithClientID(azClientId), azure.WithTenantID(azTenantId), azure.WithScope("https://management.azure.com/.default"), ) if err != nil { logger.Error(err, "failed to get Azure credentials") return } accessToken := &accessTokenStore{} // retrieve Azure credentials and updates before they expire for the identity that corresponds to the provided ID token go func() { defer stop() for { select { case <-ctx.Done(): return case credentialEvent := <-creds: if credentialEvent.Err != nil { logger.Error(credentialEvent.Err, "failed to get Azure credentials") return } token, ok := credentialEvent.Credential.(*credential.Oauth2Creds) if !ok { logger.Error(err, "failed to assert credential type") return } // update the access token used by the application to authenticate to Azure services accessToken.Set(token) logger.Info("received Azure credentials", "expiry", token.Expiry) } } }() go func() { defer stop() // simulate application running and using the Azure credentials for authentication to Azure services // periodically check and print resources that appeared in the resource group to demonstrate that the credentials are being refreshed and can be used to authenticate to Azure services armresourcesClient, err := armresources.NewClient(azSubscriptionId, accessToken, nil) if err != nil { logger.Error(err, "failed to create Azure Resource Management client") return } ticker := time.NewTicker(2 * time.Second) defer ticker.Stop() trackedResourceIDs := make(map[string]struct{}) // to track seen resources and only log new ones for { select { case <-ctx.Done(): return case <-ticker.C: resourceIds := make(map[string]*armresources.GenericResourceExpanded) pager := armresourcesClient.NewListByResourceGroupPager(resourseGroupName, nil) for pager.More() { page, err := pager.NextPage(context.Background()) if err != nil { logger.Error(err, "failed to get resources from Azure Resource Management API") return } for _, resource := range page.Value { if resource == nil { continue } resourceIds[*resource.ID] = resource } } for id, resource := range resourceIds { if _, seen := trackedResourceIDs[id]; !seen { logger.Info("new resource", "name", *resource.Name, "type", *resource.Type, "id", id) trackedResourceIDs[id] = struct{}{} } } for id := range trackedResourceIDs { if _, exists := resourceIds[id]; !exists { logger.Info("resource removed", "id", id) delete(trackedResourceIDs, id) } } ticker.Reset(1 * time.Minute) } } }() <-ctx.Done() // wait for signal to stop logger.Info("exiting...") } ``` #### Running the application 1️⃣ Configure environment variables ```bash $ export AZURE_SUBSCRIPTION_ID= $ export AZURE_CLIENT_ID= $ export AZURE_TENANT_ID= $ export AZURE_RESOURCE_GROUP_NAME=demo-rg $ export ID_TOKEN_JWT= ``` 2️⃣ Run the application ```bash $ go run main.go ``` *Sample output (successful access)* ```text 2026/02/21 17:15:28 INFO Press Ctrl+C to stop... 2026/02/21 17:15:28 INFO received Azure credentials expiry=2026-02-22T17:15:27.786Z 2026/02/21 17:15:31 INFO new resource name=demo-uami type=Microsoft.ManagedIdentity/userAssignedIdentities id=/subscriptions//resourceGroups/blog/providers/Microsoft.ManagedIdentity/userAssignedIdentities/demo-uami ``` *Authorization failure scenario* Run the application again with a resource group that the `demo-uami` user assigned managed identity has no access to: ```bash $ export AZURE_RESOURCE_GROUP_NAME=test-resource-group-1 $ go run main.go ``` *Sample output (authorization failed)* ```text 2026/02/21 17:21:36 INFO Press Ctrl+C to stop... 2026/02/21 17:21:36 INFO received Azure credentials expiry=2026-02-22T17:21:35.948Z 2026/02/21 17:21:38 ERROR failed to get resources from Azure Resource Management API err="GET https://management.azure.com/subscriptions//resourceGroups/test-resource-group-1/resources\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '' with object id '' does not have authorization to perform action 'Microsoft.Resources/subscriptions/resourceGroups/resources/read' over scope '/subscriptions//resourceGroups/test-resource-group-1' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n" 2026/02/21 17:21:38 INFO exiting... ``` - When the identity has proper access: - The application lists newly detected resources. - Credentials are automatically refreshed. - Resource additions/removals are tracked. - When access is denied: - Azure returns `AuthorizationFailed (403)`. - The application logs the error and exits gracefully. ### How the flow works 1. Your application obtains an **ID token** from an external IdP. 1. [tokenex](https://github.com/riptideslabs/tokenex) sends that token to Azure's token endpoint. 1. Azure validates: - issuer - subject - audience - federated credential configuration 1. Azure issues an **access token** bound to the User-Assigned Managed Identity. 1. Your application uses that token to call Azure APIs. ![Authentication sequence](../../assets/tokenex-msentra/azure_fed.png) ### Architectural model At a high level: External IdP (OIDC) ↓ ID Token (JWT) ↓ tokenex ↓ Azure OAuth Token Endpoint ↓ Managed Identity Access Token ↓ Azure Resource API Key separation of concerns: - External IdP: asserts identity (authentication) - Azure UAMI: defines authorization boundary (RBAC) - [tokenex](https://github.com/riptideslabs/tokenex): performs OAuth token exchange and refresh handling - Azure Resource Manager / Graph / other APIs: enforce RBAC ## Final Thoughts Federated workload identity is becoming the standard method for authenticating non-human identities in the cloud. Azure's support for federated credentials tied to User-Assigned Managed Identities enables secure, secretless authentication patterns. By combining this with [tokenex](https://github.com/riptideslabs/tokenex), you get: - A clean abstraction for cloud credential exchange - Automatic token refresh handling - A unified interface across multiple cloud providers - Reduced operational complexity - Improved security posture If you're building multi-cloud or external-IDP-integrated systems, [tokenex](https://github.com/riptideslabs/tokenex) provides a practical, production-ready way to implement secure workload federation with Azure. --- ## Secretless AI-Powered Development: Secure AWS Credentials for GitHub Copilot in Lima - URL: https://blog.riptides.io/secretless-ai-development-github-copilot-lima - Published: 2026-02-12 - Author: Nandor Kracser - Category: Development - Tags: federation, non-human identity, aws, development, ai-coding ## Secretless AI-powered development with on-the-wire credential injection Modern development increasingly happens inside different environments: local machine, containers, VMs, remote dev servers. At the same time, AI coding assistants like **GitHub Copilot** have supercharged developer productivity, but they've also amplified a critical security problem. When you're using AI to generate infrastructure code, write `boto3` scripts, or build `Terraform` configs at 10x speed, **credential management is your primary failure point and the biggest security risk**. How do you securely give a VM or container access to cloud APIs **without mounting credential files, passing secrets as environment variables, or copying keys into the environment**, especially when AI assistants are generating code that needs those credentials? **And here's the real concern**: With AI generates code you didn't write line-by-line, how confident are you that it won't accidentally log secrets, echo environment variables, or write credentials to temp files? Running AI-generated code in an environment with **zero access to credentials** means you can sleep better at night. The conventional answer is **still based on secrets**: copy your `~/.aws/credentials` file, mount it as a volume, set `AWS_ACCESS_KEY_ID` in your shell, or use instance profiles if you're lucky enough to be running in the cloud already. But this approach has serious flaws: Credential material tends to **proliferate** (copied into VM images, accidentally committed, or left behind in shared environments), the **blast radius expands** (every developer/VM/container with access can leak it), **rotation becomes manual and painful** (you have to chase down every place the secret landed), and you still get poor **auditability** (it’s hard to answer “who used which credentials, when?” and “which process touched what API?”). This post shows a different approach: **secretless development environments** powered by Riptides. We'll use **Lima VMs on macOS with GitHub Copilot** as a practical example, but the same pattern applies to Docker containers, remote SSH sessions, cloud dev environments, or any workspace where your AI-generated code runs. **Why sandboxed environments matter for AI-generated code**: Treat AI-generated code as untrusted by default, run it in a **sandboxed VM or container** with **zero credential access** so even accidental logs, debug outputs, or file writes can't leak credentials. ## The problem with development credentials Let's say you're a macOS developer building cloud-native applications. You want a clean Linux development environment, so you use [Lima](https://github.com/lima-vm/lima) to run a lightweight Ubuntu VM. Now you need to run AWS CLI commands, test Terraform configs, or call AWS APIs from Python scripts inside that VM. ### The standard approach: Copy your credentials The typical workflow looks like this: 1. **Export AWS credentials on your Mac**: ```bash export AWS_ACCESS_KEY_ID=AKIA... export AWS_SECRET_ACCESS_KEY=... ``` 2. **Pass them to the VM via environment or mount**: ```yaml # Lima config env: AWS_REGION: us-west-2 AWS_ACCESS_KEY_ID: AKIA... AWS_SECRET_ACCESS_KEY: ... ``` 3. **Or mount your credentials directory**: ```yaml mounts: - location: "~/.aws" writable: true ``` 4. **Run AWS commands inside the VM**: ```bash limactl shell default aws ec2 describe-instances ``` This works. But you've just **copied possibly long-lived secrets into an isolated environment where they're harder to rotate, easier to leak, and completely unaudited**. ### What can go wrong? Credential files are **persistent** (they remain on the VM filesystem until manually deleted), and VM snapshots or “golden images” can quietly **capture secrets** as a side effect of reproducibility. Shared VM templates often turn into **shared secrets**, any process in the VM may be able to read credential files, and while AWS CloudTrail can tell you which IAM principal made a call, it usually won’t tell you *which process* or *which workload* inside your dev VM did it. When credentials expire or are compromised, **rotation** becomes a manual update process across every VM/container/config you touched. Even if you use temporary credentials via AWS STS, **you're still handling and storing secrets**, they just happen to be short-lived secrets. ### The AI coding assistant paradox **GitHub Copilot in VS Code** promises 10x developer productivity. It generates infrastructure code, writes AWS automation scripts, and builds Terraform configurations in seconds. But this productivity gain is **bottlenecked by credential management**: Copilot can generate Terraform code, or a deployment script in seconds—yet you still end up stopping to configure AWS credentials, copy secrets into the VM, or manage API keys and access tokens. **The faster you code with AI, the more you're fighting with credentials.** And worse: the more code AI generates, the more opportunities for credential leakage. AI-generated scripts that echo variables, log debug output, or write to temporary files can accidentally expose secrets that weren't there before. ### The security benefit: AI code + sandboxed environments + zero credentials With that setup, you get multiple layers of protection: AI-generated code runs in an isolated VM/container rather than on your host; there are **no credentials on disk** to leak even if the code behaves unexpectedly; and Riptides can tie credential injection and telemetry to the **calling process**. **This means you can trust AI to generate code at high speed without worrying about accidentally creating security vulnerabilities.** Run it, test it, iterate, knowing that even if something goes wrong, credentials were never in play. ## How Riptides solves the problem Riptides eliminates stored secrets by injecting credentials **dynamically at runtime, directly into the network requests that need them**. The workload never stores, reads, or manages AWS credentials directly. Here's how it works: 1. **Every process gets a kernel-enforced identity** Riptides assigns a [SPIFFE-based workload identity](/blog/rethinking-workload-identity-at-the-kernel-level) to each process, verified and enforced at the kernel level. 2. **Credentials are injected on-the-wire** When a process (like the AWS CLI) makes an HTTPS request to AWS APIs, Riptides intercepts the connection in kernel space, fetches short-lived credentials, and injects them directly into the request using [libsigv4](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility). 3. **No secrets on disk, ever** The AWS CLI runs normally, but it never sees credentials. No `~/.aws/credentials` file. No environment variables. No mounted volumes. 4. **Automatic rotation and refresh** Credentials are scoped to a single request or short time window. They expire quickly and are automatically refreshed. 5. **Full auditability** Every credential usage is logged: which workload, which process, which API call, when. If you're familiar with our earlier credential injection posts, this is the same approach applied to development environments: - [On-the-Wire Credential Injection: Secretless AWS Bedrock Access](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) - [Secretless OCI Authentication with SPIFFE-based workload identity](/blog/secretless-oci-authentication-with-spiffe-based-workload-identity) - [On-demand credentials: Secretless AI assistant example on GCP](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp) The difference is that here we're applying it to a **local development environment on macOS**, making secure, secretless workflows practical for everyday AI-assisted coding. ## Why this matters for GitHub Copilot (and other AI coding tools) **GitHub Copilot** excels at generating cloud infrastructure code. But it can't solve the credential problem—it just makes it worse: **Without Riptides:** | Workflow | Without Riptides | With Riptides | |---|---|---| | Copilot generates AWS code | You manually configure credentials | Credentials are injected automatically when the code runs in a jailed environment | | Copilot writes Terraform | You mount `~/.aws` into your VM | No credential configuration needed; run in an isolated VM | | Copilot builds deployment scripts | You copy secrets into environment variables | Scripts execute securely in jail without ever seeing secrets | | Result | Slow, manual, error-prone credential management that negates AI gains | AI-accelerated development with defense in depth and near-zero credential overhead | You get the productivity of AI code generation **without the security risks of credential sprawl or running untrusted code with full system access**. **Note**: While this post focuses on GitHub Copilot in VS Code, the same pattern should work with other AI coding environments like **Cursor**, **Continue**, **Cody**, and similar tools that support remote development or SSH workflows. ## Why Lima VMs? [Lima](https://github.com/lima-vm/lima) is a lightweight Linux VM manager for macOS. It provides: Lima gives you fast, native-ish integration (file sharing, port forwarding, and shell access) while still running a full Linux kernel—so Riptides can load its kernel module (unlike many desktop container setups). It’s also relatively lightweight and automation-friendly thanks to declarative YAML configuration and CLI tooling. Lima is ideal for running Riptides in development because **Riptides operates at the kernel level**, requiring a Linux environment where kernel modules can be loaded. On macOS, Lima provides exactly that. ## Architecture overview ![Architecture overview: macOS host (VS Code + Copilot via Remote-SSH) into a Lima VM running jailed AWS CLI/Terraform with Riptides agent + kernel module injecting credentials](../../assets/copilot-lima-riptides/copilot-lima-riptides.jpg) **How requests flow:** 1. VS Code on macOS connects to the Lima VM via Remote-SSH 2. You use Copilot to generate code/commands, and run tools like AWS CLI and Terraform inside the VM 3. AWS CLI / Terraform attempts an HTTPS request to AWS APIs (for example `ec2.amazonaws.com`) 4. The Riptides kernel module intercepts the connection 5. The Riptides agent: - Verifies the process identity - Fetches short-lived AWS credentials (via federation or local provider) - Passes credential material to the kernel module 6. The Riptides kernel module: - Signs the HTTP request using [libsigv4](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility) - Injects the signed headers into the request 7. The request proceeds to AWS, fully authenticated 8. AWS CLI / Terraform receives a successful response — without ever seeing credentials ## Setting up Lima with Riptides ### Prerequisites You’ll need macOS with Homebrew, Lima installed (`brew install lima`), and access to the **Riptides kernel module + agent** (currently closed source; contact [Riptides Labs](mailto:info@riptides.io) to request access—open source release planned for later in 2026). ### Step 1: Start a Lima VM Create a Lima VM configuration file `riptides.yaml`: ```yaml vmType: "vz" os: "Linux" arch: "default" images: - location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img" arch: "x86_64" - location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img" arch: "aarch64" cpus: 4 memory: "8GiB" disk: "100GiB" mounts: - location: "~" writable: true - location: "/tmp/lima" writable: true provision: - mode: system script: | #!/bin/bash apt-get update apt-get install -y build-essential curl git awscli snap install terraform --classic containerd: system: false user: false ``` Start the VM: ```bash limactl start riptides.yaml ``` ### Step 2: Install Riptides Once you have access to the Riptides repository, install via apt inside the Lima VM: ```bash limactl shell riptides # Add Riptides repository (provided after contacting Riptides Labs) # Install Riptides (includes kernel module, agent, and systemd service) sudo apt-get update sudo apt-get install riptides # Verify the kernel module is loaded lsmod | grep riptides ``` ### Step 3: Connect VS Code Remote-SSH To use GitHub Copilot in the Lima VM, connect via VS Code Remote-SSH. Lima VMs integrate seamlessly with VS Code Remote-SSH—see the [Lima VS Code integration guide](https://lima-vm.io/docs/examples/vscode/) for more details. Add the SSH config to your `~/.ssh/config`: ```bash limactl show-ssh riptides >> ~/.ssh/config ``` Then connect via the VS Code Command Palette (Remote-SSH: Connect to Host). ![Connecting to Lima VM via VS Code Remote-SSH](../../assets/copilot-lima-riptides/connect-vscode-ssh.png) **For other AI coding tools**: Environments like **Cursor**, **Continue**, and **Cody** that support SSH remote development should work with the same configuration. ### Step 4: Configure Riptides workload identities To enable secretless AWS access, configure Riptides to recognize AWS CLI and Terraform processes and inject credentials automatically. #### Define the AWS service Create a Service resource that matches all AWS API endpoints: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: amazonaws namespace: riptides-system spec: addresses: - address: "*.amazonaws.com" port: 443 external: true labels: svc:name: amazonaws ``` This tells Riptides to intercept HTTPS connections to any `*.amazonaws.com` domain. #### Configure workload identity for AWS CLI Create a WorkloadIdentity for the AWS CLI process: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: awscli namespace: riptides-system spec: workloadID: awscli connection: tls: intercept: true scope: agent: id: riptides/agent/local-copilot-lima selectors: - process:name: aws ``` This configuration: This assigns identity to any process named `aws`, enables TLS interception so Riptides can inject credentials into HTTPS requests, and scopes the rule to the Lima VM agent (`local-copilot-lima`). #### Configure workload identity for Terraform Similarly, create a WorkloadIdentity for Terraform: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: terraform namespace: riptides-system spec: workloadID: terraform connection: tls: intercept: true scope: agent: id: riptides/agent/local-copilot-lima selectors: - process:name: terraform ``` With these workload identities in place, both AWS CLI and Terraform processes receive: With these workload identities in place, both AWS CLI and Terraform gain identity-scoped credential injection, per-process connection telemetry (domain/port/protocol), automatic SigV4 signing, and better audit context (Riptides records workload identity + process details while CloudTrail records the resulting AWS API calls). #### Complete the credential binding After defining these resources, create an AWS `CredentialSource` and `WorkloadCredential`, then reference that credential from the `WorkloadIdentity` egress rule (as shown in our [AWS credential injection guide](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example)). For AWS IAM trust relationship setup (OIDC + subject mapping), see [Federating non-human identities with external IdPs](/blog/federating-non-human-identities-with-external-idps-using-id-tokens-in-aws-gcp-and-azure). #### Configure AWS CLI for TLS interception Since Riptides intercepts TLS connections to inject credentials, you need to configure the AWS CLI to trust Riptides' CA certificate. Create or update `~/.aws/config`: ```ini [default] output = json region = eu-west-1 ca_bundle = /sys/module/riptides/certs/ca-certificates.crt ``` The `ca_bundle` points to the CA certificate that Riptides uses for TLS interception. (Depending on install/version, you may instead set `AWS_CA_BUNDLE` to a path like `/sys/kernel/riptides/ca-certificates.crt`.) ## Using secretless AWS access in the Lima VM Once Riptides is running, AWS CLI commands work **without any credential configuration**: ```bash # No ~/.aws/credentials file # No AWS_ACCESS_KEY_ID env var # Just run AWS commands aws ec2 describe-instances ``` Behind the scenes: 1. Riptides detects the `aws` process execution 2. The agent matches it against configured rules 3. The process is assigned a workload identity based on its runtime attributes 4. As the AWS CLI makes HTTPS requests, Riptides intercepts them 5. Short-lived credentials are injected into the request at the kernel level 6. The request proceeds to AWS, fully authenticated 7. The AWS CLI receives the response — never knowing credentials were involved ## Real-world GitHub Copilot workflows with Riptides These examples show how GitHub Copilot works seamlessly with Riptides—generating code that executes securely in a jailed environment without credential configuration. (Here, “jailed” refers to running inside the Lima VM; Riptides’ core contribution is secretless, identity-scoped credential injection and per-process visibility.) ### Copilot-generated Terraform workflow Ask GitHub Copilot: *"Help me query my EC2 instances with Terraform"* Copilot can help generate the AWS CLI command to list your instances: ![GitHub Copilot generating EC2 list command](../../assets/copilot-lima-riptides/copilot-ec2-list.png) First, list your existing EC2 instances: ![EC2 instances listed without any credential configuration](../../assets/copilot-lima-riptides/copilot-ec2-list-output.png) Copilot can help generate the Terraform configuration. Create `main.tf`: ![Copilot generating Terraform configuration](../../assets/copilot-lima-riptides/copliot-terraform.png) ```hcl # No credentials configured - Riptides injects them automatically provider "aws" { sts_region = "us-east-1" } # Query all running EC2 instances data "aws_instances" "running" { filter { name = "instance-state-name" values = ["running"] } } # Get details for each instance data "aws_instance" "details" { for_each = toset(data.aws_instances.running.ids) instance_id = each.value } # Output instance information output "instances" { value = { for id, instance in data.aws_instance.details : id => { id = instance.id instance_type = instance.instance_type ami = instance.ami private_ip = instance.private_ip public_ip = instance.public_ip tags = instance.tags } } } ``` Run Terraform: ```bash terraform init terraform plan terraform apply # View the discovered instances terraform output -json instances ``` After running the Terraform workflow, the Riptides UI shows exactly how many times credentials were injected for Terraform. In this example, you can see that the UI displayed **9 credential injections** for the Terraform process: ![Riptides UI showing 9 credential injections for Terraform](../../assets/copilot-lima-riptides/credential-count.png) **Key point**: Notice there's **no credential configuration anywhere** in the Terraform code or environment: **Key point**: Notice there's **no credential configuration anywhere** in the Terraform code or environment—no `~/.aws/credentials` file, no `AWS_ACCESS_KEY_ID` environment variable, and no credential blocks in the provider configuration. It’s just a plain provider with a region. Terraform executes successfully because **Riptides intercepts AWS SDK calls** from the Terraform binary and injects credentials transparently at the kernel level. **The key difference**: With Riptides, you can use Copilot to generate infrastructure code at full speed, then run it immediately in a jailed VM—no credential setup interrupting your flow, and no risk of the generated code accessing credentials it shouldn't. **You can confidently run AI-generated code knowing**: **You can confidently run AI-generated code knowing** the VM jail restricts filesystem access, no credentials exist to leak, and even if Copilot generated code with unintended side effects, the blast radius is minimal. ## Key benefits for AI-powered development ### Security without friction, at AI speed Riptides removes credential management from the loop: you never copy/mount/configure AWS credentials (even for Copilot-generated code), and you don’t need application changes—CLI commands, scripts, and Terraform configs just work. Protection is automatic for configured AWS-calling processes, so the flow becomes *generate → run in jail → iterate* without pausing to wire up secrets. The payoff is peace of mind: AI-generated code runs in an isolated environment with **zero credential access**. ### Better collaboration With no embedded secrets, VM templates can be shared, versioned in git, and distributed safely. Onboarding gets faster (less “here’s how to set up your AWS credentials” documentation), and teams get more consistent dev environments. ### Improved auditability Riptides logs which process made which call and makes it practical to correlate workload/process context with CloudTrail. Unexpected processes reaching for AWS APIs become visible, logged, and easier to investigate. ### Complete network observability Riptides provides real-time visibility into every network connection from your development environment. ![Riptides UI showing network connections](../../assets/copilot-lima-riptides/main-ui.png) See exactly which processes connect where: GitHub Copilot's Node.js runtime to `api.individual.githubcopilot.com:443`, AWS CLI to AWS service endpoints, Terraform to AWS APIs—all in real-time with full process-level detail. Essential for verifying AI-generated code behavior before trusting it. ## Beyond Lima: Apply this pattern anywhere While this post uses Lima VMs on macOS with GitHub Copilot as a concrete example, **the same pattern applies to**: Docker containers (no more `-v ~/.aws:/root/.aws` mounts), remote dev servers over SSH, CI/CD pipelines, Kubernetes dev clusters, and cloud IDEs like Codespaces/Cloud9/Gitpod can all benefit from the same idea. In practice, it should also work with other AI coding environments (Cursor, Continue, Cody, etc.) as long as they support SSH or remote development. The core principle remains the same: **jailed environments + workload identity + zero credentials = AI code you can run confidently**. ## Conclusion Development environments are often the weakest link in cloud security. Credentials get copied, mounted, committed to git, or left lying around in VM snapshots. Riptides eliminates this problem by bringing the same secretless, identity-based security model that we've demonstrated for production workloads ([AWS](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example), [GCP](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp), [OCI](/blog/secretless-oci-authentication-with-spiffe-based-workload-identity)) to local development environments. Key takeaways: Key takeaways: AWS CLI, Terraform and other tools work without credential configuration—whether you wrote them or Copilot did. Isolation and credential injection are enforced at the OS/kernel layer (not in application code), the generated scripts/configs run unchanged in jailed environments, and credential usage becomes auditable with process-level detail. Most importantly, Copilot’s productivity isn’t throttled by credential setup, and you can run AI-generated code with confidence because credentials never exist in the environment. Whether you're using Lima VMs, Docker containers, remote SSH sessions, or cloud-based development environments, Riptides provides a path to **secretless development** that doesn't compromise productivity. ## Learn more - [Rethinking Workload Identity at the Kernel Level](/blog/rethinking-workload-identity-at-the-kernel-level) - [Introducing libsigv4: AWS SigV4 Signatures in Portable C](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility) - [Workload Attestation and Metadata Gathering](/blog/workload-attestation-and-metadata-gathering-building-trust-from-the-ground-up) - [Lima Documentation](https://lima-vm.io/) --- ## SPIFFE Identity Federation: Extending Trust Across Boundaries - URL: https://blog.riptides.io/spiffe-identity-federation-extending-trust-across-boundaries - Published: 2026-02-09 - Author: Zsolt Varga - Category: Federation - Tags: spiffe, federation, identity Workload identity has become a foundational primitive in modern infrastructure. Static credentials do not scale, degrade over time, and rarely explain why access should be allowed. The SPIFFE specification addresses this by defining a standard way to issue short-lived, cryptographically verifiable identities to workloads. From the beginning, SPIFFE was designed with multiple administrative boundaries in mind. Real systems span clusters, cloud accounts, regions, organizations, and partners. Identity must cross those boundaries without collapsing everything into a single control plane. SPIFFE identity federation exists to solve exactly this problem. ## Trust domains as the foundation SPIFFE organizes identity around trust domains. A trust domain represents an administrative boundary with authority over: - its SPIFFE ID namespace - its trust anchors - the issuance of workload identities within that namespace A workload identity is expressed as a SPIFFE ID, for example: ``` spiffe://prod.acme.corp/frontend ``` That identity is meaningful only because the verifier trusts the authority that governs the **prod.acme.corp** trust domain. Within a single trust domain: - workloads receive SVIDs - SVIDs chain to the domain’s trust anchors - peers verify identities using those anchors Federation begins when workloads must authenticate across trust domains. ## Why federation is necessary Consider two independent trust domains: ``` spiffe://payments.acme.corp spiffe://inventory.acme.corp ``` Each domain: - operates its own control plane - issues and rotates its own identities - enforces its own policies - maintains its own trust anchors If a workload in **payments.acme.corp** needs to communicate with a workload in **inventory.acme.corp**, identity must cross an administrative boundary. Without federation, teams typically fall back to: - sharing trust anchors globally - terminating identity at gateways and re-issuing credentials - abandoning identity and relying on networks or secrets All of these approaches erase workload identity at the boundary. SPIFFE federation exists to avoid that outcome. ## Federation as defined by the SPIFFE specification Federation is a first-class concept in the SPIFFE specification. The spec defines federation as the ability for a trust domain to verify identities issued by another trust domain without: - sharing private keys - merging control planes - centralizing identity issuance To support this, the specification defines: - trust domain relationships - trust bundles - bundle endpoints - verification semantics At the same time, the spec deliberately limits its scope to identity verification. ## Trust bundles as the unit of trust The core object used in federation is the SPIFFE trust bundle. A trust bundle contains: - one or more trust anchors for a trust domain - optional metadata - no private key material Trust bundles are the only artifacts exchanged during federation. No identities are delegated, and no keys are shared. ## Standardized bundle endpoints The SPIFFE federation specification defines standardized bundle exchange using SPIFFE bundle endpoints. Bundle endpoints are HTTPS endpoints that serve trust bundles. Two authentication profiles are defined: - https_web, authenticated using Web PKI TLS - https_spiffe, authenticated using SPIFFE X.509-SVIDs The specification requires that: - bundle endpoint clients MUST support both profiles - bundle endpoint servers MUST support at least one profile - TLS configurations meet defined security requirements This ensures interoperability while allowing deployment flexibility. ## How to establish federation Assume the following trust domains: ``` spiffe://payments.acme.corp spiffe://inventory.acme.corp ``` To federate: 1. **payments.acme.corp** is federated with: - the trust domain name **inventory.acme.corp** - the bundle endpoint URL for that domain - the expected endpoint authentication profile 2. it periodically fetches the trust bundle from that endpoint 3. it validates the bundle according to the specification Federation can be symmetric or asymmetric depending on configuration. ## How verification works with federation Federation does not change how workloads authenticate. At runtime: 1. a workload presents its SVID 2. the verifier extracts the SPIFFE ID 3. the trust domain portion of the ID is identified 4. the corresponding trust bundle is selected 5. certificate chain verification proceeds normally Federated identities are verified using the same cryptographic process as local identities. ## Explicit and scoped trust SPIFFE federation is explicit and opt-in. Trust domains do not implicitly trust each other. Each federation relationship must be configured intentionally, including: - which trust domain is federated - where its bundle endpoint is located - how its bundle is authenticated - how often it is refreshed The spec enables federation but does not force trust expansion. ## What federation does not define Even with standardized bundle exchange, the SPIFFE federation spec intentionally does not define: - authorization semantics - policy models - enforcement location - federation topology - governance or lifecycle rules Federation answers one question only: Can this workload identity be cryptographically verified as originating from the stated trust domain? What that identity is allowed to do remains a separate concern. ## Common misconceptions Federation is often misunderstood. It is not: - a global root CA - transitive by default - workload-level delegation - dynamic trust based on behavior Federation operates strictly at the trust domain level and only affects identity verification. ## Why federation alone is often insufficient In theory, SPIFFE federation is clean and well-defined. In practice, teams encounter issues when: - trust relationships grow faster than policy - bundles become stale or poorly governed - verification is decoupled from enforcement - identity context disappears after the handshake These are not flaws in the specification. They are consequences of how identity is applied in real systems. Federation is most effective when verification and enforcement remain tightly coupled. ## Looking ahead The SPIFFE federation specification provides a precise, interoperable foundation for extending workload identity across administrative boundaries. What it intentionally leaves open is how federation is enforced, constrained, and observed at runtime. In a follow up post, we will explore how Riptides approaches federation, enforces it below the application layer, and how runtime context changes the security properties of federated workload identity. --- ## Zero-Touch Secrets: On-The-Wire Injection of Vault-Sourced Credentials - URL: https://blog.riptides.io/vault-credentials-on-the-wire-riptides - Published: 2026-02-02 - Author: Sebastian Toader - Category: Federation - Tags: federation, non-human identity, vault, openbao In the previous post, [Secure OpenAI API Key delivery with Riptides](/blog/ritptides-openai-apikeys), we demonstrated how Riptides securely delivers short-lived OpenAI API keys sourced from Vault/OpenBao to AI agents using kernel-enforced `sysfs` files. That approach already eliminates static secrets, reduces blast radius, and removes the need for Vault clients or tokens in workloads. This model provides several key advantages: - A dramatically reduced blast radius compared to static keys - A clean separation between identity, access policy, and application logic - Strong isolation between co-located workloads, enforced at the Linux kernel level - A familiar consumption model for developers (read a credential, use it) This post goes further. Instead of delivering credentials to workloads, **Riptides injects Vault-sourced credentials directly into outbound requests in kernel space at the moment they are sent**. API keys and cloud credentials never appear in the application user space; not in files nor in configuration. By combining Vault/OpenBao as the system of record for short-lived credentials with on-the-wire injection model of Riptides, organizations can enforce **identity-based access to OpenAI and cloud APIs** with zero secret handling in workloads, while preserving compatibility with existing applications. ## Existing Riptides capability: secretless cloud credential injection Before introducing Vault, Riptides already supported **on-the-wire injection of temporary cloud credentials** that it provisions itself, based on workload identity. These credentials are: - Short-lived - Issued on demand - Never stored or managed by developers or operators You can learn more about this capability in these posts: - [OCI authentication using SPIFFE-based workload identity](/blog/secretless-oci-authentication-with-spiffe-based-workload-identity) - [On-demand credentials for AI assistants on GCP](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp) - [On-the-wire injection for secretless AWS Bedrock access](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) In these scenarios, **Riptides itself provisions temporary cloud credentials** and injects them directly into outbound requests at write time, in kernel space. - No secrets. - No SDK changes. - No sidecars. ## Why Vault / OpenBao changes the equation The injection of Vault/OpenBao-sourced credentials does **not replace existing cloud credential injection Riptides capability**; it complements it. Many organizations already rely on **Vault or OpenBao** as their central system for issuing: - Short-lived **cloud provider credentials** - **Database credentials** (both static and dynamic) - API keys for internal and external services For these teams, replacing Vault is neither desirable nor realistic. This is where the approach presented in this post comes in. By integrating Vault/OpenBao with Riptides via [tokenex](https://github.com/riptideslabs/tokenex), organizations can now: - Keep **Vault/OpenBao as the credential authority** - Continue using existing Vault policies, TTLs, and audit logs - Eliminate secret distribution and handling on the infrastructure where workloads run - Apply **the same on-the-wire injection model** to credentials sourced from Vault In other words, if you already use Vault to issue credentials, you can now **consume those credentials without ever exposing them to workloads**. ## From secure delivery to full elimination of secrets in workloads The sysfs-based approach presented previously already offers strong guarantees. But it still involves **materializing credentials in user space**, even if briefly and under kernel control. On-the-wire injection takes the final step. With this model: - Credentials are fetched from Vault/OpenBao - Authentication to Vault/OpenBao is **JWT-based and tokenless** - Credentials are injected **directly into outbound requests in kernel space on demand** - The workload never sees, stores, or processes the credential We refer to this as **on-the-write credential injection**. ## How it works at a high level 1. **A workload initiates an outbound request** For example: - An AI agent calling OpenAI - A service accessing AWS APIs 2. **Riptides intercepts the request in kernel space** No application changes. No SDK wrapping. 3. **Credentials are requested from Vault/OpenBao** - No Vault tokens are stored or distributed - Vault policies remain fully in control 4. **Credentials are injected into the request at write time** - OpenAI API keys are added to HTTP headers - Cloud credentials are injected in the appropriate protocol-specific form 5. **The request leaves the node authenticated** The workload itself remains entirely unaware of the credential. ## What we’ll show next In the remainder of this post, we’ll demonstrate how: - **OpenAI API keys issued by Vault/OpenBao** can be injected on the wire using Riptides - **AWS credentials sourced from Vault/OpenBao** can be injected into cloud API requests using Riptides > **Prerequisites** > This guide assumes: > - Vault or OpenBao is already running > - JWT authentication is configured in Vault/OpenBao to trust the Riptides control plane as an OIDC issuer > - The OpenAI secrets engine plugin is enabled and configured > - The AWS secrets engine plugin is enabled and configured > - Roles for generating OpenAI API keys and AWS credentials are already defined ## On-the-wire injection of OpenAI API keys sourced from Vault/OpenBao ### Define the external OpenAI service in Riptides ```yaml apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: openai-api namespace: riptides-system spec: addresses: - address: api.openai.com port: 443 labels: app: openai-api external: true ``` This Riptides `Service` object declares OpenAI as an **external dependency**. It enables Riptides to apply identity-aware egress policies when workloads communicate with the OpenAI API. ### Configure Vault/OpenBao as a credential source ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialSource metadata: name: vault-openai-apikeys namespace: riptides-system spec: vault: address: http://localhost:8200 jwtAuthMethodPath: jwt role: path: openai/creds/my-role audience: ["vault"] type: token: source: api_key ``` This `CredentialSource` tells Riptides: - Which Vault/OpenBao instance to connect to - Which JWT authentication role to use - Which secret path to read - That the credential should be treated as a **bearer token** The OpenAI API key behaves as a bearer token and must be sent in the HTTP request header as: ```http Authorization: Bearer ``` By setting the credential `type` to `token`, Riptides activates its built-in bearer token injection mechanism. The `source` field specifies which attribute in the Vault/OpenBao response contains the token value. Riptides exchanges a workload identity JWT for a short-lived OpenAI API key, without requiring Vault tokens or SDKs inside the workload. ### Define the workload identity for the AI agent ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: ai-agent namespace: riptides-system spec: scope: agent: id: workloadID: ai-agent selectors: - process:name: ``` This object binds a **SPIFFE-based workload identity** to a specific process. Only the matching process is treated as the AI agent and is allowed to: - Authenticate to Vault/OpenBao - Receive OpenAI API keys - Use those credentials when calling the OpenAI API ### Bind the credential source to the workload ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialBinding metadata: name: vault-openai-apikeys-binding namespace: riptides-system spec: workloadID: ai-agent credentialSource: vault-openai-apikeys propagation: injection: selectors: - app: openai-api ``` This `CredentialBinding` connects the AI agent identity with the Vault credential source and defines **how the credential is applied**. In this case, Riptides injects the OpenAI API key directly into outbound requests sent to the OpenAI API. ## On-the-wire injection of AWS credentials sourced from Vault/OpenBao ### Define the external AWS service in Riptides ```yaml apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: aws-api namespace: riptides-system spec: addresses: - address: *.amazonaws.com port: 443 labels: app: aws-api external: true ``` This `Service` object declares the AWS API as an **external dependency**, allowing Riptides to apply identity-aware egress controls to AWS API calls. ### Configure Vault/OpenBao as an AWS credential source ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialSource metadata: name: vault-aws-creds namespace: riptides-system spec: vault: address: http://localhost:8200 jwtAuthMethodPath: jwt role: path: aws/creds/my-role audience: ["vault"] type: aws: {} ``` By setting the credential `type` to `aws`, Riptides activates its built-in AWS credential injection mechanism. Under the hood, this uses the [`libsigv4`](https://github.com/riptideslabs/libsigv4) library to sign AWS API requests in kernel space. ### Define the workload identity for the AWS client ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: aws-cli namespace: riptides-system spec: scope: agent: id: workloadID: aws-cli selectors: - process:name: aws ``` This workload identity applies specifically to the AWS CLI process. ### Bind the AWS credential source to the workload ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialBinding metadata: name: vault-aws-creds-binding namespace: riptides-system spec: workloadID: aws-cli credentialSource: vault-aws-creds propagation: injection: selectors: - app: aws-api ``` With this binding in place, Riptides injects Vault-issued AWS credentials directly into outbound AWS API requests without exposing credentials to user space, environment variables, or configuration files. ## Key takeaways Riptides supports **two complementary models** for secretless access: - **Riptides-provisioned temporary credentials injected on the wire** Ideal when you want Riptides to directly issue and inject cloud credentials. - **Vault/OpenBao-sourced credentials injected on the wire** Ideal when Vault is already your source of truth for cloud, database, or API credentials. Both approaches share the same core properties: - No long-lived secrets - Kernel-level enforcement - Strong workload isolation - Minimal operational overhead - Clear auditability and policy control On-the-wire injection of credentials represents the **most secure end state**: credentials never exist in workload user space at all. And when that model isn’t feasible for business or technical reasons, the **sysfs-based delivery approach** remains available, still far more secure than traditional methods, and fully managed by Riptides at the kernel level. Together, these options give organizations a **practical, incremental path** toward eliminating secrets from modern AI and cloud-native systems. With these takeaways in mind, the next question is how this model fits into your existing security architecture. ## Who should use which model? Both credential injection models serve different organizational needs and maturity levels. **Use Riptides provisioned temporary credentials if:** - You want the simplest path to secretless access for cloud and AI APIs - You are not already standardized on Vault/OpenBao - You prefer Riptides to handle the credential lifecycle end-to-end - You want an immediate reduction in secret sprawl with minimal platform changes This model is ideal for teams adopting secretless infrastructure for the first time or for greenfield platforms. **Use Vault/OpenBao-sourced credentials if:** - Vault/OpenBao is already your system of record for credentials - You issue short-lived cloud, database, or API credentials via Vault/OpenBao today - You want to preserve existing policies, audit trails, and governance controls - You need to consume credentials that Riptides cannot provision natively, but that Vault/OpenBao supports via a secrets engine plugin This model is ideal for enterprises with mature security programs that want to **eliminate secret handling at runtime without disrupting existing Vault/OpenBao workflows**. Both approaches share the same core guarantees: identity-based access, short-lived credentials, kernel-level enforcement, and zero secret distribution to developers or operators. --- ## Supplying short-lived OpenAI API keys to AI agents with Riptides - URL: https://blog.riptides.io/ritptides-openai-apikeys - Published: 2026-01-26 - Author: Sebastian Toader - Category: Credentials - Tags: federation, credentials, openai, vault, openbao, tokenex We are entering the era of **agentic AI**, where AI agents are no longer experimental tools or isolated assistants. They are becoming core components of modern software systems, executing tasks, making decisions, and interacting with external services on behalf of users and organizations. As adoption accelerates, enterprises are rapidly embedding AI agents into business-critical workflows. Under the hood, most of these agents rely on **GenAI platform APIs**, with **OpenAI** being the most widely used provider today. To access these APIs, AI agents authenticate using **API keys**. This same pattern applies to other major providers such as **Mistral** and **Anthropic Claude**, and is likely to remain common across the broader GenAI ecosystem. ### The core problem: long-lived static API keys OpenAI API keys are: - **Long-lived** - **Static** - **Highly sensitive** This model was acceptable when only a small number of centrally managed applications accessed AI APIs. It breaks down in an environment where large numbers of AI agents are deployed dynamically across clusters, regions, and teams. Long-lived static API keys introduce several systemic issues: - **Security risk**: a leaked key grants broad access until manually revoked - **Operational burden**: rotation, revocation, and distribution must be managed continuously - **Compliance challenges**: static credentials conflict with zero-trust, least-privilege, and modern audit requirements In practice, these keys often end up embedded in configuration files, CI pipelines, container images, or environment variables, significantly increasing the blast radius of any compromise. ### Vault and OpenBao help — but not enough HashiCorp Vault and OpenBao address part of this problem by supporting **dynamic OpenAI API keys** through a [secrets engine](https://www.hashicorp.com/en/blog/managing-openai-api-keys-with-hashicorp-vault-s-dynamic-secrets-plugin). With this approach: - API keys are generated **on demand** - Keys are **short-lived** - Vault/OpenBao automatically **revokes keys upon expiration** This is a major improvement over static secrets. Credentials are no longer perpetual, and the window of exposure is dramatically reduced. However, this does not fully solve the problem. Even temporary OpenAI API keys must still be: - Securely delivered to the AI agent - Protected from other workloads running on the same infrastructure - Refreshed seamlessly as they expire To achieve this, developers typically need to: - Integrate Vault/OpenBao SDKs into every AI agent - Provision credentials that allow agents to authenticate to Vault/OpenBao - Implement renewal, retry, and error-handling logic As the number of agents grows, this approach becomes increasingly complex and fragile. > According to Gartner: > *[Gartner predicts that 40% of enterprise applications will feature task-specific AI agents by 2026, up from less than 5% in 2025.](https://www.gartner.com/en/newsroom/press-releases/2025-08-26-gartner-predicts-40-percent-of-enterprise-apps-will-feature-task-specific-ai-agents-by-2026-up-from-less-than-5-percent-in-2025)* At this scale, secret distribution and credential management are no longer application-level concerns — they become **platform-level security challenges**. ## How Riptides solves this Riptides addresses this challenge by starting from a different premise: **identity must be the root of trust**. ### Identity first, secrets second With Riptides: - Every workload runs with a **verifiable, SPIFFE-based workload identity** - Identity is **enforced at runtime**, not inferred from configuration - Access decisions are based on **who the workload is**, not what secrets it happens to possess ### Exchanging identity for OpenAI API keys Riptides relies on [tokenex to access Vault and OpenBao](/blog/tokenex-adds-vault-openbao-support-exchanging-id-tokens-jwts-for-secrets-without-static-credentials) secrets through their native **JWT authentication mechanisms**, exchanging workload identity tokens for secrets without issuing or storing any Vault/OpenBao tokens. Using [tokenex](https://github.com/riptideslabs/tokenex), Riptides: 1. Issues **ID tokens (JWTs)** to verified workloads 1. Encodes the workload’s **SPIFFE identity** into those tokens 1. Exchanges the ID token for a **short-lived OpenAI API key** via Vault or OpenBao This ensures that: - Only **authenticated and authorized workloads** can obtain OpenAI credentials - OpenAI API keys are **temporary by default** - No static OpenAI keys are embedded in application code, configuration, or images ### Secure delivery enforced by the kernel Riptides retrieves short-lived OpenAI API keys from Vault/OpenBao on demand and places them into a dedicated `sysfs` **file provided by the Riptides Linux kernel module, ensuring that read access is enforced at the kernel level**. Key properties of this approach: - Access to the sysfs file is **enforced in the Linux kernel**, not by application logic - Only the workload bound to the expected **SPIFFE identity** is authorized to read the file’s contents - Other processes on the same node, even if co-located, are prevented from accessing the secret From the AI agent’s perspective, consumption is trivial: - The API key is read via a **simple file operation** - No Vault/OpenBao SDK is required - No custom secret refresh logic is needed This makes secret consumption as simple **as a file read**, while ensuring that access control is **stronger than environment variables or in-process memory**, and cannot be bypassed by misconfiguration or code changes. ### What this enables By combining verifiable workload identity, short-lived credentials, and kernel-level enforcement, Riptides provides: - **Just-in-time issuance and delivery of short-lived OpenAI API keys**, reducing credential exposure windows - **Zero long-lived secrets on the host**: neither developers nor operators ever provision, store, or rotate API keys on the infrastructure where AI agents run - **Strong, kernel-enforced isolation between co-located workloads**, limiting blast radius and preventing lateral access to credentials - **Clear separation of identity, access policy, and application logic**, enabling consistent enforcement and auditable controls Together, these properties align with least privilege, zero trust, and defense-in-depth principles, while simplifying compliance with security frameworks, and internal audit requirements. ## Setting this up with Riptides This section walks through how to configure Vault/OpenBao and Riptides so that AI agents can securely obtain short‑lived OpenAI API keys without embedding Vault clients, handling Vault tokens, or managing secrets directly in application code. The configuration is presented step by step, with each snippet accompanied by an explanation of **what it does**, **why it is needed**, and **how it fits into the overall flow**. > **Prerequisites** > This guide assumes: > - Vault or OpenBao is already running > - JWT authentication is configured in Vault/OpenBao to trust the Riptides Control Plane as an OIDC issuer > - The OpenAI secrets engine plugin is installed We do not cover JWT auth configuration in detail here. A complete example can be found in this [post](/blog/tokenex-adds-vault-openbao-support-exchanging-id-tokens-jwts-for-secrets-without-static-credentials) For instructions on installing the OpenAI secrets engine plugin, see: https://github.com/gitrgoliveira/vault-plugin-secrets-openai > **Note** > In all examples below, if you are using HashiCorp Vault instead of OpenBao, replace the `bao` CLI with `vault`. ### Enable the OpenAI secrets engine ```bash bao secrets enable openai ``` This command enables the OpenAI secrets engine at the `openai/` path. Once enabled, Vault/OpenBao can dynamically generate OpenAI API keys instead of relying on static, long‑lived credentials. ### Configure the OpenAI secrets engine ```bash bao write openai/config \ admin_api_key="$OPENAI_ADMIN_KEY" \ admin_api_key_id="$OPENAI_ADMIN_KEY_ID" \ organization_id="$OPENAI_ORG_ID" ``` Here you configure the secrets engine with an **administrative OpenAI API key**. This key is used *only* by Vault/OpenBao to create and revoke project‑scoped, short‑lived API keys on behalf of workloads. This administrative key never leaves Vault/OpenBao and is not exposed to AI agents. ### Create a role for generating OpenAI API keys ```bash bao write openai/roles/my-role \ project_id="$OPENAI_PROJ_ID" \ service_account_name_template="vault-{{.RoleName}}-{{.RandomSuffix}}" \ ttl=15m \ max_ttl=1h ``` This role defines **how Vault/OpenBao generates OpenAI API keys**: - Which OpenAI project the keys belong to - How the backing OpenAI service account is named - The default lifetime (`ttl`) and maximum lifetime (`max_ttl`) of issued keys Every API key generated through this role is **short‑lived by design**, limiting blast radius and exposure. ### Create a policy allowing access to generated API keys ```bash bao policy write read-openai-apikeys -< path: openai/creds/my-role audience: ["vault"] ``` This `CredentialSource` tells Riptides: - Which Vault/OpenBao instance to contact - Which JWT auth role to use - Which secret path to read Riptides will exchange a workload identity JWT for a short‑lived OpenAI API key, without requiring Vault tokens or SDKs in the workload. ### Define the workload identity for the AI agent ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: ai-agent namespace: riptides-system spec: scope: agent: id: workloadID: ai-agent selectors: - process:name: [] ``` This object binds a **SPIFFE-based workload identity** to a specific process. Only the matching process is treated as the AI agent and is allowed to: - Authenticate to Vault/OpenBao - Receive OpenAI API keys - Use those credentials when calling the OpenAI API ### Bind the credential source to the workload ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialBinding metadata: name: vault-openai-apikeys-binding namespace: riptides-system spec: workloadID: ai-agent credentialSource: vault-openai-apikeys propagation: sysfs: {} ``` This `CredentialBinding` connects the AI agent identity with the Vault credential source and specifies **how the credential is delivered**. By selecting `sysfs`, Riptides exposes the secret through a our **Linux kernel module managed file**. ### Discover the sysfs file path ```bash kubectl get credentialbindings.core.riptides.io -n riptides-system vault-openai-apikeys-binding -o yaml ``` This command shows the resolved credential binding, including the exact sysfs file path where the OpenAI API key is exposed. ### Example output ```yaml status: state: OK sysfs: files: - path: /sys/module/riptides/credentials/2e08019a-a435-55e3-9ba7-f01c09b9fc2b/crb-vault-openai-apikeys-binding/vault.json type: CREDENTIAL ``` The file path shown above is created and managed by the **Riptides Linux kernel module**. Only the authorized workload can read its contents. ### Contents of the sysfs credential file ```json { "api_key": "sk-svcacct-V8rgR1rZ-MVmeVUR-....", "api_key_id": "key_1Nj...", "service_account": "vault-my-role-...", "service_account_id": "user-gLNLYNq....." } ``` This file contains the short‑lived OpenAI API key and associated metadata. The key exists only for its configured TTL and is revoked automatically by Vault/OpenBao. ### Validate the API key ```bash curl https://api.openai.com/v1/models \ -H "Authorization: Bearer " \ -H "OpenAI-Organization: " \ -H "OpenAI-Project: " ``` Output: ```json { "object": "list", "data": [ { "id": "gpt-4-0613", "object": "model", "created": 1686588896, "owned_by": "openai" }, { "id": "gpt-4", "object": "model", "created": 1687882411, "owned_by": "openai" }, { "id": "gpt-3.5-turbo", "object": "model", "created": 1677610602, "owned_by": "openai" }, { "id": "gpt-5.2-codex", "object": "model", "created": 1766164985, "owned_by": "system" }, { "id": "gpt-4o-mini-tts-2025-12-15", "object": "model", "created": 1765610837, "owned_by": "system" }, { "id": "gpt-realtime-mini-2025-12-15", "object": "model", "created": 1765612007, "owned_by": "system" }, { "id": "gpt-audio-mini-2025-12-15", "object": "model", "created": 1765760008, "owned_by": "system" }, { "id": "chatgpt-image-latest", "object": "model", "created": 1765925279, "owned_by": "system" } ... ... ] } ``` This request demonstrates that the API key retrieved via Riptides and Vault/OpenBao works as expected. ### What the AI agent needs to do The AI agent simply reads the `api_key` field from the sysfs file and uses it when calling the OpenAI API. - No Vault/OpenBao integration is required - No long‑lived secrets are present on the host - Access is enforced at the Linux kernel level This provides a **simple, secure, and auditable** way for AI agents to consume GenAI APIs at scale. ## Why it matters As AI agents become a foundational building block of modern systems, **credential handling must evolve**. Riptides enables an operating model where: - AI agents never hold long-lived secrets - OpenAI API keys are issued **just in time** and revoked automatically - Identity is the primary security primitive - Security improves as systems scale, rather than degrading This approach: - Reduces blast radius - Simplifies development and operations - Aligns with zero-trust and least-privilege principles - Makes large-scale agentic systems **operationally sustainable** In an ecosystem rapidly moving toward autonomous workloads, **identity-first, short-lived credentials are no longer optional — they are essential**. Riptides brings this model to GenAI workloads today, starting with OpenAI and extending naturally to the broader AI platform landscape. In this post, we showed how short-lived OpenAI API keys can be securely delivered to AI agents using identity and kernel-level enforcement. In the next post, we’ll go one step further and show how Riptides can inject these keys **directly into outbound requests**, so workloads never read or handle API keys at all. --- ## tokenex adds Vault & OpenBao support: Exchanging ID tokens (JWTs) for secrets without static credentials - URL: https://blog.riptides.io/tokenex-adds-vault-openbao-support-exchanging-id-tokens-jwts-for-secrets-without-static-credentials - Published: 2026-01-19 - Author: Sebastian Toader - Category: Credentials - Tags: tokenex, vault, openbao, credentials ## Introducing Vault & OpenBao support in **[tokenex](https://github.com/riptideslabs/tokenex)** open source library Since its first release, **[tokenex](https://github.com/riptideslabs/tokenex)** has focused on **identity-first credential acquisition** exchanging short-lived identity tokens (JWT) for cloud credentials just-in-time, without baking secrets into code, files, or images. Today, we’re extending that model beyond cloud IAM. We’re excited to announce **native support for HashiCorp Vault and OpenBao as credential providers in [tokenex](https://github.com/riptideslabs/tokenex)**. With this new capability, **[tokenex](https://github.com/riptideslabs/tokenex)** can exchange **ID tokens (JWTs)** for secrets stored in **Vault or OpenBao**, using their built-in **JWT authentication** flows; no static Vault tokens, no long-lived credentials, and no manual secret distribution. This addition makes Vault and OpenBao **first-class participants in tokenex’s identity-driven workflow**, allowing applications to retrieve both: * cloud-native credentials (AWS, GCP, Azure, OCI), and * infrastructure or application secrets (databases, APIs, internal services) using the **same identity-based access pattern**. In the next section, we’ll explain **why this integration matters**, how it complements tokenex’s existing capabilities, and what it unlocks for teams building secure, scalable systems. ## Why we built this feature **[tokenex](https://github.com/riptideslabs/tokenex)** was originally created to provide a **unified, consistent interface for obtaining and refreshing cloud credentials** across multiple providers including **AWS, GCP, Azure, and OCI — by exchanging identity tokens for temporary credentials** and streaming those credentials over a channel so applications never have to implement bespoke refresh logic themselves. It already supports: * **AWS**: Exchanging ID tokens for temporary session credentials via Workload Identity Federation * **GCP**: Exchanging ID tokens for access tokens via federated identity * **Azure**: Exchanging ID tokens for access tokens using OAuth and Entra ID * **OCI**: Exchanging ID tokens for User Principal Session Tokens (UPST) * **Generic token passthrough** and Kubernetes secret watching for flexible integrations This model enables developers to write **single credential consumption logic**, regardless of provider, while **[tokenex](https://github.com/riptideslabs/tokenex)** handles: * Identity token exchange * Credential refresh and rotation * Streaming updates to applications While this already removed much of the complexity around cloud authentication, there was still a clear gap: **enterprise secrets management**. Many real‑world systems don’t only need cloud API credentials. They also rely on **database credentials, API keys, and other sensitive configuration**, which are typically managed by platforms like **HashiCorp Vault** or **OpenBao**. These systems excel at issuing **dynamic, short‑lived secrets**, enforcing least privilege, and providing strong auditability, but consuming those secrets safely still requires glue code and identity plumbing. By adding **Vault/OpenBao as a first‑class credentials provider**, **[tokenex](https://github.com/riptideslabs/tokenex)** now allows workloads to: * Exchange a **JWT (ID token)** for secrets using **Vault/OpenBao JWT authentication** * Retrieve **static or dynamic secrets** (for example, database credentials) without embedding Vault‑specific logic With this addition, **[tokenex](https://github.com/riptideslabs/tokenex)** evolves from a cloud‑credential helper into a **general identity‑to‑secret exchange layer**. Applications authenticate once using identity, and **[tokenex](https://github.com/riptideslabs/tokenex)** handles the rest, regardless of whether the target is a cloud API or a centralized secrets manager. This feature is a natural extension of tokenex’s core philosophy: **minimize credential handling in applications, centralize trust in identity, and let platforms issue short‑lived secrets on demand.** ## Demo: Using [tokenex](https://github.com/riptideslabs/tokenex) with OpenBao to publish PostgreSQL credentials ### 1. Deploy OpenBao and PostgreSQL In this step, we deploy **OpenBao** and **PostgreSQL** as containers using **Docker Compose**. **OpenBao configuration**: Create a file named `vault.hcl` with the following contents: ```hcl ui = true storage "file" { path = "/vault/data" } listener "tcp" { address = "0.0.0.0:8200" tls_disable = 1 } api_addr = "http://127.0.0.1:8200" cluster_addr = "http://127.0.0.1:8201" plugin_auto_register = true plugin_auto_download = true plugin_download_behavior = "fail" plugin_directory = "/opt/openbao/plugins" ``` > **Note**: > For simplicity, TLS is disabled and file-based storage is used. > This configuration is suitable for demos and local development only. **Docker Compose setup** Create a `docker-compose.yml` file: ```yaml services: openbao: restart: unless-stopped image: openbao/openbao:2.4 container_name: openbao ports: - "8200:8200" cap_add: - IPC_LOCK volumes: - ./vault.hcl:/vault/config/vault.hcl:ro - vault-data:/vault/data environment: VAULT_ADDR: "http://127.0.0.1:8200" command: vault server -log-level=debug -config=/vault/config/vault.hcl postgres: image: postgres:18 container_name: postgres restart: unless-stopped shm_size: 128mb environment: POSTGRES_USER: postgres POSTGRES_PASSWORD: mysecretpassword ports: - "5432:5432" volumes: - pg-data:/var/lib/postgresql volumes: vault-data: pg-data: ``` ### 2. Initialize and unseal OpenBAO ```bash docker exec openbao bao operator init -format=json > vault-init.json docker exec openbao bao operator unseal $(jq -r '.unseal_keys_b64[0]' vault-init.json) docker exec openbao bao operator unseal $(jq -r '.unseal_keys_b64[1]' vault-init.json) docker exec openbao bao operator unseal $(jq -r '.unseal_keys_b64[2]' vault-init.json) ``` ### 3. Configure JWT authentication in OpenBao ```bash # Enable JWT authentication method docker exec -e VAULT_TOKEN=$(jq -r '.root_token' vault-init.json) \ openbao bao auth enable jwt ``` ```bash # Configure the JWT authentication method docker exec -e VAULT_TOKEN=$(jq -r '.root_token' vault-init.json) \ openbao bao write auth/jwt/config \ bound_issuer="" \ oidc_discovery_url="" \ oidc_client_id="" \ oidc_client_secret="" ``` * `bound_issuer`: `` – Issuer URL of your OIDC provider * `oidc_discovery_url`: `` – URL for OIDC metadata discovery (keys, endpoints) ```bash # Create a named role 'secret-reader' that authorizes JWTs with specific subject and audience claims # Assign the 'read-secrets' policy to this role docker exec -e VAULT_TOKEN=$(jq -r '.root_token' vault-init.json) openbao \ bao write auth/jwt/role/secret-reader \ user_claim="sub" \ bound_audiences="" \ bound_subject="" \ token_policies="read-secrets" \ token_type="service" \ expiration_leeway=150 \ not_before_leeway=150 \ role_type="jwt" ``` * `bound_audiences`: `` – Expected `aud` claim in JWT tokens * `bound_subject`: `` – Expected `sub` claim in JWT tokens ```bash # Create the 'read-secrets' policy # Grants read access to the 'pg-dyn-dbuser' dynamic DB credential and the 'pg-dbuser1' static DB credential docker exec -i -e VAULT_TOKEN=$(jq -r '.root_token' vault-init.json) openbao \ bao policy write read-secrets -<") // Create the Vault credentials provider vaultProvider, err := vault.NewCredentialsProvider(ctx, logger, "http://localhost:8200", nil) if err != nil { logger.Error(err, "Failed to create Vault credentials provider") } dynDBCredsChan, err := vaultProvider.GetCredentials( ctx, idTokenProvider, vault.WithJWTAuthMethodPath("jwt"), vault.WithJWTAuthRoleName("secret-reader"), vault.WithSecretFullPath("database/creds/pg-dyn-dbuser"), ) if err != nil { logger.Error(err, "Failed to get dynamic database credentials") } // Process dynamic database credentials wg.Add(1) credentialsConsumer(logr.NewContext(ctx, logger.WithValues("secret_engine", "database", "credential_type", "dynamic", "secret_path", "/creds/pg-dyn-dbuser")), &wg, dynDBCredsChan, logDBCreds) staticDBCredsChan, err := vaultProvider.GetCredentials( ctx, idTokenProvider, vault.WithJWTAuthMethodPath("jwt"), vault.WithJWTAuthRoleName("secret-reader"), vault.WithSecretFullPath("database/static-creds/pg-dbuser1"), ) if err != nil { logger.Error(err, "Failed to get static database credentials") } // Process static database credentials wg.Add(1) credentialsConsumer(logr.NewContext(ctx, logger.WithValues("secret_engine", "database", "credential_type", "static", "secret_path", "/static-creds/pg-dbuser1")), &wg, staticDBCredsChan, logDBCreds) // Wait for all goroutines to finish wg.Wait() } func credentialsConsumer(ctx context.Context, wg *sync.WaitGroup, credsChan <-chan credential.Result, credsLogger func(logr.Logger, *credential.VaultSecret)) { go func() { defer wg.Done() logger := logr.FromContextOrDiscard(ctx) for { select { case creds, ok := <-credsChan: if !ok { logger.Info("credentials channel closed") return } if creds.Err != nil { logger.Error(creds.Err, "Error receiving credentials", errors.GetDetails(creds.Err)...) return } dbSecret, ok := creds.Credential.(*credential.VaultSecret) if !ok { logger.Error(errors.New("invalid credential type"), "expected *credential.VaultSecret") return } credsLogger(logger, dbSecret) case <-ctx.Done(): log.Println("Context cancelled, shutting down credentials handler") return } } }() } func logDBCreds(logger logr.Logger, dbSecret *credential.VaultSecret) { // Database secrets typically contain username and password username := dbSecret.Data["username"].(string) password := dbSecret.Data["password"].(string) logger.Info("credential", "username", username, "password", password) } ``` **Sample output**: ```bash 0 info | 11:00:29.398131 | Press Ctrl+C to stop... caller=main.go:41 0 info | 11:00:29.428805 | credential caller=main.go:220 secret_engine=database credential_type=static secret_path=/static-creds/pg-dbuser1 username=dbuser1 password=Qj-6l6OWghtonpucH6Vd 2 info | 11:00:29.428858 | Published Vault secret logger=vault_credentials caller=creds.go:245 secret_path=database/static-creds/pg-dbuser1 expiresAt="2026-01-15 13:51:17.428782 +0100 CET m=+6648.032605334" 2 info | 11:00:29.428879 | Using RefreshOn time from credentials logger=vault_credentials caller=creds.go:252 refreshOn="2026-01-15 13:51:22.428782 +0100 CET m=+6653.032605334" 1 info | 11:00:29.428885 | Scheduling credential refresh logger=vault_credentials caller=creds.go:260 refreshIn=1h50m52.999899041s refreshBuffer=0s secret_path=database/static-creds/pg-dbuser1 2 info | 11:00:29.446979 | Published Vault secret logger=vault_credentials caller=creds.go:245 secret_path=database/creds/pg-dyn-dbuser expiresAt="2026-01-15 12:15:29.446969 +0100 CET m=+900.050791918" 0 info | 11:00:29.446989 | credential caller=main.go:220 secret_engine=database credential_type=dynamic secret_path=/creds/pg-dyn-dbuser username=v-jwt-ript-pg-dyn-d-QKRDQydUME1zlQtgdyY6-1768474829 password=t7gZd5h-7BwZr2lB0iFO 1 info | 11:00:29.447002 | Scheduling credential refresh logger=vault_credentials caller=creds.go:260 refreshIn=11m57.937258345s refreshBuffer=3m2.06274128s secret_path=database/creds/pg-dyn-dbuser ``` **What we see in the output**: * For the **static PostgreSQL user** `dbuser1`, the password is managed and rotated by the OpenBAO database secrets engine. The user was originally created with the password `pwd1`. OpenBAO rotates this password **only when the rotation time is reached**, at `expiresAt="2026-01-15 13:51:17.428782 +0100 CET"`. Because this is a static role, **[tokenex](https://github.com/riptideslabs/tokenex) cannot retrieve a new password before the rotation occurs**. It must wait until OpenBAO performs the rotation. Once the password has been rotated, Tokenex re-fetches and publishes the updated credentials shortly after, at `refreshOn="2026-01-15 13:51:22.428782 +0100 CET"`. * For the **dynamic PostgreSQL user** `v-jwt-ript-pg-dyn-d-QKRDQydUME1zlQtgdyY6-1768474829`, OpenBAO creates a brand-new database user with a unique password. This credential is short-lived and expires in approximately 15 minutes. Unlike static credentials, **[tokenex](https://github.com/riptideslabs/tokenex) can proactively request a new dynamic credential before the current one expires**. In this example, a refresh is scheduled at `refreshIn=11m57.937258345s` with a `refreshBuffer=3m2.06274128s`, ensuring continuous access without relying on password rotation. ## Example use cases ### 1. Zero-Trust service & database access Workloads authenticate using a JWT and exchange it for secrets at runtime: * Short-lived database credentials (e.g. PostgreSQL) * Service-specific API keys or tokens * Automatically rotated by Vault/OpenBao * Fully auditable and identity-scoped No secrets in config files, CI pipelines, or environment variables. No shared credentials between services. ### 2. Trusted secret orchestrators A trusted orchestrator (Kubernetes controller, job runner, workflow engine) can: * Authenticate using its own workload identity * Use Tokenex to fetch secrets on behalf of workloads * Enforce centralized policy, intent, and approval flows * Act as a controlled trust boundary in regulated environments ## Final thoughts By combining: * **Identity-based authentication** * **Centralized secrets management** * **Short-lived credentials** * **Explicit, auditable token exchange** you end up with systems that are: * Easier to reason about * Harder to misuse * Safer by default **[tokenex](https://github.com/riptideslabs/tokenex)** doesn’t replace Vault or OpenBao — it **connects them seamlessly to modern identity systems**, allowing secrets to be accessed only when and where identity has been verified. --- ## Secretless OCI Authentication with SPIFFE-based workload identity - URL: https://blog.riptides.io/secretless-oci-authentication-with-spiffe-based-workload-identity - Published: 2026-01-12 - Author: Sebastian Toader - Category: Federation - Tags: federation, non-human identity, oci ## On-the-wire credential injection: Secretless OCI access example In our view, **every workload must have a verifiable identity**, and only workloads with a **trusted, cryptographically provable identity** should be allowed to access protected resources, **without relying on secrets**. This approach is essential to defend against modern threats, where increasingly sophisticated attacks routinely target static credentials, configuration files, and long-lived keys. Secrets eventually leak; identities can be continuously verified. Riptides is built around this principle: **SPIFFE-based workload identities**, enforced at runtime, with access decisions tied to who the workload is, not what secrets it happens to possess. For a deeper exploration of the problem space and why the industry must move beyond credentials, see the following posts: - [Workload Identity Without Secrets: A Blueprint for the Post‑Credential Era](/blog/workload-identity-without-secrets-a-blueprint-for-the-post-credential-era) - [The Hidden Risk in Service Mesh mTLS: When Your Sidecar Becomes a Trojan Horse](/blog/the-hidden-risk-in-service-mesh-mtls-when-your-sidecar-becomes-a-trojan-horse) - [Growing Threat of npm Supply Chain Attacks and the Runtime Fix That Stops It](/blog/growing-threat-of-npm-supply-chain-attacks) - [Shai‑Hulud 2.0: A Technical Breakdown and Why Secrets Need to Die](/blog/shai-hulud-2-0-a-technical-breakdown-and-why-secrets-need-to-die) - [When Remote Code Execution Isn’t the End — Designing for Containment](/blog/when-remote-code-execution-isnt-the-end---designing-for-containment) You can also learn how Riptides assigns trusted identities to processes at the kernel level here: [Workload Attestation and Metadata Gathering: Building Trust from the Ground Up](/blog/workload-attestation-and-metadata-gathering-building-trust-from-the-ground-up) ## Overview In this post, we show how to securely access **OCI resources without creating or managing secrets**, using the **OCI CLI** as client/example. This pattern applies to *any* client application accessing `OCI`. Riptides integrates transparently with the **OCI SDK**, enabling this capability **without requiring any application code changes**. If you’re interested in how Riptides supports other cloud providers, see: - [On-the-Wire Credential Injection: Secretless AWS Bedrock Access Example](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) - [On‑Demand Credentials: Secretless AI Assistant Example on GCP](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp) In this walkthrough, we configure secretless access for the **OCI CLI** using Riptides. Under the hood, this leverages: - Riptides’ open‑source [oci-req-signer-c](/blog/announcing-oci-req-signer-c-a-lightweight-c-library-for-oracle-cloud-request-signing) library - OCI **Workload Identity Federation** For background, see: - [Workload Identity Federation](https://www.ateam-oracle.com/workload-identity-federation) - [OCI simplifies multi‑cloud workloads with OCI IAM Workload Identity Federation](https://blogs.oracle.com/cloud-infrastructure/oci-iam-workload-identity-federation) ## The standard way: Authenticating and authorizing the client To access OCI resources, the **OCI CLI** traditionally authenticates as an **IAM user** with appropriate permissions. This is the conventional model used by most cloud SDKs and CLIs today. For this demo: - Create an IAM group called `IAMUserViewers` - Add a test IAM user to the group - Attach the following policy: ```yaml Allow group IAMUserViewers to inspect users in tenancy ``` The `OCI CLI` requires a [configuration file](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm) that specifies which user it authenticates as. This file typically contains sensitive material such as: - `key_file` (private signing key) - `pass_phrase` protecting the private key - `security_token_file` for session-based auth In this model, the client application is **directly responsible for handling credentials**. Even when carefully secured, these files become high-value targets: they must be stored somewhere, protected at rest, rotated regularly, and kept out of logs, backups, and build artifacts. This credential-centric approach is functional, but it tightly couples application execution with secret management, increasing both operational overhead and security risk. Even if all best practices are followed, this model **does not protect against supply‑chain attacks**, as discussed in the posts linked earlier. Listing users using this approach: ```shell oci iam user list | jq '.data[].name' "testuser1" "testuser2" ``` ## On‑the‑wire Credential injection with Riptides Now let’s look at the same operation in a **Riptides managed environment**. Riptides eliminates stored secrets by injecting credentials **dynamically at runtime**. The client never stores, reads, or manages OCI credentials directly. ## Prerequisites ### Register Riptides Control Plane as an external IDP To enable secretless access, OCI must be able to **trust identities issued by Riptides and map them to an OCI IAM principal**. This is done in two steps: 1. Create an OCI service user that will be impersonated 1. Configure OCI to trust Riptides as an OIDC identity provider and map workload identities to that user #### Step 1: Create an OCI service user for impersonation OCI requires a concrete IAM principal to authorize API calls. Instead of authenticating as a human user, we create a **service user** that will be impersonated using short-lived credentials. The following service user definition creates a service user in OCI Identity Domains: ```json { "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User"], "urn:ietf:params:scim:schemas:oracle:idcs:extension:user:User": { "serviceUser": true }, "userName": "testserviceuser1" } ``` **What this does**: - Declares a standard user object - Marks the user as a service user, not a human - Creates an identity that can be safely impersonated by workloads Add this service user to the `IAMUserViewers` group so it inherits the required permissions. At this point, we have an OCI principal that is allowed to access the Identity API, but **no credentials have been issued or stored yet**. #### Step 2: Trust Riptides as an OIDC Identity Provider Next, register the Riptides Control Plane as an **OIDC identity provider** in OCI. This enables OCI to trust tokens issued by Riptides and exchange them for **User Principal Session Tokens (UPSTs)**. ```json { "active": true, "allowImpersonation": true, "issuer": "https:///oidc", "name": "Token Trust JWT to UPST", "oauthClients": ["1d92..."], "publicKeyEndpoint": "https:///oidc/keys", "impersonationServiceUsers": [ { "rule": "sub eq spiffe://acme.org/oci-cli", "value": "2a86..." } ], "subjectType": "User", "type": "JWT", "schemas": [ "urn:ietf:params:scim:schemas:oracle:idcs:IdentityPropagationTrust" ] } ``` **What this configuration enables**: - `issuer`: Identifies the Riptides Control Plane as the trusted OIDC issuer - `oauthClients`: Restricts trust to a specific OAuth 2.0 client registered for the Riptides Control Plane - `publicKeyEndpoint`: Allows OCI to fetch the public keys used to verify ID tokens signed by Riptides Control Plane - `impersonationServiceUsers`: Defines how a workload identity maps to an OCI IAM service user In this example: - If the ID token contains a `sub` claim equal to `spiffe://acme.org/oci-cli` - OCI will issue temporary credentials for the service user with ID `2a86...` - `type: JWT` indicates that the trust relationship is based on JWT-formatted ID tokens **How this fits into the secretless flow**: 1. Riptides issues an ID token (JWT SVID) to a workload, containing a SPIFFE-based subject 1. OCI verifies the token using the configured trust 1. OCI exchanges the token for a short-lived UPST 1. The workload impersonates the service user without ever handling secrets At no point does the application receive, store, or manage credentials. Secrets exist, but they are managed by the platform, not the application. ### Setting up the client application with Riptides From Riptides’ perspective, the `OCI IAM service API is an external dependency`. It is not discovered automatically. To enable credential injection, Riptides must first be told **which external service the workload will communicate with**. This is done by registering the OCI Identity API as an external service in the Riptides Control Plane using a Kubernetes custom resource: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: oci-identity-api namespace: riptides-system spec: addresses: - address: identity.eu-frankfurt-1.oci.oraclecloud.com # OCI Identity REST API service endpoint port: 443 # Port the client connects to labels: app: oci-identity-api # Label for matching this service external: true # Indicates this is an external service, not managed by Riptides ``` **What this configuration does**: - Declares the **OCI Identity REST API endpoint** as a known external service - Allows Riptides to match **outbound connections** from workloads to this destination ### Defining how workloads obtain OCI credentials At this point, Riptides knows **where** the workload will connect (OCI Identity API). Next, we define **how Riptides should obtain credentials** for that workload. This is done in two steps: 1. Define *how* to obtain temporary OCI credentials 1. Bind those credentials to a specific workload identity **Step 1: Define a credential source**: A **CredentialSource** describes *how Riptides exchanges identity for temporary OCI credentials*. In this case, the source is OCI itself, using OCI IAM Workload Identity Federation. ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialSource metadata: name: oci-cred-1 namespace: riptides-system spec: oci: # Temporary credentials sourced from OCI region: eu-frankfurt-1 clientId: 1d92... # the OAUth 2.0 client id defined in OCI for the Riptides Control Plane clientSecret: idcscs-.... # the client secret defined in OCI for the Riptides Control Plane identityDomainUrl: https://idcs-......identity.oraclecloud.com # the URL of the identity domain where the impersonated OCI service user is defined tenancyOcid: ocid1.tenancy.oc1........ ``` **What this configuration defines**: - `region`: The OCI region where credentials will be issued. - `clientId / clientSecret`: OAuth 2.0 credentials used by the **Riptides Control Plane**, not the application, to interact with OCI IAM. - `identityDomainUrl`: The OCI Identity Domain where the impersonated service user is defined. - `tenancyOcid`: Identifies the OCI tenancy where authentication and authorization occur. This resource does **not** issue credentials on its own. It simply defines *how* credentials can be obtained when needed. **Step 2: Bind credentials to a workload identity**: Next, we associate the credential source with a specific workload identity using a **WorkloadCredential**: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadCredential metadata: name: oci-cli-cred-1 namespace: riptides-system spec: credentialSource: oci-cred-1 # Source of temporary credentials workloadID: oci-cli # Workload ID to get OCI temporary credentials for ``` **What this does**: - Links the `oci-cli` workload identity to the OCI credential source - Ensures that **only workloads with this identity** can obtain these credentials At this point, no credentials are issued yet; the configuration merely defines the relationship. **How it works at runtime**: When a workload with the `oci-cli` identity needs to call OCI: 1. The **Riptides Control Plane issues an ID token** for the workload. 1. The token’s `sub` claim follows the SPIFFE format: `spiffe:///`

In this demo: `spiffe://acme.org/oci-cli` 1. OCI validates this token using the previously configured trust relationship. 1. OCI exchanges the token for **short-lived credentials** impersonating the service user. 1. Riptides injects these credentials transparently into outgoing requests. 1. Credentials are **automatically refreshed** before expiration. At no point does the application: - See credentials - Store credentials - Handle rotation or expiration logic Secrets exist, but they are **entirely managed by the platform, not the workload. ### Assigning Workload IDs to processes We define **which processes can be assigned the `oci-cli` workload ID, and under what conditions**: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: oci-cli-wid namespace: riptides-system spec: scope: agent: id: # Scope: node(s) where this workload identity can be assigned workloadID: oci-cli # The workload ID assigned to matching processes selectors: - process:name: [oci] # Runtime process attribute that must match to get this identity egress: # Egress rules for credential injection - selectors: - app: oci-identity-api # Service endpoint(s) targeted by this rule credentialName: oci-cli-cred-1 # Credentials to inject, referenced from WorkloadCredential connection: tls: intercept: true # Intercept traffic and inject credentials into HTTP requests ``` **How it works**: 1. **Riptides Agent** runs on nodes and acts as the bridge between the Control Plane and the Linux kernel module. 1. Workload IDs and credentials issued by the Control Plane are **restricted to processes on the node where the agent runs** — this is controlled by the *scope* field in the *WorkloadIdentity* CR. Multiple agents can also be targeted if needed. 1. The **Linux kernel module** monitors running processes and checks their runtime attributes against the *spec.selectors* values. Only matching processes are assigned the workload ID. 1. When a process with an assigned workload ID sends a request to a service referenced in the *egress* rules, **the temporary credentials from the specified WorkloadCredential are injected directly into the request.** In this example, any process named *oci* will receive **OCI temporary credentials** automatically when sending requests to the OCI Identity service endpoint. The original HTTP request sent by `OCI CLI` to OCI Identity service endpoint is modified by Riptides' Linux kernel module as it injects the temporary credentials on the fly. The modified HTTP request requires resigning with the correct OCI signature. Our Linux kernel module accomplishes this using our [oci-req-signer-c](https://github.com/riptideslabs/oci-req-signer-c) library, implemented in portable C with Linux kernel compatibility. This enables credentials to be injected **and signed at the kernel level** just before the request is sent, ensuring full OCI authentication **without exposing keys to the client application**. **Why is this matters**: Unlike sidecars, environment variables, or SDK hooks: - Identity is bound to **process execution**, not deployment metadata - Credentials are scoped to **specific destinations** - Enforcement happens **at the kernel boundary** - Compromising the application does **not** automatically expose credentials From the application’s point of view, authentication “just works.” From a security perspective, access is tightly constrained, observable, and auditable. ## Running the Client Application The **OCI SDK still expects a configuration file**, so we provide **dummy credentials** that satisfy the SDK’s syntax requirements but are **never used for authentication**. Riptides intercepts the request, injects valid temporary credentials, and **re-signs the request transparently at runtime**. ```ini [DEFAULT] user=ocid1.user.oc1..dummy fingerprint=00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 key_file=/var/tmp/dummy_priv_key.pem tenancy=ocid1.tenancy.oc1...... region=eu-frankfurt-1 ``` The private key referenced here must be **syntactically valid**, but it has **no permissions and no security value**. At runtime, Riptides replaces it with workload-bound private key derived from the workload identity. ```shell oci --cert-bundle /sys/module/riptides/certs/ca-certificates.crt iam user list | jq '.data.[].name' "testuser1" "testuser2" "testserviceuser1 ``` **What actually happened**: - The OCI CLI believed it was using local credentials - The real credentials were **never present in files, environment variables, or process memory** - Authentication and request signing occurred **inside the kernel**, just before the request was sent - Credentials were **scoped, temporary, and automatically refreshed** ### Key points - **No credentials are ever stored** on the client machine or in configuration files. - Credentials are provided **just-in-time** for each request, minimizing the risk of leaks or misuse. - The **application workflow remains unchanged**, from the client’s perspective, authentication happens automatically and transparently. ## When the On-the-wire credential injection is not an option There may be situations where you **cannot or do not want to use on-the-wire credential injection**. In these cases, Riptides still provides credentials to the `OCI SDK` **without exposing them on the filesystem**: - The configuration file, private key file, and session token file are made available via `sysfs`. - Only processes with the **appropriate workload identity** can access these files. To disable on-the-wire injection, set `connection.tls.intercept: false` in the `WorkloadIdentity` custom resource. **Running `OCI CLI` with sysfs-based credentials**: You can point the `OCI CLI` to the Riptides managed configuration files as follows: ```shell oci oci --auth security_token --cert-bundle /sys/module/riptides/certs/ca-certificates.crt --config-file /sys/module/riptides/credentials/4e1ef9dd-fa21-513d-8505-7e9ef13b9be0/oci-cli-cred-1/oci_config iam user list | jq '.data.[].name' "testuser1" "testuser2" "testserviceuser1 ``` The **exact path** of `oci_config` can be retrieved from the `WorkloadCredential` custom resource status fields. **Why is this approach safe**: - Credentials **never appear on disk** or in environment variables - Only processes with the **matching workload identity** can read the configuration - The OCI CLI behaves normally, but secrets are **still tightly scoped and ephemeral** This provides a **fallback option** when on-the-wire injection is not feasible, without compromising the security guarantees of the platform. ## Final Thoughts At Riptides, we are strong advocates of [SPIFFE-based workload identities](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust) as the foundation for secure, scalable non‑human authentication. The benefits are concrete and measurable: - **Elimination of static keys** — no long-lived secrets to steal or rotate - **Short-lived, impersonated credentials** — automatically issued and refreshed - **Full auditability** — every action is traceable to a workload This is more than an implementation detail; it’s a security philosophy: **no static secrets, no blind trust, only verifiable workload identities at runtime.** --- ## When Remote Code Execution Isn’t the End — Designing for Containment - URL: https://blog.riptides.io/when-remote-code-execution-isnt-the-end---designing-for-containment - Published: 2026-01-06 - Author: Marton Sereg - Category: Security - Tags: SPIFFE, kernel, RCE, identity, mTLS Remote Code Execution vulnerabilities remain a persistent reality in modern software systems. Recent issues like react2shell show how even widely used frameworks can expose unexpected execution paths through subtle interactions between rendering logic and user-controlled input. Often it’s not an obvious mistake, but an edge case: deserialization quirks, parser behavior, or features used just slightly outside their intended bounds. This post isn't about pretending we can eliminate RCE entirely. It's about what happens after code execution, and why that moment often determines whether you have a contained incident or a full-blown breach. We'll walk through a concrete demo and show how a process-level, identity-first approach changes the outcome: the vulnerability still exists, the attacker still gets a shell, but lateral movement stops. At Riptides we believe that this is where cryptography becomes an enforcement mechanism rather than a theoretical control. When every workload carries a strong, SPIFFE-based identity and all service-to-service communication is authenticated and encrypted with mutual TLS, a compromised process does not automatically inherit the ability to move laterally. The attacker may have code execution, but they do not have keys, trust relationships, or identity they can reuse elsewhere. By binding access decisions to cryptographically verifiable workload identities, rather than IPs, networks, or ambient credentials, you materially reduce the blast radius of an RCE and turn what would have been a platform wide breach into a localized, containable event. ## RCE Is a Fact of Life in Modern Software There’s a comforting narrative that serious vulnerabilities mostly live in obscure libraries or poorly maintained projects. Reality keeps proving otherwise. Log4Shell was a global example of how a single feature in a ubiquitous dependency could turn into mass remote code execution overnight. Beyond that headline incident, RCEs continue to surface in template engines, deserializers, file format parsers, image processors, and administrative endpoints. The important takeaway isn’t which project was affected. It’s that RCE is a recurring property of complex systems. If your security posture assumes “this will never happen to us,” you’re implicitly betting on perfection. ## Static Scanners are not enough Static analysis and dependency scanning are table stakes. You should absolutely run SCA, patch aggressively, and enforce policy around known vulnerabilities. But scanners operate in a world of *knowns*: known CVEs, known bad versions, known patterns. They’re very good at answering the question, “Is this dependency associated with a published vulnerability?” But once an exploit has already executed, scanners are out of the loop entirely. At that point, you’re no longer dealing with a vulnerable dependency graph, you’re dealing with an attacker within your infrastructure. That’s why modern security strategies increasingly assume breach. Not because prevention is pointless, but because it’s incomplete. You need to design your runtime so that a single missed bug doesn’t automatically turn into unrestricted access. This is what zero trust is about. Not ZTNA, but the philosophy. ## What Attackers Do After RCE Is Predictable From the attacker’s perspective, RCE isn’t the end goal. It’s the entry point. The first step is often to establish a stable reverse shell. Outbound connections are easier to make reliable, more likely to bypass firewalls and NAT, and easier to blend into normal traffic. Once inside, the next phase is reconnaissance. Attackers inspect environment variables, configuration files, mounted secrets, process arguments, and filesystem layout. They probe internal DNS and try to understand what services exist and how trust is enforced. This is where implicit trust becomes dangerous. In many environments, simply being inside a container or VM grants broad access. Internal services assume callers are legitimate because they’re “on the inside.” This problem isn’t limited to legacy, perimeter-based security models. To some extent, it exists even with modern proxies and service meshes: trust is often attached to the sidecar or the proxy, not to the actual workload process. Once an attacker gains code execution, any process that can talk through that proxy may inherit the same level of trust. That’s how RCE turns into lateral movement, and lateral movement is how incidents become breaches. ## Containment Matters More Than Perfect Prevention Controls like rootless containers, AppArmor, seccomp, and read-only filesystems exist because we assume processes can be compromised. They’re designed to limit what an attacker can do on a single machine: restrict syscalls, prevent privilege escalation, and make persistence harder. These are all important, well-established layers of defense. But many real-world breaches hinge on what a compromised process can reach over the network. The critical questions become: can it open connections to internal services? Can it authenticate to databases or APIs? Does simply being “inside” grant it access to the rest of the system? In practice, lateral movement is often less about escaping the sandbox and more about reusing implicit network trust. That’s why controlling who can talk to what — and under what identity — should be a top priority in any containment strategy. ## The Missing Question: Who Is This Process? Zero trust is often discussed at the network or user level, but after an RCE the most important question is surprisingly simple: > Can this process prove who it is? But if the answer is something like these: - "yes, it’s inside this pod" - "yes, it has this IP address" - "yes, it has the API keys", you’re relying on the attacker’s favorite property: once they land anywhere, they inherit the same trust as the workload. What you want instead is: - Authentication based on identity, not location - Verification before any connection is established - A clear distinction between the legitimate workload process and everything else running alongside it If a newly spawned process can’t authenticate, it shouldn’t be able to communicate even if it’s running in the same container. ## How Riptides Changes the Post-RCE Outcome Riptides enforces identity at the process level rather than treating the pod or node as a single trusted unit. Workloads are attested from the kernel, and identities are issued to what is actually running. Internal communication is secured with mutual TLS by default, without relying on sidecars or network overlays. The practical effect is straightforward: a process that doesn’t have an attested identity cannot authenticate to internal services. So after an RCE, the attacker may still have a shell — but that shell is just another process. Without identity, it can’t connect to databases, APIs, or other workloads. The vulnerability still exists, but the blast radius collapses. Riptides doesn’t prevent RCE, but it prevents what usually follows. ## Demo: Same Vulnerability, Very Different Outcome To make this concrete, we used a simple internal app called Support-Assistant. It’s intentionally nothing special: a chat UI backed by an LLM provider, with a Postgres database used as a tool to retrieve and summarize support tickets. We added a deliberately vulnerable `/debug` endpoint that allows command execution. This is not meant to represent a realistic bug — real RCEs are subtler — but it creates the same capability: arbitrary code execution inside the backend container. ![Debug Page](../../assets/mtls-rce/mtls-screenshot-1-debugpage.png) From there, the demo follows the standard attacker playbook. We trigger a reverse shell, explore the environment, discover the database endpoint, download `psql`, and query the support tickets. Finally, we exfiltrate the data to an external endpoint. ![Remote Shell](../../assets/mtls-rce/mtls-screenshot-2-remote-shell.png) At this point, the RCE has already turned into lateral movement and data access. Then we enable Riptides enforcement for Postgres: internal connections now require identity-authenticated mTLS. The application continues to function normally, because the real backend process can authenticate. ![Riptides Identity](../../assets/mtls-rce/mtls-screenshot-3-riptidesui.png) But the reverse shell process fails immediately when it tries to connect. The malicious shell has no identity, so it has no access. It's still the same vulnerability, but with a completely different outcome. ![Contained Breach](../../assets/mtls-rce/mtls-screenshot-4-remote-shell-fail.png) ## Takeaway What matters is what happens after execution, and whether attackers can move laterally. Do processes implicitly trust each other? Are credentials discoverable at runtime? Can any process authenticate to databases or internal APIs simply by being “inside”? The answers to those questions decide whether an RCE stays local or cascades across your environment. By enforcing workload identity at the process level and requiring mutual TLS for internal communication, the impact of an RCE is constrained, and easier to contain and recover from. --- ## Testing Linux Kernel Modules with Bats - URL: https://blog.riptides.io/behind-the-scenes-how-we-test-at-riptides - Published: 2025-12-15 - Author: Balint Molnar - Category: Testing - Tags: testing, kernel, metrics, bats ## Introduction: Reliability at the Core At Riptides, reliability is not an add-on; it is the constraint that drives how we build and ship kernel-level tooling. Testing defines our development process because the module operates inside the Linux kernel, intercepting networking flows and [attaching SPIFFE-based workload identities](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). At this level of privilege, even minor regressions can translate into unpredictable system behaviour. Rigorous validation is therefore non-negotiable. In earlier posts, we highlighted features and user experience. Here, we focus on how we ensure correctness across the entire lifecycle: from isolated logic checks to full system-level validation [across distributions, kernel versions, vendor patches, and cloud backports](/blog/from-build-to-root-cause-how-riptides-debugs-its-kernel-module-in-real-clusters). What follows is a practical overview of the testing layers that keep the module predictable and safe in every supported environment. ## Unit Tests Inside the Code Our first validation layer lives directly in the codebase. When compiled with the TEST build flag, the module exposes a suite of internal unit tests that verify core logic during [build time](/blog/beyond-the-limits-scaling-our-kernel-module-build-pipeline-even-further). These checks are intentionally small, fast, and isolated, allowing us to detect regressions long before they reach integration testing or any release pipeline. Kernel-module development leaves little room for assumptions. There are no external libraries to fall back on, and missing functionality must be implemented in-kernel. This makes tight unit-level validation essential: it ensures that utility functions, data structures, and protocol logic behave exactly as expected before they ever interact with the broader system. ## Infrastructure Testing with Bats Once local checks pass, the code enters our infrastructure test suite. This layer validates real system behaviour: module loading, error paths, boundary conditions, and interactions with the underlying OS. One of the earliest lessons was that ***“Linux is not Linux”***. Differences across distributions, kernel versions, vendor patches, and cloud-provider backports are substantial, **some kernels ship with patch levels above 1500**. Broad, repeated, multi-distro coverage is the only reliable way to match the environments our customers actually run. A recurring challenge in kernel module testing is provisioning and tearing down environments in a consistent, automated way. Because the tests must interact directly with the kernel, the most natural control surface is the shell. After evaluating several options, we standardised on [**Bats (Bash Automated Testing System)**](https://github.com/bats-core/bats-core), a *TAP-compliant* framework that automates repetitive shell-driven testing tasks and provides a maintainable structure for system-level validation. It execute commands exactly as a user or CI job would, thus is a strong fit for configuring, loading, and inspecting kernel modules. Bats alone solves test orchestration, but meaningful validation requires expressive assertions. The surrounding ecosystem: **bats-assert**, **bats-file**, and other community extensions provides the tooling needed to check command outcomes, inspect filesystem state, and verify expected output. Because the framework is open source and simple to extend, we can adapt it when kernel specific edge cases require custom logic. Together, these components give us a maintainable, shell-native test environment that scales well across kernels, distributions, and CI pipelines. ### Why Bats Works for Us Bats fills an important gap in our workflow: it behaves like an enhanced Bash environment, close enough to manipulate kernel modules directly, yet structured enough to support a large, repeatable integration suite. It offers: - A natural workflow for integration tests built around kernel modules - Lifecycle hooks (`setup`, `setup_file`, `teardown`, `teardown_file`) that keep repetitive work organized - A mature, well tested tool trusted by a broad community - Strong helpers from `bats-assert` and `bats-file` for reliable, expressive checks - Well structured, easy to follow [documentation](https://bats-core.readthedocs.io/en/stable/) ### Where Bats Falls Short No tool is perfect. In our environment, the limitations of Bash still apply: - Bash is a constrained programming language - Debugging failures can be slower than in more expressive test frameworks - Buffered output can obscure logs or cause inconsistencies - No built-in fail-fast behaviour We design our test suite with these constraints in mind and apply compensating patterns when necessary. Bats inherits both the strengths and the constraints of Bash. In practice, this means: - Bash is a constrained programming language - Debugging failures can be slower than in more expressive test frameworks - Buffered output can obscure logs or cause inconsistencies - No built-in fail-fast behaviour We design test patterns and supporting utilities around these constraints, ensuring the suite remains readable, deterministic, and maintainable even within Bash’s boundaries. ### A Note on TAP and Why It Matters Bats outputs results in **TAP (Test Anything Protocol)**, a long-standing, interoperability focused, and language agnostic format widely used across UNIX tooling. TAP’s line-oriented output structure makes test results straightforward to parse, aggregate, and integrate into CI pipelines, log processors, and external dashboards without custom adapters. This uniformity matters in a kernel testing workflow. TAP allows us to correlate Bats results with kernel logs, system trace data, dmesg output, and CI metadata using the same parsing logic end-to-end. When investigating failures, especially those involving timing, concurrency, or subtle kernel–user-space interactions, having consistent, interoperable test output significantly reduces the debugging surface area. We also covered the [kernel-level debugging tools](/blog/practical-linux-kernel-debugging-from-pr-debug-to-kasan-kfence) we rely on in a previous post, and TAP integrates cleanly with those workflows as well. ## Testing Metrics Metrics form a critical contract between the kernel module and the user-facing components of our platform. Because our UI relies heavily on these metrics for visibility, troubleshooting, and behavioural analysis, validating their correctness is a core part of the infrastructure test suite. >We’ve extensively blogged about how we leverage [kernel-level metrics](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability) for testing and for attaching SPIFFE-based identities to workloads, forming a critical part of our validation pipeline. To ensure reliability, we run a lightweight in-house [metrics exporter](https://github.com/riptideslabs/ebpf-tracing-demo) that retrieves the module’s current metric output during testing. The exported data is then inspected using `bats-file`, which allows us to perform precise, file-level assertions on the retrieved payload. For example, `assert_file_contains` helps us verify that required metric fields appear exactly as expected, confirm that naming conventions remain stable, and ensure that values follow the correct formats. This layer of testing catches issues that structural tests alone cannot, such as missing fields, unexpected formatting changes, or regressions in counter behaviour. By combining live metric retrieval with file-based assertions, we validate not only that the module emits metrics, but that it emits the [metrics our platform and our customers depend on](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability). ![userspace-enc](../../assets/how-we-test/ui-image.png) ## Daily Debug-Kernel Validation Once per day, we run the full [infrastructure suite against a debug kernel](/blog/building-linux-driver-at-scale-our-automated-multi-distro-multi-arch-build-pipeline) configured with instrumentation, designed to reveal subtle system-level defects. These kernels enable detection of memory-safety issues, race conditions, lock-ordering bugs, and other deep failures that rarely surface under normal runtime conditions. The primary tools we rely on include: **KASAN (Kernel Address Sanitizer)** KASAN instruments memory accesses and detects out-of-bounds reads/writes, use-after-free conditions, and other memory-safety violations. Its fault reports frequently expose latent bugs that are extremely difficult to trigger outside an instrumented environment. **kmemleak** kmemleak performs a reachability-based scan of the kernel heap to identify allocations that are no longer referenced. While not a real-time leak detector, it reliably surfaces leaks that may not crash a system immediately but would gradually erode stability or memory availability over extended workloads. **KFENCE (Kernel Electric Fence)** KFENCE offers low-overhead memory corruption detection suitable for long-running or production-like tests. While its detection set is narrower than KASAN’s, its minimal runtime cost allows us to catch subtle out-of-bounds accesses and invalid writes during workloads that would be impractical under KASAN’s heavy instrumentation. **lockdep** lockdep monitors lock acquisition paths and validates ordering rules across kernel synchronization primitives. It detects deadlock risks, incorrect lock nesting, lock inversion, and other concurrency issues that only appear under specific timing or load conditions. The purpose of this daily run is not merely achieving a clean “all tests passed.” The debug-kernel environment examines memory usage, synchronization behaviour, concurrency boundaries, and error paths with far greater scrutiny than any standard configuration. These checks are executed across all kernel versions and distributions in use by our customers, ensuring long-term reliability and consistent behaviour. >We’ve previously blogged in detail about the kernel-level debug tooling we use to uncover memory, concurrency, and system-level issues: [Practical Linux Kernel Debugging: From pr_debug() to KASAN/KFENCE](/blog/practical-linux-kernel-debugging-from-pr-debug-to-kasan-kfence) ## A Glimpse Into a Real Bats Test To make the testing approach more concrete, here is a small, self-contained example, demonstrating how a typical Bats test is structured. It shows lifecycle hooks, command execution, and basic assertions using `bats-assert` and `bats-file`. ```bash load 'test_helper/bats-support/load.bash' load 'test_helper/bats-assert/load.bash' load 'test_helper/bats-file/load.bash' setup() { # Prepare environment for the test touch "$BATS_TEST_TMPDIR/example.txt" echo "hello world" > "$BATS_TEST_TMPDIR/example.txt" } teardown() { # Clean up after each test rm -f "$BATS_TEST_TMPDIR/example.txt" } @test "example file contains expected text" { run cat "$BATS_TEST_TMPDIR/example.txt" assert_success assert_output --partial "hello world" # Use bats-file for file-level assertions assert_file_contains "$BATS_TEST_TMPDIR/example.txt" "hello" } ``` This example highlights the workflow we rely on across the wider suite: - setup/teardown ensure test isolation - commands are executed exactly as they would in a shell - bats-assert validates output and return codes - bats-file inspects file contents with precise assertions Even in more complex scenarios as metrics validation, kernel interactions, error paths, etc, the same structure scales predictably and reliably. ## Conclusion Testing at Riptides is not a checkbox exercise, it is a layered system of safeguards that ensures correctness at every stage of development. Unit tests verify core foundational logic, Bats-driven infrastructure tests validate real-world behaviour **across kernels and distributions**, and daily debug-kernel runs surface subtle memory, concurrency, and ordering issues that only appear under stress. By combining these layers, we maintain high confidence that both the kernel module and the broader platform behave predictably, safely, and consistently across all supported environments. As the product evolves, the test infrastructure evolves with it, keeping reliability a defining characteristic of the Riptides stack. --- ## Supercharge Kafka security with Riptides - URL: https://blog.riptides.io/supercharge-kafka-security-with-riptides - Published: 2025-12-08 - Author: Sebastian Toader - Category: Security - Tags: secret-injection, identity, workload-id, spiffe, security Riptides uses an [identity first](/blog/the-riptides-vision-identity-first-infrastructure) security model built on **SPIFFE workload identities**. This eliminates static secrets, service accounts, and manual certificate management. Each process receives a [verifiable identity at the kernel level](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), and all network traffic is authenticated based on that identity. Because Riptides operates at the kernel level, it integrates seamlessly into existing workloads with no code or configuration changes. Zero-trust, secretless authentication, and automated credential management remove the error-prone steps of storing, distributing, and handling secrets—steps that are often the source of leaks and operational headaches. In this post, we’ll show how these principles apply to a real-world system: **Apache Kafka**, and the Confluent Platform Demo running on Kubernetes. It’s a good example because the demo has multiple interconnected components communicating over: - plain-text, TLS, and mTLS channels, - across protocols like HTTP, LDAP, and Kafka, - and using authentication methods ranging from no-auth and Basic Auth to OAuth bearer tokens, SASL, and mTLS client certificates. ## 1. Assigning workload identities The first step in using [Riptides](https://riptides.io) is to define **workload identities** for every process, including health checks, init containers, setup scripts, and long-running services. A workload identity is built from **process metadata**. The Riptides Driver, a [Linux kernel module](/blog/when-ebpf-isnt-enough-why-we-went-with-a-kernel-module), matches the metadata of any process initiating or accepting network connections against the configured workload identities. When a match is found, Riptides issues a certificate (an **x509 SVID**) containing the SPIFFE workload identity and attaches it to the process and its network traffic. To explore workload identities in more depth, see: - [Introduction to SPIFFE: Secure Identity for Workloads](/blog/introduction-to-spiffe-secure-identity-for-workloads) - [The Critical Role of Unique Workload Identity in Modern Infrastructure](/blog/the-critical-role-of-unique-workload-identity-in-modern-infrastructure) - [Workload Attestation and Metadata Gathering: Building Trust from the Ground Up](/blog/workload-attestation-and-metadata-gathering-building-trust-from-the-ground-up) The Riptides UI is especially useful here. It shows all processes generating network traffic, along with their metadata, giving a **full map of inbound and outbound connections**. You can see which workloads are talking to each other, both within the datacenter and to external sources, whether the communication is encrypted, and whether secrets or authentication are involved. Riptides operates in the Linux kernel, collecting [full telemetry on every inbound and outbound connection in real time](/blog/securing-workloads-with-kernel-telemetry-and-metrics). With this visibility, you can assign workload identities, enforce policies, and govern communications across the environment with precision. Here is a snippet illustrating how workload identities are defined for the `Kafka` and `LDAP` components: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: cp-demo-kafka-broker namespace: riptides-system spec: scope: agentGroup: id: riptides/agentgroup/eu-west-1-all-os selectors: - k8s:container:name: kafka k8s:label:app.kubernetes.io/component: kafka process:name: java workloadID: cp-demo/kafka-broker ``` This configuration assigns the **cp-demo/kafka-broker** workload identity to any **java** process running in a container named **kafka** within a pod labeled **app.kubernetes.io/component: kafka**. When such a process initiates or receives network traffic, Riptides issues an x509 SVID with the SPIFFE ID: **spiffe://riptides.io/cp-demo/kafka-broker**. In this case, **riptides.io** is the [SPIFFE trust domain](/blog/introduction-to-spiffe-secure-identity-for-workloads) configured in the Riptides Control Plane. ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: cp-demo-openldap namespace: riptides-system spec: scope: agentGroup: id: riptides/agentgroup/eu-west-1-all-os selectors: - k8s:container:name: openldap k8s:label:app.kubernetes.io/component: openldap process:name: slapd workloadID: cp-demo/openldap ``` This configuration assigns the **cp-demo/openldap** workload identity to any **slapd** process running in a container named **openldap** within a pod labeled **app.kubernetes.io/component: openldap**. ## 2. Securing plain-text communication Some of the Confluent demo components communicate in `plain text`. For example, Kafka's Metadata Service (MDS) communicates with LDAP over **unencrypted port 389**: ``` # Configure MDS to talk to AD/LDAP KAFKA_LDAP_JAVA_NAMING_FACTORY_INITIAL: com.sun.jndi.ldap.LdapCtxFactory KAFKA_LDAP_COM_SUN_JNDI_LDAP_READ_TIMEOUT: 3000 KAFKA_LDAP_JAVA_NAMING_PROVIDER_URL: ldap://openldap:389 ``` With Riptides, this traffic is transparently upgraded from **plain text** to **mTLS** using the following workload identity configuration: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: cp-demo-openldap namespace: riptides-system spec: addresses: - address: cp-openldap.confluent-demo.svc.cluster.local port: 389 labels: app: openldap --- apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: cp-demo-kafka-broker namespace: riptides-system spec: scope: agentGroup: id: riptides/agentgroup/eu-west-1-all-os selectors: - k8s:container:name: kafka k8s:label:app.kubernetes.io/component: kafka process:name: java workloadID: cp-demo/kafka-broker allowedSPIFFEIDs: inbound: - spiffe://riptides.io/cp-demo/kafka-broker egress: - connection: tls: mode: MUTUAL selectors: - app: openldap allowedSPIFFEIDs: - spiffe://riptides.io/cp-demo/openldap --- apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: cp-demo-openldap namespace: riptides-system spec: scope: agentGroup: id: riptides/agentgroup/eu-west-1-all-os selectors: - k8s:container:name: openldap k8s:label:app.kubernetes.io/component: openldap process:name: slapd workloadID: cp-demo/openldap allowedSPIFFEIDs: inbound: - spiffe://riptides.io/cp-demo/kafka-broker - spiffe://riptides.io/cp-demo/openldap ingress: - connection: tls: mode: MUTUAL port: 389 ``` This configuration ensures that when processes with **cp-demo/kafka-broker** workload id connects to the **openldap** service on port **389** is done via mTLS. Riptides automatically uses the x509 SVID certificates generated for workloads to enforce mTLS at the kernel layer. It also applies **ingress** and **egress** rules, ensuring only permitted workloads communicate. The screenshot below shows the final state after Riptides has transparently upgraded all plaintext Kafka connections to LDAP into mTLS at the kernel layer. ![Riptides Connection Inventory](../../assets/kafka-with-riptides/imageinblog03_a.jpg) ## 3. Allowing existing TLS/mTLS traffic to pass through If a component already uses TLS or mTLS, Riptides can operate in **pass-through mode**. The traffic remains unchanged, but identities are still assigned and communication rules enforced. For example, the `kstreams-app` component performs mTLS authentication with Kafka. Riptides allows the original client certificate to pass through: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: cp-demo-kafka-cluster-encrypted-listeners namespace: riptides-system spec: addresses: - address: cp-kafka.confluent-demo.svc.cluster.local port: 10091 # SASL_SSL listener with OAuth/token authentication - address: cp-kafka.confluent-demo.svc.cluster.local port: 11091 # SSL listener with client certificate authentication - address: cp-kafka.confluent-demo.svc.cluster.local port: 8091 # Metadata Service (MDS) https listener labels: app: kafka-cluster-encrypted-listeners --- apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: cp-demo-kafka-broker-0-encrypted-listeners namespace: riptides-system spec: addresses: - address: cp-kafka-0.cp-kafka-headless.confluent-demo.svc.cluster.local port: 10091 # SASL_SSL listener with OAuth/token authentication - address: cp-kafka-0.cp-kafka-headless.confluent-demo.svc.cluster.local port: 11091 # SSL listener with client certificate authentication - address: cp-kafka-0.cp-kafka-headless.confluent-demo.svc.cluster.local port: 8091 # Metadata Service (MDS) https listener labels: app: kafka-broker-0-encrypted-listeners --- apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: cp-demo-kafka-broker namespace: riptides-system spec: scope: agentGroup: id: riptides/agentgroup/eu-west-1-all-os selectors: - k8s:container:name: kafka k8s:label:app.kubernetes.io/component: kafka process:name: java workloadID: cp-demo/kafka-broker allowedSPIFFEIDs: inbound: - spiffe://riptides.io/cp-demo/kafka-broker ingress: - allowedSPIFFEIDs: - spiffe://riptides.io/cp-demo/kafka-connect - spiffe://riptides.io/cp-demo/streams-demo-app connection: tls: mode: PERMISSIVE port: 11091 egress: - connection: tls: mode: MUTUAL selectors: - app: openldap allowedSPIFFEIDs: - spiffe://riptides.io/cp-demo/openldap --- apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: cp-demo-streams-demo-app namespace: riptides-system spec: scope: agentGroup: id: riptides/agentgroup/eu-west-1-all-os workloadID: cp-demo/streams-demo-app egress: - allowedSPIFFEIDs: - spiffe://riptides.io/cp-demo/kafka-broker selectors: - app: kafka-cluster-encrypted-listeners - app: kafka-broker-0-encrypted-listeners - app: kafka-broker-1-encrypted-listeners - app: kafka-broker-2-encrypted-listeners - allowedSPIFFEIDs: - spiffe://riptides.io/cp-demo/schema-registry selectors: - app: schema-registry selectors: - k8s:container:name: streams-demo k8s:label:app.kubernetes.io/component: streams-demo process:name: java ``` This configuration shows that the **streams-demo** Java application is permitted to initiate connections **only** to the **cp-demo/schema-registry** and **cp-demo/kafka-broker** workloads. The identity it presents when initiating these connections is **cp-demo/streams-demo-app**. On the server side, the **cp-demo/kafka-broker** workload is configured to accept connections on port **11091** only from the **cp-demo/kafka-connect** and **cp-demo/streams-demo-app** workloads in **PERMISSIVE** mode. **PERMISSIVE mode** means that TLS or mTLS is established by the workloads in **user space**, rather than by Riptides in the [kernel space](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). For brevity, this post includes only configuration snippets that illustrate the concepts; the full configuration for the demo application is longer and not shown here. Nevertheless, if you are interested in a full demo, please [get in touch with us](https://riptides.io/request-a-demo). The screenshot below shows that Riptides detects and allows an existing TLS connection between the `streams demo app` and `kafka` to pass through unchanged, since the workloads already establish a secure channel on their own: ![Riptides Connection Inventory](../../assets/kafka-with-riptides/imageinblog04_a.jpg) ## 4. Just-in-time secret injection on the wire Some components use **Basic Auth** for health checks. For example, the `Schema Registry` health check is configured like this: ```bash curl --user schemaregistryUser:schemaregistryUser --fail --silent --insecure https://cp-schemaregistry.confluent-demo.svc.cluster.local:8085/subjects --output /dev/null || exit 1 ``` With Riptides, credentials don’t need to be stored in config files, environment variables, or command-line flags. Instead, they’re injected directly into the network stream at the kernel level, completely transparently to the application. We won’t dive into the internals here — those are covered in detail in the linked posts below - but this shows how secretless authentication works in practice for this demo. For a deeper exploration of the mechanism and its security implications, see: - [On-the-Wire Credential Injection: Secretless AWS Bedrock Access example](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) - [On demand credentials – Secretless AI assistant example on GCP](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp) - [Workload Identity Without Secrets: a Blueprint for the Post-Credential Era](/blog/workload-identity-without-secrets-a-blueprint-for-the-post-credential-era) - [Shai-Hulud 2.0: A Technical Breakdown and Why Secrets Need to Die](/blog/shai-hulud-2-0-a-technical-breakdown-and-why-secrets-need-to-die) ## 5. Simplifying Kafka client configuration So far, the Confluent demo required **no changes**, all security upgrades were handled purely via Riptides configuration. To fully remove secrets from client workloads, Kafka clients are reconfigured to connect in **plain-text mode**. Riptides then transparently upgrades all traffic to **mTLS at the kernel level**, keeping secrets out of userspace while ensuring encryption and authentication. **This eliminates keystores, truststores, and client-side credentials, reducing operational overhead and the risk of leaks. Servers only need to trust the Riptides CA to validate client certificates.** ## 6. Truststore and keystore files locked to server workloads Certain server-side features, such as Kafka extracting principals from client certificates still rely on keystores and truststores. With Riptides, these files are **dynamically generated, updated, and locked down.**. Riptides provides each Kafka broker with its own truststore and keystore via **sysfs**, ensuring that only **the Kafka workload can read them**. To use these files, Kafka simply needs to be reconfigured to point to the sysfs paths instead of its original keystore and truststore locations. Because Riptides manages these files automatically, it handles the **full lifecycle**: - Rotates certificates when the originals are updated - Adds new Certificate Authorities (CAs) as needed - Ensures that Kafka always has up-to-date, valid trust material without manual intervention This approach eliminates the operational burden of managing keystores/truststores while maintaining strong security guarantees for the Kafka server. ## Key takeaways Riptides lets teams secure both new and existing workloads without adding operational complexity. By enforcing **zero-trust principles**, enabling **secretless communication**, and **automating secret and certificate management**, it eliminates the traditional pain points and risks of manual credential handling. All workloads are uniquely identified at the kernel level, ensuring **no unidentified or rogue process can communicate**. Existing applications continue to operate with minimal or no changes, while security is applied consistently across the environment. In short, Riptides makes **identity-first, automated security the default**, reducing operational risk, simplifying compliance, and letting teams focus on building and running applications rather than managing secrets. --- ## Introducing Riptides Conditional Access: Fine-Grained, Time-Aware Security Policies - URL: https://blog.riptides.io/introducing-riptides-conditional-access-fine-grained-time-aware-security-policies - Published: 2025-11-27 - Author: Nandor Kracser - Category: Security - Tags: zero-trust, conditional-access, XACML, OPA, workload-identity, microservices, SPIFFE ## The Evolution of Zero Trust Security In modern zero trust architectures, identity-based access control has become the foundation of secure communications. At Riptides, we've built a system that [automatically issues X.509 certificates (SVIDs) to workloads](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) based on process selectors, allowing services to authenticate each other without hardcoded credentials. Beyond securing workload-to-workload traffic with [automatic mTLS](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), Riptides also [federates identities across clouds](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure) and [injects short-lived credentials directly into workloads](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example). Rooted in SPIFFE, it gives workloads seamless access to third-party APIs without any static or long-lived secrets. But authentication alone isn’t enough. ***What if access should only be allowed during certain hours? What if a credential must be usable only once? What if permissions need to vanish right after an emergency deploy?*** That’s where **Riptides Conditional Access** comes in, extending our policy engine with time-based, usage-limited, and context-aware controls, all evaluated through Open Policy Agent (OPA). ## How Riptides Works Riptides uses declarative YAML configuration files to define workload identities, TLS policies, and secrets. Here's what a typical policy looks like: ### Identity Configuration ```yaml # API Gateway service handling external requests - selectors: - process:uid: 1000 process:name: api-gateway destination:port: [8080, 8443] workloadID: api-gateway svid: x509: dnsNames: - api.acme.corp - gateway.acme.corp ttl: 3600s allowedSPIFFEIDs: inbound: - spiffe://{{.TrustDomain}}/frontend - spiffe://{{.TrustDomain}}/mobile-app outbound: - spiffe://{{.TrustDomain}}/payment-service - spiffe://{{.TrustDomain}}/user-service # Payment Service processing transactions - selectors: - process:uid: 1001 process:name: payment-svc destination:port: 9000 workloadID: payment-service svid: x509: dnsNames: - payment.internal.acme.corp ttl: 3600s allowedSPIFFEIDs: inbound: - spiffe://{{.TrustDomain}}/api-gateway - spiffe://{{.TrustDomain}}/order-service ``` ### Service Discovery ```yaml # Payment service backend - addresses: - address: payment-service.internal port: 9000 - address: payment-svc-01.us-west-2.internal port: 9000 - address: payment-svc-02.us-west-2.internal port: 9000 labels: service: payment tier: backend region: us-west-2 # User service backend - addresses: - address: user-service.internal port: 8080 - address: users-db-proxy.internal port: 5432 labels: service: user-management tier: backend ``` ### Dynamic Secret Sources with TokenEx Riptides uses [**TokenEx**](https://github.com/riptideslabs/tokenex) to dynamically fetch short-lived cloud credentials via workload identity federation—eliminating long-lived secrets entirely. [**TokenEx**](https://github.com/riptideslabs/tokenex) is our new open source Go library (short for Token Exchange), to handle fetching and refreshing credentials so everything stays short-lived by default. ```yaml # Fetch AWS credentials dynamically using AWS Workload Identity Federation webserver: aws-s3-access: source: type: tokenex-aws roleArn: arn:aws:iam::123456789012:role/prod-s3-reader region: us-east-1 # Fetch GCP access tokens using GCP Workload Identity Federation accounting: gcp-bigquery-access: source: type: tokenex-gcp serviceAccount: bigquery-reader@acme-prod.iam.gserviceaccount.com # Fetch Azure access tokens using Azure Workload Identity Federation api-gateway: azure-keyvault-access: source: type: tokenex-azure clientId: 12345678-1234-1234-1234-123456789012 tenantId: 87654321-4321-4321-4321-210987654321 ``` TokenEx exchanges SPIFFE SVIDs for cloud provider credentials (AWS session tokens, GCP/Azure access tokens and OCI UPSTs) and automatically refreshes them before expiration. >To learn more about Tokenex you can check our post: [Introducing TokenEx: An Open Source Go Library for Fetching and Refreshing Cloud Credentials](/blog/introducing-tokenex-an-open-source-go-library-for-fetching-and-refreshing-cloud-credentials) ### The OPA Engine Under the hood, Riptides feeds these YAML policies into an Open Policy Agent (OPA) evaluator (`pkg/eval/socket.rego`). When a connection is initiated, the agent: 1. **Augments** the connection with runtime metadata (process info, destination, labels, etc) 2. **Evaluates** the connection against OPA policies loaded from YAML 3. **Returns** matching policies with certificate issuance, TLS mode, allowed peers, and credentials Here's a simplified flow from the connection evaluation logic: ```go func (c *evalCommand) HandleCommand(cmd *driver.Command) (*driver.Command, error) { // Get connection metadata input := cmd.GetOpaEval().GetConnection() // Augment with process/system labels augmentationResp, err := c.augmenter.Augment(taskContext) if err != nil { return nil, err } // Merge labels if input.Labels != nil { maps.Copy(input.Labels, augmentationResp.Labels) } else { input.Labels = augmentationResp.Labels } // Evaluate against OPA policies res, err := c.eval.Eval(ctx, input) if err != nil { return nil, err } // Return policy decision return buildResponse(res), nil } ``` ## The Problem: Static Policies Aren't Enough Today's YAML policies are **static**, they define *who* can connect to *what*, but they don't capture *when*, *how often*, or *under what conditions*. Real-world security scenarios demand more: ### Use Case 1: Emergency Break-Glass Access Your on-call engineer needs temporary admin access to production databases during a P0 incident but only for 2 hours, and only once. ### Use Case 2: Time-Window Credential Rotation AWS credentials should only be valid during a specific deployment window (e.g., 2 AM - 3 AM UTC) to minimize blast radius if leaked. ### Use Case 3: Rate-Limited API Keys A service account should have a bearer token that works for exactly 100 API calls, then automatically revokes. ### Use Case 4: Compliance-Driven Time Fencing PCI DSS requires that production database access is only permitted during business hours (9 AM - 5 PM EST) for non-emergency personnel. These examples just scratch the surface, there are countless scenarios where dynamic, context-aware policies are essential for enforcing least-privilege access safely. ## Enter: Conditional Access Policies Riptides Conditional Access extends the YAML policy format with **conditional blocks** that leverage OPA's powerful policy language. Here's what's coming: ### Time-Based Access Control ```yaml # Grant database access only during business hours - selectors: - process:name: postgresql destination:port: 5432 workloadID: prod-db-admin svid: x509: dnsNames: - admin.db.acme.corp ttl: 3600s conditionalAccess: timeWindow: start: "09:00:00" end: "17:00:00" timezone: "America/New_York" daysOfWeek: [1, 2, 3, 4, 5] # Monday-Friday allowedSPIFFEIDs: inbound: - spiffe://acme.corp/dba-team ``` ### Usage-Based Access Control ```yaml # Single-use emergency credentials - selectors: - process:name: curl destination:port: 443 workloadID: emergency-deploy credentialName: aws-emergency-cred conditionalAccess: usageLimit: maxCount: 1 resetOnExpiry: false allowedSPIFFEIDs: outbound: - spiffe://acme.corp/prod-api ``` ### HTTP Path & Method-Based Access Control ```yaml # Restrict access to specific API endpoints and HTTP methods - selectors: - process:name: [node, python3] destination:port: 443 workloadID: api-client svid: x509: dnsNames: - client.api.acme.corp ttl: 3600s conditionalAccess: allOf: # Only allow read operations - httpMethod: allowed: [GET, HEAD, OPTIONS] # Restrict to specific paths - httpPath: allowed: - /api/v1/users/* - /api/v1/orders/read denied: - /api/v1/admin/* - /api/v1/users/*/delete allowedSPIFFEIDs: outbound: - spiffe://acme.corp/api-server ``` ### Combined Conditions: Break-Glass Access ```yaml # Emergency access: valid for 2 hours, usable once, only by specific engineer - selectors: - process:name: psql destination:port: 5432 workloadID: break-glass-db-access svid: x509: dnsNames: - emergency.db.acme.corp ttl: 7200s # 2 hours conditionalAccess: allOf: - timeWindow: duration: 7200s # 2 hours from first use startOnFirstUse: true - usageLimit: maxCount: 1 - requiredLabels: user:oncall: "true" incident:severity: "P0" allowedSPIFFEIDs: inbound: - spiffe://acme.corp/oncall-engineer ``` ## How It Works: The OPA Integration Riptides' architecture is influenced by the [XACML (eXtensible Access Control Markup Language)](https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xacml) standard, a widely adopted framework for attribute-based access control (ABAC) that separates policy enforcement, decision-making, administration, and context enrichment into distinct components. ***What is XACML?*** XACML is an OASIS standard that defines a policy language and architecture for expressing and evaluating access control policies. It was designed to enable fine-grained, attribute-based authorization across diverse systems. The standard emphasizes separation of concerns, and keeping enforcement, decision-making, and policy administration independent, while supporting attribute-based access control where decisions are based on properties of the subject, resource, action, and environment. Its extensibility allows for custom attributes, conditions, and policy combinators, making it adaptable to different security requirements. While Riptides doesn't strictly implement the XACML specification (we use OPA/Rego instead of XACML's XML-based policy language), we adopt its architectural patterns to achieve similar goals. The separation of enforcement from decision making, combined with rich contextual information, gives us both the performance of kernel-level interception and the flexibility of declarative policy evaluation. ***Why userspace policy evaluation?*** Early in Riptides' development, we explored kernel-based policy evaluation using WASM, but ultimately moved policy decisions to userspace for better flexibility and debuggability. Read more about this architectural decision in our blog post: [From Kernel WASM to User-Space Policy Evaluation: Lessons Learned at Riptides](/blog/from-kernel-wasm-to-user-space-policy-evaluation-lessons-learned-at-riptides). ### XACML-Inspired Architecture Components In Riptides, the **Policy Enforcement Point (PEP)** is our kernel module and eBPF code that intercepts connection attempts at the network layer and enforces policy decisions. The **Policy Decision Point (PDP)** is the Riptides agent running in userspace, which evaluates policies using OPA and returns access decisions to the PEP. The **Policy Administration Point (PAP)** is the Riptides Controlplane, the central hub that loads YAML configuration files and distributes policies to all agents across your infrastructure. Finally, the **Policy Information Point (PIP)** is our augmentation layer, which enriches connection metadata with process information, labels, and runtime context. This separation ensures that enforcement happens at wire speed in the kernel without userspace context switches for data plane operations, while policy decisions remain flexible in userspace and can be updated without kernel changes. Policies are expressed declaratively in YAML, defining "what" should happen rather than "how" to enforce it. The PIP provides rich context by augmenting connections with over 50 labels covering process, node, and container metadata. ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ Riptides Conditional Access Architecture │ └─────────────────────────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────────┐ │ Policy Administration Point (PAP) │ │ ┌────────────────────────────────────────────────────────┐ │ │ │ Riptides Controlplane │ │ │ │ • Central configuration hub for all agents │ │ │ │ • Loads YAML policies: │ │ │ │ - Workload Identities (SVIDs) │ │ │ │ - Service Discovery │ │ │ │ - Credentials (TokenEx) │ │ │ │ - Conditional Access Policies │ │ │ │ • Distributes to all agents │ │ │ └────────────────────────────────────────────────────────┘ │ └────────────────────────────┬─────────────────────────────────┘ │ Pushes policies to agents ▼ ┌──────────────────────────────────────────────────────────────┐ │ Policy Decision Point (PDP) - Userspace │ │ ┌────────────────────────────────────────────────────────┐ │ │ │ Riptides Agent + Open Policy Agent (OPA) │ │ │ │ • Receives policies from Controlplane │ │ │ │ • Compiles Rego policies │ │ │ │ • Augments connection context (PIP) │ │ │ │ • Evaluates connections against rules │ │ │ │ • Returns ALLOW/DENY + obligations to kernel │ │ │ └────────────────────────────────────────────────────────┘ │ └──────────────────────────────┬───────────────────────────────┘ │ Returns │ policy │ decision │ ▼ ┌──────────────────────────────────────────────────────────────┐ │ Policy Enforcement Point (PEP) - Kernel │ │ ┌────────────────────────────────────────────────────────┐ │ │ │ Kernel eBPF Module (lowest level) │ │ │ │ • Intercepts socket operations │ │ │ │ • Captures connection metadata │ │ │ │ • Sends to agent for policy decision │ │ │ │ • Enforces agent decisions (ALLOW/DENY) │ │ │ │ • Issues X.509 SVIDs │ │ │ │ • Injects credentials │ │ │ │ • Manages TLS handshakes │ │ │ └────────────────────────────────────────────────────────┘ │ └──────────────────────────────┬───────────────────────────────┘ │ Intercepts │ connection │ attempts │ │ ┌──────────────────────────────▼───────────────────────────────┐ │ Workload Process │ │ (e.g., API Gateway connecting to Payment Service) │ └──────────────────────────────────────────────────────────────┘ Example Flow: API Gateway → Payment Service ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1. [PAP] Controlplane loads YAML policies and pushes to all agents 2. API Gateway (UID 1000, process: api-gateway) attempts connection to payment-service.internal:9000 3. [PEP - Kernel] eBPF hooks socket creation at lowest level, extracts: • PID, UID, process name • Destination IP, port • Protocol (TCP) Sends to agent for decision 4. [PDP - Agent] Augments with runtime context (PIP): • Labels: {service: api-gateway, region: us-west-2, ...} • Current time: 2025-11-27T14:30:00Z • Usage count: 42 connections today 5. [PDP - OPA] Evaluates against compiled policy from Controlplane: rego: matching_policies contains policy if { some p in data.policies # From Controlplane is_subset(input.labels, p.selectors) # Check conditional access time_in_window(p.conditionalAccess.timeWindow, input.timestamp) usage_under_limit(p.conditionalAccess.usageLimit, input.usage) # Business hours check: 9 AM - 5 PM EST hour := time.clock(input.timestamp)[0] hour >= 9; hour < 17 # ✅ PASS (2:30 PM EST) } 6. [PDP → PEP] Agent returns policy decision to kernel: { "decision": "ALLOW", "svid": { "dnsNames": ["api.acme.corp"], "ttl": 3600 }, "allowedPeers": ["spiffe://acme.corp/payment-service"], "tlsMode": "MUTUAL" } 7. [PEP - Kernel] Enforces decision at lowest level: • Issues X.509 certificate (api.acme.corp) • Intercepts TLS handshake • Validates peer SPIFFE ID: spiffe://acme.corp/payment-service • Connection established ✅ ``` ### How the Components Work Together ### 1. Policy Loading & Compilation (PAP → PDP) At startup, the agent (PDP) reads YAML configuration files (PAP) and **compiles** the OPA policy once. The Rego policy logic itself is static—what changes is the **data** fed into it at evaluation time: ```go func (c *evalCommand) OnDataUpdate(data map[string]any) error { // Compile OPA policy once with static policies from YAML // Policy contains the evaluation rules (socket.rego) // Data contains the YAML configuration (identities, services, credentials) opa, err := eval.NewOpaEvaluator(context.Background(), c.logger, data) if err != nil { return err } c.eval = opa // Compiled policy ready for evaluation return nil } ``` The key insight: **The Rego policy is compiled once. Only the input data (connection metadata + runtime context) changes per evaluation.** ### 2. Connection Interception (PEP) When a process initiates a connection, the **kernel eBPF module (PEP)** intercepts it at the socket layer and sends metadata to the userspace agent (PDP) for a policy decision. ### 3. Context Enrichment (PIP → PDP) The agent's augmentation layer (PIP): - Captures metadata (PID, UID, process name, destination IP/port) - Augments with labels (hostname, kernel version, Docker tags, custom labels) - Adds runtime context (current timestamp, usage counters from state store) - Queries the **pre-compiled** OPA policy with this enriched input ### 4. Policy Evaluation (PDP) The **pre-compiled** Rego policy in the agent (PDP) evaluates the dynamic input against static policy rules: ```python matching_policies contains policy if { some p in data.policies # Static policies from YAML (loaded at startup from PAP) some selectorset in p.selectors is_subset(input.labels, selectorset) # Match process/destination from PIP context # NEW: Evaluate conditional access using runtime data evaluate_conditional_access(p.conditionalAccess, input) policy := prepare_policy_response(p, input) } ``` **Key Architecture Points:** - **Rego policy**: Compiled once at startup, contains evaluation logic (lives in PDP) - **`data.policies`**: Static YAML configuration from PAP (identities, services, credentials) - **`input`**: Dynamic per-connection data enriched by PIP (metadata, timestamp, usage counters) - **Evaluation**: Fast—no recompilation, just data lookup and rule matching ### 5. Policy Enforcement (PDP → PEP) If the policy matches *and* conditions pass, the agent (PDP) returns a decision to the kernel module (PEP): - **ALLOW** - Issue X.509 SVID with specified DNS names and TTL, inject credentials, configure TLS mode - **DENY** - Block the connection at the kernel layer - **OBLIGATIONS** - Additional actions (e.g., log the connection, increment usage counters) The kernel module (PEP) enforces the decision: - Allows or blocks the connection - Intercepts TLS handshake if needed (for mTLS or credential injection) - Reports enforcement events back to the agent for audit logging ### 4. Certificate Issuance & Credential Injection If the policy matches *and* conditions pass: - Issue X.509 SVID with specified DNS names and TTL - Inject credentials (AWS, GCP, bearer tokens) into the connection - Configure TLS mode (mTLS, SIMPLE, intercept) - Enforce allowed SPIFFE IDs for peer validation ## Conditional Access: Phased Rollout Plan Riptides Conditional Access will be rolled out in stages, reflecting a careful, iterative process. Each phase is informed by internal testing, collaboration with design partners, and ongoing user feedback, ensuring a robust, production-ready feature set that evolves with real-world requirements. ### Phase 1: Time-Based Access - Absolute time windows (`start`/`end` times) - Day-of-week filtering - Timezone support ### Phase 2: HTTP-Based Access - HTTP path-based access control (Layer 7 policies) - HTTP method restrictions (GET/POST/PUT/DELETE) - Header-based conditions ### Phase 3: Stateful Conditions - Connection count limits - Request rate limiting - Token depletion tracking - Relative durations (`startOnFirstUse` + `duration`) - Integration with Redis/etcd for distributed state ### Phase 4: Context-Aware & Audit - Required label matching (e.g., `incident:severity=P0`) - IP allowlists/denylists - Geolocation-based access (cloud region constraints) - Custom OPA policy hooks - Conditional access event logging - Compliance reports (SOC 2, PCI DSS) - Policy violation alerts ## Why This Matters Conditional Access extends Riptides from a **workload identity platform** into a dynamic, context-aware access control system, fully compatible with Zero Trust principles. By combining: - **Process-level selectors** (who is making the connection) - **Service discovery** (what they're connecting to) - **Credential injection** (what secrets they need) - **Conditional policies** (when and how they can access) ...you get a system that continuously enforces least-privilege access. Access rights can automatically expire, be limited in use, or adapt to time, context, and compliance requirements—reducing the risk of over-privileged credentials, eliminating manual break-glass processes, and closing security gaps. ## Example: Full Conditional Access Policy Here's a realistic production policy combining all features: ```yaml # Production database access with multiple safeguards - selectors: - process:name: [psql, pgcli] destination:port: 5432 workloadID: prod-db-access svid: x509: dnsNames: - db.prod.acme.corp ttl: 1800s # 30 minutes credentialName: postgres-admin conditionalAccess: allOf: # Only during business hours - timeWindow: start: "09:00:00" end: "17:00:00" timezone: "America/New_York" daysOfWeek: [1, 2, 3, 4, 5] # Max 10 connections per hour - usageLimit: maxCount: 10 window: 3600s # Only allow read operations via HTTP - httpMethod: allowed: [GET, HEAD] # Restrict to specific database query endpoints - httpPath: allowed: - /api/v1/query/* - /api/v1/reports/* denied: - /api/v1/admin/* # Must have DBA role label - requiredLabels: user:role: "dba" access:level: "admin" allowedSPIFFEIDs: inbound: - spiffe://acme.corp/dba-team connection: tls: mode: MUTUAL ``` This policy ensures: - ✅ Only DBAs can connect - ✅ Only during business hours - ✅ Rate-limited to 10 connections/hour - ✅ Only read operations (GET/HEAD) allowed - ✅ Restricted to query/report endpoints (no admin access) - ✅ Automatically revokes after 30 minutes - ✅ Full mutual TLS authentication ## Conclusion Riptides Conditional Access brings fine-grained, time-aware security controls to workload identity. By extending our YAML policy format with OPA-powered conditions, you can enforce least-privilege access dynamically, without sacrificing developer velocity. --- ## Shai-Hulud 2.0: A Technical Breakdown and Why Secrets Need to Die - URL: https://blog.riptides.io/shai-hulud-2-0-a-technical-breakdown-and-why-secrets-need-to-die - Published: 2025-11-26 - Author: Janos Matyas - Category: Security - Tags: credentials, identity, zero-trust ## What Actually Happened - A Technical Postmortem In November 2025, threat actors relaunched **Shai-Hulud**, calling it *"The Second Coming."* This version is significantly more aggressive, infecting **hundreds** of npm packages and digging deep into developer machines and CI runners. Here's the flow, in plain technical terms: ### **1. Compromised npm Packages** Multiple popular npm libraries were hijacked (Zapier SDK, AsyncAPI specs, PostHog SDKs, Postman and others). The attacker added a **malicious `preinstall` hook** to each `package.json`. That means: - The payload runs **before** the package is even installed. - It executes whether you use npm, pnpm, yarn, or anything that honors lifecycle scripts. ### **2. Bootstrapping Through Bun** Each infected package runs a script, `bun_setup.js`, which: 1. Checks if **Bun** (a JS runtime) is installed. 2. If not, silently downloads and installs Bun. 3. Once Bun is present, executes the real payload: `bun_environment.js`. This second payload is megabytes in size and heavily obfuscated. ### **3. Deep System Recon & Credential Harvesting** `bun_environment.js` does the following: - Launches **TruffleHog** internally to scrape secrets from: - Git repos - Local files - Environment variables - Shell history - Extracts cloud credentials (AWS, GCP, Azure) - Collects npm and GitHub tokens - Gathers metadata, environment variables, and system info The payload works on **Linux, macOS, and Windows**. >TruffleHog is an open source powerful and popular secrets scanning tool ### **4. Worm-like Self-Propagation** The malware: - Uses stolen **npm tokens** from `.npmrc`\ - Enumerates packages the victim can maintain\ - Downloads them\ - Injects its own preinstall backdoor\ - Publishes a new version under the victim's identity This turns every compromised developer into a new infection source. ### **5. Clever Exfiltration via GitHub** Instead of talking to a suspicious server, Shai-Hulud uses **GitHub** as its exfiltration channel: - Creates **public repos** with random names - Uploads encoded JSON files: - `cloud.json` - `environment.json` - `contents.json` - `truffleSecrets.json` It even registers the victim machine as a **self-hosted runner** and plants a malicious GitHub Actions workflow for persistence. ### **6. Impact at Scale** - **600+ npm packages** infected - **20,000+ GitHub repos** created for exfiltration - **15,000 leaked secrets** This is the largest npm supply-chain incident since the ecosystem existed. ## Static Secrets Are the Real Vulnerability The core issue Shai-Hulud 2.0 exploited wasn't Bun, npm, GitHub, or JavaScript. It was the **industry's addiction to long-lived credentials**. ### Why Static Secrets Are Fundamentally Broken #### **1. They Don't Rotate** Tokens meant to be "temporary" end up living for months or years. Shai-Hulud 2.0 harvested npm and GitHub tokens and immediately re-used them to: - Publish malicious package versions - Create GitHub repos - Push exfiltrated data If a token doesn't expire, it *will* be abused. #### **2. Secrets Live Everywhere** Developers scatter secrets across: - `.npmrc` - Shell environment variables - CI runner environments - Config files - Local Git clones Embedding TruffleHog in malware effectively creates an automated, instant credential harvesting tool. #### **3. CI/CD Runners Are Gold Mines** Most CI systems expose: - Long-lived GitHub tokens - Cloud creds in environment variables - Overprivileged roles like `repo:write` or `publish` Shai-Hulud harvested them effortlessly. #### **4. Using GitHub for Exfiltration Makes Defense Hard** Blocking GitHub API traffic also breaks all legitimate developer workflow. So the malware blends in perfectly, under the radar. ## Moving Beyond Secrets: The Identity-First Future The solution isn't "rotate tokens faster" or "put them in a vault." The solution is **no more static secrets**. Use **ephemeral, identity-based access**. ### The Better Model - **Short-lived credentials** - **SPIFFE identity-based authentication** - **Just-in-time access** - **No secret material stored on disk** - **Least privilege, enforced by policy** - **Telemetry-rich control plane** This is the model cloud-native security should be rallying around, and this is also the model **Riptides** is built on. Workloads shouldn't "have a secret."They should **have an identity**. ## Why This Couldn't Happen With Riptides Let’s look at the same attack path in a real application. In this demo we walk through an npm supply-chain compromise that scrapes API keys from a running pod, and then show how Riptides breaks the chain by removing static credentials and delivering them just-in-time to the right process. ## Conclusion: Riptides Is What the Supply Chain Needs Static secrets are a liability. Attackers don't break through your zero trust perimeter, they just find a token lying around and *become you*. Riptides solves this by: - Eliminating long-lived secrets - Issuing workload identities through SPIFFE - Enforcing least-privilege policies within the Linux kernel - Providing kernel-level telemetry - Delivering ephemeral identity instead of permanent power **Identity beats secrets. Riptides makes identity nativ, all the way down from the Linux kernel.** --- ## From Build to Root Cause: How Riptides Debugs Its Kernel Module in Real Clusters - URL: https://blog.riptides.io/from-build-to-root-cause-how-riptides-debugs-its-kernel-module-in-real-clusters - Published: 2025-11-24 - Author: Peter Balogh - Category: Kernel - Tags: kernel, linux, automation ## Scaling Kernel Module Debugging with EKS, Debug Kernels, and Automation At Riptides, we secure workload-to-workload (process-to-process) communication using [SPIFFE identities and in-kernel policy enforcement](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). Our kernel module hooks into the Linux networking stack, tags traffic with identities, and enforces mTLS automatically. Because this logic runs in kernel space, bugs aren't just annoying, they can take down the whole node. A single memory leak, a hidden race condition, or a stack bug can destabilize the entire system. So the real question is: >How do you debug a kernel module under real workloads, real traffic, and real Kubernetes scheduling quirks and do it repeatedly without guessing? In this post, we take a production-like environment built around EKS, custom Amazon Linux 2023 debug kernels, Packer-built AMIs, SSM-based versioning, and automated EC2 test runners and show how we use this setup to debug and surface the bugs that only show up under real load. >**Important note:** We use EKS as an example in this post because it's our primary cloud for development and testing, but the same pipeline works across all major clouds. For every Riptides major version/release candidate, we run the full debug process on AWS, Azure, GCP, and OCI using automation we've built for each. For on-prem customers — whether bare metal or VM-based — we recreate the same environment so we can test and reproduce any potential issues before deployment. This post extends the concepts from our earlier deep-dive: [Practical Linux Kernel Debugging: From pr_debug() to KASAN/KFENCE](/blog/practical-linux-kernel-debugging-from-pr-debug-to-kasan-kfence) which focused on how we debug kernel modules using pr_debug, dynamic debug, KASAN, KFENCE, kmemleak, lockdep, and other instrumentation. It also builds on the foundation of our automated [multi-distro, multi-architecture driver build pipeline](/blog/beyond-the-limits-scaling-our-kernel-module-build-pipeline-even-further). But today we're shifting focus from building the driver to showing how we debug it automatically at scale. ## Why Debugging a Kernel Module Requires Special Infrastructure User-space debugging tools don't work inside the kernel. When something fails in kernel space, you're often left with: - A kernel panic - A hung CPU - Silent corruption detected only minutes later - A cryptic dump in dmesg - No stack trace at all Effective kernel debugging requires: - A kernel compiled with full debugging instrumentation - A realistic workload environment - Automated reproducibility - Centralized visibility into kernel behavior These above led to our Riptides Debug Kernel Pipeline, which combines: - Custom EKS debug clusters - Custom Amazon Linux 2023 debug AMIs - Packer-based kernel recompilation - CloudWatch-based kernel-level observability - CI-driven EC2 debug runners ## Building the Riptides Debug Kernel Our debug kernel is based on the same AL2023 kernel used by EKS, but with every relevant debugging feature enabled. **KASAN – Detecting use-after-free, buffer overflows**: ``` CONFIG_KASAN CONFIG_KASAN_GENERIC CONFIG_KASAN_OUTLINE CONFIG_KASAN_STACK ``` These KASAN options enable the Kernel Address Sanitizer to detect memory safety bugs, such as use-after-free, buffer overflows, and stack out-of-bounds errors—using generic instrumentation with outlined checks and full stack tracing. **KFENCE – Low-overhead heap corruption detection**: ``` CONFIG_KFENCE CONFIG_KFENCE_SAMPLE_INTERVAL=50 ``` KFENCE complements KASAN by catching memory safety bugs in low-frequency sampling mode (ideal on production-like workloads). **Kernel memory leak detection (kmemleak)**: ``` CONFIG_DEBUG_KMEMLEAK CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=400 ``` We run periodic kmemleak scans on all nodes and stream the results into CloudWatch automatically. **Locking, concurrency, and atomic safety**: ``` CONFIG_LOCKDEP CONFIG_DEBUG_LOCKDEP CONFIG_DEBUG_SPINLOCK CONFIG_DEBUG_MUTEXES CONFIG_PROVE_LOCKING CONFIG_DEBUG_ATOMIC_SLEEP ``` Crucial for catching: - Deadlocks - Incorrect lock ordering - Sleeping under spinlocks - Misuse of atomic contexts **Stack correctness and overflow detection**: ``` CONFIG_DEBUG_STACKOVERFLOW CONFIG_DEBUG_STACK_USAGE CONFIG_STACKPROTECTOR CONFIG_STACKPROTECTOR_STRONG ``` Important because our module processes network packets on tight kernel stacks. **Memory corruption & SLUB debugging**: ``` CONFIG_PAGE_POISONING CONFIG_PAGE_OWNER CONFIG_DEBUG_PAGEALLOC CONFIG_SLUB_DEBUG CONFIG_SLUB_DEBUG_ON ``` These options make silent corruption visible immediately. **KCSAN – Race condition detection**: ``` CONFIG_KCSAN CONFIG_KCSAN_REPORT_RACE_ONCE=0 ``` Allows us to detect data races inside our driver when multiple sockets, namespaces, or processes interact simultaneously. **Tracing & instrumentation**: ``` CONFIG_FTRACE CONFIG_FUNCTION_TRACER CONFIG_KPROBES CONFIG_KRETPROBES CONFIG_DEBUG_INFO_DWARF5 CONFIG_BPF_EVENTS ``` These features let us dynamically hook into kernel functions, trace execution, and debug performance-critical paths. **Safety nets: fail fast on kernel anomalies**: ``` CONFIG_PANIC_ON_OOPS CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC CONFIG_BOOTPARAM_HUNG_TASK_PANIC ``` We force panics on issues that might otherwise go unnoticed, ensuring clear debugging signals. ## Building the AMI with Packer Packer automates: - Pulling the base AL2023 EKS AMI - Extracting the matching kernel source version - Rewriting .config with our debug features - Recompiling the kernel - Publishing versioned AMIs - Writing metadata into SSM ### Versioning Debug AMIs via SSM Parameters To track every debug kernel build, we use an SSM parameter naming scheme: - /riptides/debug/al2023/{timestamp} - versioned history - /riptides/debug/al2023/latest - pointer to the newest build This lets us: - Reproduce historical issues - Gradually roll out updated kernels in CI - Track evolution of kernel instrumentations This gives us a repeatable and versioned debug kernel, consistent across CI, EKS nodes, and standalone EC2 environments. ## A Debug-Optimized EKS Cluster We run our debug kernel in a dedicated EKS cluster, where Kubernetes naturally generates the noisy, complex environment that exposes timing issues and race conditions: - CNI networking - kubelet heartbeats - inter-pod TLS - container lifecycle events - DNS queries - frequent short-lived TCP connections This environment surfaces race conditions and timing issues that a quiet VM simply never reveals. ### Launch Template Configuration To run custom AMIs in EKS, we configure: - Node bootstrap scripts - Automatic registration to the EKS control plane - A debug user with developer SSH keys - CloudWatch agent for kernel logs and kmemleak outputs Every node becomes a self-contained debug environment. ### Centralized Kernel Observability Every debug node streams to CloudWatch: - dmesg - kmemleak scheduled scans - KASAN reports - lockdep warnings - hung task detections - panic dumps - stack traces ## Provisioning the Debug Cluster with Terraform Our debug workflow doesn't start at the kernel level, it starts with the infrastructure. To ensure that the full test suite runs in a consistent, reproducible, production-realistic debug environment, we fully automate provisioning of the debug EKS clusters using Terraform. The cluster is not a one-off experiment. It is a first-class production-grade environment with repeatable configuration, deterministic AMIs, and zero manual steps. We maintain a dedicated Terraform stack that provisions: - VPC, subnets, NAT gateways - EKS control plane - Managed node groups referencing our debug AMIs - IAM roles for nodes, CNI - CloudWatch log groups for kernel debugging output - SSM parameters for AMI discovery - Bootstrap user data configures the extra systemd services for CloudWatch observability and provisions developers' public SSH keys. Terraform gives us: - deterministic cluster creation - trackable infrastructure changes via Git - version-pinning of AMIs through SSM - truly reproducible debug environments ## Declarative Deployment Manifests: How We Describe Multi-Cluster Riptides Deployments Once Terraform brings the cluster online, the cluster is still empty. To ensure our debug pipeline remains fully reproducible, we define every deployment to every cluster using a declarative manifest file. This manifest acts as the single source of truth for both our control-plane cluster and workload clusters inside the debug environment. Unlike hand-written helm install commands or environment-specific scripts, the manifest explicitly declares: - which cluster to deploy to - where that cluster is located - what components belong there - which chart, version, registry, namespace, and values file to use This makes deployments deterministic, auditable, and safely CI-friendly. ```yaml cpCluster: clusterName: riptides-controlplane clusterRegion: eu-west-1 deployments: frontend: chart: frontend version: v0.1.4 registry: ghcr.io/riptideslabs/helm namespace: debug-cp valuesFile: frontend-values.yaml controlplane: chart: controlplane version: v0.1.9 registry: ghcr.io/riptideslabs/helm namespace: debug-cp valuesFile: cp-values.yaml workloadCluster: clusterName: riptides-debug-apps clusterRegion: eu-west-1 deployments: agent: chart: agent version: v0.1.17 registry: ghcr.io/riptideslabs/helm namespace: riptides-system valuesFile: agent-values.yaml ``` This manifest describes two clusters: - The Control-Plane Cluster - Runs the Riptides control plane and frontend components. - The Workload Cluster - Runs the Riptides driver and agent, where the debug kernel instrumentation lives. >The workload cluster cluster runs Amazon Linux 2023 debug kernels, both clusters are provisioned by Terraform, and both are deployed automatically through GitHub Actions. ### How GitHub Actions Consumes This Manifest The deployment workflow reads the manifest, and for each cluster: 1. Extracts clusterName and clusterRegion 2. Fetches AWS credentials from GitHub Secrets 3. Generates a kubeconfig dynamically 4. For every component, GitHub Actions executes a `helm upgrade --install` command using its configured values file. This makes deployments: - deterministic - reproducible - multi-cluster aware ![Riptides Debug Pipeline Diagram](../../assets/run-riptides-on-debug-kernel/run-on-debug.jpg) ## Periodic CI Runs of the Full Test Suite on the Fully Instrumented Debug Kernel Cluster debugging is powerful, but we also need periodic validation of the main branch on a debug kernel. We avoid doing this on every PR, since running our full test suite on a heavily instrumented debug kernel takes a significant amount of time. So we maintain a **self-hosted EC2 GitHub runner** using the same debug AMI. - A debug EC2 instance boots - The Riptides driver installs - Full test suite runs - Kernel logs + kmemleak outputs are collected - Results are sent back to GitHub - The instance is terminated This gives developers instant feedback on subtle kernel issues. ## Conclusion Building and debugging Linux kernel modules at scale is challenging, but with the right automation and infrastructure, it becomes manageable. By using EKS, custom debug kernels, Packer, Terraform, and GitHub Actions, Riptides has built a reliable pipeline for rapid iteration and deep kernel visibility. This approach speeds up development, increases deployment confidence, and provides a strong foundation for future growth. **Key Takeaways**: - Debugging kernel modules requires a special-purpose kernel, not a production one. - We have built a fully instrumented Amazon Linux 2023 debug kernel with KASAN, KFENCE, KCSAN, lockdep, and SLUB debugging. - Packer automates debug kernel compilation, AMI creation, developer access, EKS setup, and CloudWatch integration. - We use SSM parameters to version every debug AMI, giving us a reproducible history of kernel builds. - Terraform provisions the debug clusters declaratively, creating VPCs, IAM roles, EKS control planes, and node groups consistently and repeatably. - Our GitHub Actions deployment pipeline automatically deploys Riptides components to the clusters using manifest files. - Our debug EKS cluster provides real-world traffic for validating identity enforcement, TLS behavior, and socket interactions. - CloudWatch centralizes kernel-level logs, kmemleak output, concurrency detector warnings, and panic traces. - A self-hosted EC2 GitHub runner ensures periodically the main branch is tested on the debug kernel. **The result is a reproducible, automated debugging pipeline that lets us catch kernel bugs under production-like conditions.** --- ## Bringing SPIFFE to OAuth for MCP: Secure Identity for Agentic Workloads - URL: https://blog.riptides.io/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads - Published: 2025-11-17 - Author: Zsolt Rappi - Category: SPIFFE - Tags: SPIFFE, Oauth2, AI, OIDC, MCP, Agentic AI remains the headline topic of our industry, but the consensus is becoming clear: AI agents are **workloads**. They require identities, credentials, and policies just like any other component in a distributed system. What makes them different is not that they need identity, but how they use it: demanding fine-grained authorization controls that reflect their dynamic behavior, decision boundaries, and delegated autonomy. Treating AI as a first-class workload identity subject unlocks consistent governance, observability, and trust across both human and machine interactions. At the same time, SPIFFE has emerged as the industry standard for workload identity, a standard we’ve believed in from the very beginning, and one of the core principles around which Riptides was built. SPIFFE provides a consistent and secure way to issue, rotate, and verify credentials for workloads, services, and now AI agents at scale. Yet, until recently, one crucial piece of the puzzle was missing, the connective layer that could bring SPIFFE-based identity into the agentic world. ## MCP: The Emerging Fabric of Agentic Communication The Model Context Protocol (MCP) is rapidly becoming the backbone for secure, structured communication between AI agents and the tools or data providers they depend on. It defines a common language for agents to discover, describe, and interact with capabilities hosted across a distributed network of MCP servers. As MCP evolves into the connective tissue of agentic ecosystems, it inherits a critical challenge: **_how to use OAuth securely while supporting massive numbers of dynamically registered, short-lived agents_**. In earlier posts, we explored how [Riptides secures MCP communication](/blog/securing-mcp-communication-with-riptides) using workload-based identity and [kernel-enforced controls](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), and how [SPIFFE and OAuth2 together can form the foundation for workload authentication](/blog/spiffe-meets-oauth2-current-landscape-for-secure-workload-identity-in-the-agentic-ai-era) in OAuth-based systems. In this post, we take that vision further and discuss how we have implemented the emerging OAuth RFC drafts for SPIFFE-backed OAuth and building a demo MCP application that showcases this integration in a Riptides environment. This fusion unlocks a new model of **self-registering, self-authenticating workloads, particularly AI agents**, forming a key building block for scalable, autonomous, and verifiable agentic ecosystems. ## The Identity Problem in Agentic AI Agentic AI systems rely on dynamic, autonomous workloads that come and go as tasks evolve. An agent might spawn new processes to handle context expansion, fetch data from a remote MCP server, or delegate work to a specialized sub-agent. In every case, the system must **establish mutual trust between participants** and it must do so automatically, at scale. Traditionally, workload authentication in OAuth has been secret-based: each client is registered at the authorization server with a \*_client ID and secret_. This model breaks down in dynamic environments where workloads are ephemeral and scale on demand, since credentials must be provisioned, stored, and rotated securely. In an agentic ecosystem where hundreds or thousands of short-lived agents and tool instances may interact fluidly, this approach is simply not sustainable. Static credentials also violate core zero-trust principles: identity should be **ephemeral, verifiable, and bound to the workload itself**, not tied to configuration files or long-lived and shared secrets. ## SPIFFE and OAuth SPIFFE (Secure Production Identity Framework for Everyone) defines a standard for issuing and verifying cryptographically strong identities for workloads. Each workload receives a SPIFFE Verifiable Identity Document (SVID), an X.509 certificate or JWT, that attests to its identity within a trust domain (for example, spiffe://riptides.io/workload/mcp-server). These credentials are short-lived, automatically rotated, and anchored in strong cryptographic trust roots managed by an issuer. They establish _who_ a workload is, verifiably and dynamically, without relying on pre-shared secrets. OAuth, by contrast, defines how workloads obtain access tokens for APIs, but it traditionally assumes that clients are **pre-registered with a client ID and secret** at the authorization server. That static model works for stable applications but breaks down in systems where workloads are ephemeral or autonomously created, such as agentic AI. ![dyn-client-reg](../../assets/spiffe-oauth-poc/dynamic-client-reg.jpg) By treating SPIFFE credentials as **software statements** within OAuth, we can bridge this gap. A workload can prove its identity, self-register as an OAuth client, and immediately perform standard OAuth flows, all without manual configuration or stored secrets. Here's how [Dynamic Client Registration with SPIFFE](https://datatracker.ietf.org/doc/draft-kasselman-oauth-dcr-trusted-issuer-token/) would look like: ![dyn-client-reg](../../assets/spiffe-oauth-poc/dynamic-client-reg-spiffe.jpg) ## SPIFFE-Backed OAuth in Riptides: Implementing Emerging RFCs The [Riptides Controlplane](https://riptides.io/request-a-demo) functions as both a SPIFFE issuer and an OAuth/OIDC authorization server within the Riptides ecosystem. This architecture made it straightforward to layer a SPIFFE-based authentication system in front of the existing OIDC server. Riptides-managed workloads, [which automatically receive SPIFFE credentials at the kernel level](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), can seamlessly authenticate with the Controlplane’s OAuth endpoints and obtain tokens to access protected resources, such as remote MCP servers. In our demo setup, both the MCP client and MCP server run as Riptides-managed workloads. Each receives a valid SPIFFE identity issued by the Controlplane and injected at the kernel level. When communication is required, for instance, when an MCP client calls a remote MCP server, the workloads use their SPIFFE credentials to self-register and authenticate via OAuth, without any manual configuration. At a high level, the lifecycle works as follows: - Obtain SPIFFE credentials: Workloads receive SVIDs from the Controlplane, automatically injected into traffic at the kernel level, without code changes required. - Present credentials: The agent uses its SVID as the proof element in OAuth flows (client registration, token request, etc.). - Validate credentials: The OAuth server verifies the SVID against the trusted issuer. - Token introspection: The MCP server validates tokens at the Controlplane’s introspection endpoint, authenticating itself via its own SVID. This approach transforms OAuth into a dynamic, identity-aware trust fabric, naturally aligned with MCP’s model of tool discovery, invocation, and secure agent-to-agent communication. ![dyn-client-reg](../../assets/spiffe-oauth-poc/integration.jpg) The two workloads, the agent and the MCP server, are defined in Riptides using the following `WorkloadIdentity` specifications: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: weather-mcp-server spec: scope: agentGroup: id: spiffe-mcp-demo-agent-group selectors: - process:uid: [501, 1001] process:name: [python] process:cmdline: python mcp_server.py workloadID: weather-mcp-server --- apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: weather-agent spec: scope: agentGroup: id: spiffe-mcp-demo-agent-group selectors: - process:uid: [501, 1001] process:name: [python] process:cmdline: python mcp_client.py workloadID: weather-agent ``` ## Why SPIFFE-Backed OAuth Matters for MCP and Agentic AI With SPIFFE-backed OAuth in Riptides, workloads gain several operational and security advantages: - **No manual setup:** Workloads self-register automatically using identity-based proof. - **No code changes:** Identities are issued automatically and injected into traffic at the kernel level. - **No secrets at rest:** Authentication relies on short-lived, ephemeral certificates instead of static client secrets. - **Cross-domain scalability:** Trust is based on cryptographic identity rather than configuration. - **Automatic rotation:** SPIFFE’s short-lived credentials ensure continuous renewal and revocation. - **Traceable accountability:** Every token and registration event is tied to a verifiable workload identity, which can be audited and correlated with runtime telemetry. - **Reduced attack surface:** Removing client secrets and configuration files minimizes the risk of credential leakage. - **Consistent policy enforcement:** Identity and authorization policies are applied uniformly at the Controlplane and enforced at the workload level. For MCP in particular, this enables remote servers to accept connections from legitimate agents on demand, enforce policies based on concrete workload identities, and avoid distributing sensitive credentials at scale. Agents can discover and invoke tools seamlessly, while operators maintain full governance, traceability, and auditability across the system. Together, these benefits enable organizations to scale agentic AI systems securely, maintaining both operational efficiency and a strong security posture while preserving full auditability and governance. ## Operational Challenges and Practical Considerations While this model is powerful, several operational considerations must be addressed: - **Token binding and proof-of-possession (PoP):** Tokens should be cryptographically bound to the presenting identity to prevent replay attacks. - **Client lifecycle management:** Even with dynamic registration, workloads require proper lifecycle management, including deregistration, credential revocation, and policy updates, all of which must be auditable. - **Trust-domain governance:** Operators need to define and enforce which SPIFFE issuers are trusted, along with the constraints that apply across trust boundaries. Addressing these considerations ensures that dynamic, SPIFFE-backed OAuth systems remain secure, auditable, and manageable at scale. ## Conclusion: Dynamic Trust for Agentic AI Treating SPIFFE credentials as software statements for OAuth eliminates manual configuration, removes static secrets, and enables secure, automated interactions between agents and tools. For MCP and the broader agentic AI ecosystem, this approach is foundational. It transforms OAuth from a friction point into a dynamic, identity-aware trust fabric, turning identity itself into the API that governs who can do what, when, and why, all in a scalable, auditable, and automated manner. At Riptides, we envision a world where all non-human workloads, from ephemeral AI agents to distributed services, can authenticate, authorize, and interact securely without manual intervention. By combining kernel-enforced SPIFFE identities with dynamic OAuth flows, we aim to make trust, governance, and policy enforcement invisible but verifiable, so teams can focus on building capabilities rather than managing credentials. This is the foundation for truly autonomous, secure, and auditable agentic systems at scale, where identity drives both innovation and safety. --- ## Announcing oci-req-signer-c: A Lightweight C Library for Oracle Cloud Request Signing - URL: https://blog.riptides.io/announcing-oci-req-signer-c-a-lightweight-c-library-for-oracle-cloud-request-signing - Published: 2025-11-03 - Author: Sebastian Toader - Category: Kernel - Tags: kernel, oci At Riptides, we’ve been building a solution that securely federates workloads with Oracle Cloud Infrastructure (OCI) — as well as with AWS, GCP, and Azure. During this process, we faced a technical challenge that led us to create — and now open source — **[oci-req-signer-c](https://github.com/riptideslabs/oci-req-signer-c)**, a minimal C implementation of the OCI request signing algorithm. This is part of our broader effort to enable deep, kernel-level identity federation across all major cloud providers, a topic we explored in detail in our post, [Why Cloud-Native Federation Isn’t Enough for Non-Human Identities in AWS, GCP, and Azure](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure). ## The context In our architecture, workloads obtain **temporary OCI credentials (User Principal Session Tokens)** by exchanging **ID tokens** issued by the **Riptides Control Plane**. These credentials are then **injected into OCI API requests on the fly** entirely transparently to the workloads themselves. Here’s the key technical detail: Each OCI API request must include a valid **Authorization header** signed with the **RSA private key** associated with the temporary credential. In our system: - An **RSA keypair** is generated dynamically for each session. - The **public key** is sent to OCI along with the **ID token** to request a temporary credential. - The **private key** stays in our control and is used to **sign outgoing OCI API requests**. All of this happens **in kernel space**, inside a **Linux kernel module** responsible for intercepting and modifying outgoing HTTP(S) traffic. >You can also read more about our kernel-based architecture and how we integrate SPIFFE and kTLS in [Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) ## The problem While OCI provides official SDKs for Python, Java, Go, and others, none of them include a C implementation of the signing algorithm. Moreover we couldn't find one which is suitable for use in kernel modules or other low-level environments. We needed a **pure C** implementation, with **no dynamic allocations** and **no dependency on libc features** unavailable in the kernel. When we couldn’t find one, we decided to build it. >We encountered similar challenges with AWS request signing, which led us to develop and open source [libsigv4 — a portable C library implementing AWS SigV4 with kernel compatibility](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility). ## Our solution — [oci-req-signer-c](https://github.com/riptideslabs/oci-req-signer-c) [oci-req-signer-c](https://github.com/riptideslabs/oci-req-signer-c) is a **lightweight, dependency-minimal C library** for computing OCI-style HTTP request signatures. It’s designed to work both in **user space** and **kernel space**, or anywhere a small, efficient, and auditable signer is required. ### Features - **Self-contained** — minimal dependencies, builds as `.a` or `.so` - **No dynamic memory allocation** — kernel and embedded safe - **RSA private key (DER format)** support - **Simple API** for signing any HTTP request - **Customizable system header handling** (`OCI_SYSTEM_HEADER`) - Produces the HTTP Authorization header (key and value) for OCI requests, including the computed signature and all required metadata ## Example usage ```c #include "oci_signer.h" // Example-specific buffer size constants #define EXAMPLE_AUTH_HEADER_MAX_LEN 4096 int main() { // Sample request parameters const char *method = "GET"; const char *uri = "/20160918/instances"; const char *host = "iaas.us-phoenix-1.oraclecloud.com"; const char *date = "Thu, 05 Jan 2014 21:31:40 GMT"; const char *payload = ""; // Example key_id in format: tenancy/user/fingerprint const char *key_id = "ocid1.tenancy.oc1..aaaaaaaaba3pv6wkcr4jqae5f15p2b2m2yt2j6rx32uzr4h25vqstifsfdsq/" "ocid1.user.oc1..aaaaaaaat5nvwcna5j6aqzjcaty5eqbb6qt2jvpkanghtgdaqedqw3rynjq/" "20:3b:97:13:55:1c:5b:0d:d3:37:d8:50:4e:c5:3a:34"; // Alternative: Example key_id using session token format // const char *key_id = "ST$aaaaaaaa7tz3aaaaaaaaaymq2maaaaaaabfwiljtdnfgqaaaa"; // Initialize OCI signer parameters oci_signer_params_t signer_params = {0}; // Zero-initialize the structure // Set the private key in DER format using the binary type signer_params.private_key.data = ...; signer_params.private_key.len = ...; signer_params.key_id = oci_signer_string((unsigned char *)key_id); signer_params.method = oci_signer_string((unsigned char *)method); signer_params.uri = oci_signer_string((unsigned char *)uri); signer_params.payload = oci_signer_string((unsigned char *)payload); signer_params.headers[0].key = oci_signer_string((unsigned char *)"host"); signer_params.headers[0].value = oci_signer_string((unsigned char *)host); signer_params.headers[1].key = oci_signer_string((unsigned char *)"date"); signer_params.headers[1].value = oci_signer_string((unsigned char *)date); signer_params.num_headers = 2; // Set the required crypto functions signer_params.sha256 = ...; signer_params.rsa_sign_sha256 = ...; signer_params.base64_encode = ...; // Buffer for the Authorization header unsigned char *auth_header_buf = malloc(EXAMPLE_AUTH_HEADER_MAX_LEN); memset(auth_header_buf, 0, EXAMPLE_AUTH_HEADER_MAX_LEN); oci_signer_header_t auth_header = { .value = {.data = auth_header_buf} }; int status = oci_signer_sign(&signer_params, &auth_header, EXAMPLE_AUTH_HEADER_MAX_LEN); if (status == OCI_SIGNER_OK) { printf("Authorization header value: %.*s\n", auth_header.value.len, auth_header.value.data); } else { fprintf(stderr, "Failed to sign the request\n"); } free(auth_header_buf); free(key_data); } ``` ## How we use it at Riptides This library powers our **kernel-space credential injection system**, enabling OCI authentication **without any user-space process or SDK**. Our kernel module: 1. Obtains a **User Principal Session Token** (temporary credential) from OCI using an **ID token** and a **ephemeral RSA public key**. 2. Intercepts outgoing OCI API requests. 3. **Injects** the temporary credential and **recomputes the Authorization signature** with [oci-req-signer-c](https://github.com/riptideslabs/oci-req-signer-c) using the matching private key. This enables **transparent, secure, and short-lived identity federation** for workloads accessing OCI resources. ## Why we open sourced it We realized this signing challenge isn’t unique to us. Anyone building **low-level OCI integrations**, **embedded systems**, or **custom networking stacks** might need a C implementation of OCI’s signing logic. By open sourcing [oci-req-signer-c](https://github.com/riptideslabs/oci-req-signer-c), we aim to: - Help developers implement **secure OCI integrations** in constrained environments - Enable experimentation in **edge and kernel space** contexts - Encourage community collaboration and extensions ## Get started You can check out the source here: 👉 [https://github.com/riptideslabs/oci-req-signer-c](https://github.com/riptideslabs/oci-req-signer-c) We’d love feedback, especially from embedded developers running into the same challenges. ## Closing thoughts What started as an internal engineering need to make OCI request signing possible inside a Linux kernel module has evolved into a reusable, open-source library for the broader community. We’re excited to share [oci-req-signer-c](https://github.com/riptideslabs/oci-req-signer-c) and look forward to seeing how others use it in their projects. --- ## Growing Threat of npm Supply Chain Attacks and the Runtime Fix That Stops It - URL: https://blog.riptides.io/growing-threat-of-npm-supply-chain-attacks - Published: 2025-11-03 - Author: Marton Sereg - Category: Identity - Tags: AI, Supply-chain, Secret-injection Speed in modern engineering comes from reusing open-source components, but that same dependency chain has become one of the most exploited attack surfaces on the internet. This post walks through a realistic npm supply-chain compromise, how attackers turn a poisoned package into a full-blown breach, and a clean demo that shows a practical mitigation: **just-in-time secret injection.** ## **The Attack Story** Supply-chain compromises happen across every language ecosystem — PyPI, RubyGems, Go modules, but npm remains the most frequently targeted. Over the past few years, we've seen large-scale incidents (like **[Shai-Hulud recently](/blog/shai-halud-and-the-secret-hunters-how-npm-installs-turn-into-intrusions))**, where a malicious npm package silently spread through CI systems and exfiltrated credentials from thousands of machines. That's why we're using npm as our example in this post. It's representative of how real-world supply-chain attacks unfold across any modern stack. Every supply-chain breach starts the same way: *with trust*. You install a dependency, like a new logging utility or a small helper buried ten layers deep, and assume it does what it says on the tin. But a single compromised maintainer account or poisoned package version can quietly turn that trust into an entry point. A malicious package can execute automatically during install or build-time lifecycle scripts, such as *`preinstall` or `postinstall`*. From there, the payload runs in the context of your CI pipeline or developer environment with all the same privileges your tools have. That’s where the real damage happens. These payloads are rarely loud or destructive; they’re designed to blend in. Most are short, heavily obfuscated scripts that scan for secrets in environment variables, `.npmrc` tokens, cached SDK credentials, or local kubeconfigs. Once they find anything interesting, they exfiltrate it. Often via a single `POST` request to an attacker-controlled endpoint disguised as a harmless telemetry or analytics domain. Armed with these secrets, an attacker can publish backdoored images to your container registry, or inject a hidden GitHub Actions workflow that grants long-term persistence. The poisoned package was just the initial infection, the stolen credentials are the real payload. From there, the path is well-worn: the attacker waits for your deployment pipeline to pull their backdoored image, which eventually runs inside a Kubernetes pod with access to sensitive runtime secrets — OpenAI or Anthropic API keys, database credentials, or service tokens. Once inside, they can exfiltrate data, explore internal APIs, and move laterally across your environment. In other words: a single malicious npm install can become a full-scale cloud breach. ## **Why Static Scanners Aren’t the Whole Story** Most teams already run dependency and vulnerability scanners and they absolutely should. They catch outdated packages, known CVEs, typosquats, and dangerous permissions before they ship. But scanners live in a world of *known vulnerabilities*. Supply-chain attacks thrive in the world of *unknown behavior*. By the time a signature or rule exists, the exploit has already run in thousands of build environments. Even the best scanners share a couple of unavoidable blind spots: - **Metadata ≠ behavior, and install-time ≠ runtime.** Scanners evaluate package names, versions, and known vulnerabilities — they don’t observe what the code actually does when it runs. A new or modified package can execute obfuscated install-time logic that scrapes environment variables or fetches a payload; once executed, the scanner has already done its job and won’t see the runtime exfiltration. - **Signal fatigue.** Security teams drown in alerts. Dozens of “medium” findings pile up, and a single critical anomaly can hide among the noise or get postponed until “after the release.” So even with the best coverage, a package can pass every check, execute malicious code, and leave no trace until it’s too late. That’s why **defense in depth** matters. Static analysis tells you *what you’re installing*; runtime guardrails decide *what it’s allowed to do once it runs.* The rest of this post focuses on that second layer: how runtime identity and just-in-time secret injection make a compromise far less valuable for an attacker. ## **How Attackers Move Laterally** Once attackers get code execution, they follow a fast, repeatable playbook: - **Grab credentials:** scan env vars, .npmrc, kubeconfigs, CI tokens. - **Pivot to CI/registry:** push backdoored images or add workflows to gain persistence. - **Run in production:** poisoned images or workflows deploy into pods/servers that receive runtime secrets. - **Harvest and escalate:** use DB keys, cloud tokens, or service accounts to access more systems. - **Persist and monetize:** create long-lived accounts, exfiltrate data quietly, or sell access. The simple lesson: if secrets are discoverable at runtime, a small compromise becomes a full breach. Remove those secrets from the attack surface and you dramatically reduce the blast radius. ## **Just-in-Time Secret Injection** Just-in-time injection means credentials aren’t baked into images, env vars, or files. They’re provisioned only to the specific process that needs them, just when it needs them. Delivery can happen in several ways: placed “on the wire” (for example, by adding headers to outbound HTTP calls), or written to an ephemeral file that’s only readable by that process. Why this matters: - **No persistent target:** If keys never exist as files or long-lived env vars inside a pod, there’s nothing for an install-time or run-time scraper to grab. - **Process-scoped delivery:** Injection is tied to a SPIFFE workload identity. A different process does not receive the secret, even in the same VM or pod. - **Minimal operational friction:** Injection happens at runtime and doesn’t require code changes or secret rotation across images. Policies can be updated centrally and take effect immediately. - **Auditable and revocable:** Every injected event can be logged and audited. If a key is suspected, you can revoke the provider-side secret and the workload loses access without redeploying images. - **Complementary to existing controls:** SCA and static policies still matter. Injection is an additional layer that greatly reduces the payoff of any successful compromise. ## **Demo walkthrough — support-assistant, Postgres, and a poisoned npm package** This demo shows the exact attack chain described above, and how just-in-time injection breaks it. ### **Setup** - A small web chat (Support-Assistant) that calls an LLM provider (OpenAI/Anthropic). The LLM uses a Postgres database of support tickets as a tool: the assistant issues queries to Postgres to fetch and summarize ticket data. - A simulated poisoned npm payload that, when run in a build or container, scans environment variables and posts any found secrets to an external sink (we use a local `ngrok`). - Kubernetes deployment for the backend, with its API keys normally delivered as a Secret into the pod environment. ### **Walkthrough** 1. **Show the app working** At first everything looks fine. The Support-Assistant UI works as expected: you type a question, it fetches results from Postgres, asks the LLM for a summary, and returns the answer. It's a completely ordinary helper agent, until one of its dependencies turns hostile. ![demo-working-ui.png](../../assets/supply-chain/1-demo-working-ui.png) 1. **A poisoned package scrapes the environment** A malicious npm package quietly executes during runtime and starts scanning environment variables. The screenshot below shows what happens next: our simulated payload sends the collected keys to an ngrok endpoint. The POST request includes both the OpenAI and Anthropic API keys. ![demo-ngrok-with-apikeys.png](../../assets/supply-chain/2-demo-ngrok-with-apikeys.png) The **Riptides Connection Inventory** page shows these outbound requests to unknown ngrok IPs. But a single successful request like this is all an attacker needs to steal API keys and escalate. ![demo-riptides-connections2.png](../../assets/supply-chain/3-demo-riptides-connections2.png) Even though the poisoned package doesn’t break the application itself, it silently opens connections and exfiltrates secrets. In a real incident, that one request would be enough to pivot deeper into your infrastructure. 1. **Remove static credentials from the pod** Next, we strip the pod of its persistent secrets by setting the API keys to none. Now the system has nothing to leak, but the application also fails to call the LLM. In practice, you’d configure secret injection first, then remove environment variables, but here we intentionally break the app to show that the keys really are gone. ![demo-ui-not-working.png](../../assets/supply-chain/4-demo-ui-not-working.png) 1. **Apply Riptides just-in-time injection** Now we turn on Riptides’ **on-the-wire credential injection**. Instead of handing credentials to the environment, Riptides injects them dynamically into legitimate requests at runtime. Here's a small configuration snippet of how it's configured for an identity: ![demo-riptides-egress.png](../../assets/supply-chain/5-demo-riptides-egress.png) The app immediately resumes normal behavior without restarting pods or re-deploying images. ![demo-working-ui-2.png](../../assets/supply-chain/6-demo-working-ui-2.png) 1. **Re-run the scraper** Finally, we check the malicious scraper again. It still executes, but now it has nothing to steal. The exfiltrated payload shows empty values: the exploit’s payoff is gone. In a real environment, that one change — removing persistent secrets and injecting them just-in-time — turns a full-scale breach into a contained event. ![demo-ngrok-without-apikeys.png](../../assets/supply-chain/7-demo-ngrok-without-apikeys.png) ## **Practical recommendations — what to do today** 1. **Treat static scanning as one layer, not the answer.** Keep dependency scanning and vetting in place, but combine them with runtime controls so a slipped package has no persistent payoff. 2. **Eliminate persistent credentials in images and pods.** Stop shipping long-lived secrets in images or as env vars. Replace them with short-lived or injected credentials for high-value targets (LLM providers, DBs, cloud admin scopes). 3. **Bind secrets to workload identity, not host or pod.** Deliver credentials only to the process that needs them (process-scoped identities / SPIFFE-style). A reverse shell in the pod should not automatically inherit access. 4. **Make injection auditable and revocable.** Log every injection and policy change. Centralized revocation and audit trails reduce MTTR and make investigations possible. 5. **Harden CI and publish paths.** Limit CI runner privileges, rotate publish tokens, and watch registry publishes and workflow changes — make it harder for initial escalation to succeed. ## **Closing — measurable risk reduction** A poisoned npm package is a plausible, common starting point for a large breach. You can’t catch every compromised dependency, but you *can* make compromises far less valuable. Removing persistent runtime secrets and delivering credentials just-in-time to a verified workload identity converts a likely data breach into a contained incident. That shift — fewer secrets exposed, fewer privileges leaked, and clearer audit trails — is the kind of measurable risk reduction security teams, engineering leaders, and auditors will actually care about. If you’re interested in other secret-injection posts, check out our examples — [On‑Demand Credentials: Secretless AI Assistant (GCP)](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp) and [On‑the‑Wire Credential Injection: Secretless AWS/Bedrock](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example). --- ## When eBPF Isn’t Enough: Why We Went with a Kernel Module - URL: https://blog.riptides.io/when-ebpf-isnt-enough-why-we-went-with-a-kernel-module - Published: 2025-10-27 - Author: Balint Molnar - Category: Security - Tags: kernel, ebpf, security, tls ## The Core Question: Do We Really Need a Kernel Module? TL;DR: eBPF is perfect for watching and shaping packets, but when you need to create, sign, and protect the identities behind them, you need the kernel itself. Whenever we explain Riptides to customers, at conferences or meetups, there’s always one question: “*Why a kernel module instead of eBPF?*” It’s a fair one, as eBPF has transformed observability and network control in Linux. It’s safe, flexible, and powerful for tracing, filtering, and enforcing lightweight policies. But when it comes to cryptography, identity, and deep kernel integration, eBPF’s sandboxed nature becomes a limiting factor. This post breaks down that choice - why Riptides runs a kernel module instead of eBPF, what that enables us to do, and what eBPF’s sandboxed model still can’t. We’ll look at concrete use cases as key generation, TLS orchestration, credential injection, and just-in-time credential delivery and show where eBPF reaches its limits. Our kernel module operates at the TCP layer and intercepts incoming/outgoing connections. It issues ephemeral X.509 certificates and binds those keys/certs to the process that initiated the connection. On a new TCP connection the driver: - checks destination and policy, - initiates a TLS handshake, - establishes kTLS for data-in-transit (when appropriate). Since enterprises today are tightly coupled with cloud providers, external federation—enabling systems to trust identities without long-lived secrets—is critical. We’ve written [why cloud-native federation isn’t enough for non-human identities](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure), but in short: workloads often authenticate with short-lived ID tokens from external providers. These tokens must be securely retrieved and rotated. While cloud providers offer SDKs and tools, they still require a root secret to bootstrap token retrieval, leaving gaps in distribution and isolation. Riptides solves this in two ways: - **Dynamic sysfs delivery** - expose credentials just-in-time via sysfs, scoped so only the intended workload can read them exactly when needed. - **On-the-wire injection** - replace auth headers with valid short-lived credentials at the point of egress, transparently to the application. ## What We Do at Riptides Riptides is the non-human identity fabric for workloads and AI agents. We eliminate credential sprawl by issuing and rotating short-lived SPIFFE based identities automatically, moving access control from the network to the workload using familiar primitives — X.509 certs, JWT tokens, TLS, etc - without requiring application changes. > A good technical overview of our solution is described in the [Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) post Our architecture consists of both a kernel module and a user-space agent. The kernel module operates at the TCP layer, intercepting new connections. When a workload innitiates a connection, the kernel module: - validates the destination and policy, - performs a TLS handshake, - and enables kTLS for encryption at the record layer. Riptides issues ephemeral X.509 certificates and binds them to the process that initiated the connection, ensuring that identities are both short-lived and process-scoped. As most enterprises today use cloud providers, federation is critical as well. Riptides can also federate accross cloud providers, or inject secrets/credentials towards 3rd parties, without needing a bootstrap secret. We do this in two different ways: - Dynamic sysfs delivery: credentials are exposed just-in-time through sysfs, scoped to the requesting workload only. - On-the-wire injection: by transparently replacing outbound authentication headers with short-lived credentials, so applications never handle them directly. ### Some of The Key Tasks We Handle - Asymmetric key and certificate generation (keys, CSRs, certs) - TLS handshake orchestration and kTLS enablement - On-the-wire credential injection (signing and header replacement) - Just-in-time credential exposure via sysfs ## Kernel-based solutions (what we implemented and why) ### Certificate generation (keys, CSR, certs) The [Linux kernel crypto API](https://docs.kernel.org/crypto/architecture.html) provides certificate generation or big-integer asymmetric key generation. The asymmetric-key subsystem expects keys to be supplied/loaded from userspace — i.e., the kernel API can sign/verify with a key that’s already present, or create a signature blob (**create_signature**), but it won’t create RSA/ECDSA keypairs or produce CSRs for you. We needed in-kernel key generation to tightly bind identities to processes and to avoid exposing long-lived private material in userspace. Instead of reinventing crypto primitives, we ported a small crypto library to run inside the kernel: - initially we ported [BearSSL](https://bearssl.org) (adapting libc usage to kernel headers), which gave us asymmetric keypair generation inside the kernel. BearSSL is compact and suitable for constrained environments. - we experimented with a tiny WebAssembly approach to produce CSRs in a sandboxed way (CSR generation in WASM and PEM output to userspace). That gave us a separation boundary, but we later refactored away from in-kernel WASM for security and complexity reasons — see our blogpost [From Kernel WASM to user-space policy evaluation.](/blog/from-kernel-wasm-to-user-space-policy-evaluation-lessons-learned-at-riptides) - and moved to supporting native CSR and certificate generation within the kernel. We keep CSR issuance in the control plane (userspace CA or external CA) but generate keys and create CSRs in privileged kernel context so private material never leaves the kernel memory. ![two arch](../../assets/ebpf-modul/two-arch.jpg) ### TLS Handshake At the time of publishing this blogpost, the kernel crypto API is unable to handle TLS handshake directly inside the kernel. There was/is a debate on the mailing list whether implement the TLS 1.3 [handshake inside the kernel](https://lwn.net/Articles/896746/). kTLS exists in Linux but it handles only the record layer - encryption/decryption of TLS records - not the full handshake. The kernel networking stack offers a [handshake offload API](https://docs.kernel.org/networking/tls-handshake.html): a socket file descriptor can be passed to a userspace handshake agent (via netlink), the agent completes the handshake, then returns the socket back to the kernel and sets the TLS ULP (userland handshake model). Projects like Oracle’s [ktls-utils](https://github.com/oracle/ktls-utils) show that pattern in practice. Others like Tempesta releasing tech studies about the performance of the [handshake happening inside the kernel](https://tempesta-tech.com/research/kernel_tls_hs.pdf). We chose the second approach for Riptides: we perform the TLS handshake inside the kernel module using the in-kernel crypto stack that we ported. Doing the handshake inside the kernel: - lets us bind cert/key lifecycle to kernel object lifecycle and to process context, - allows immediate kTLS activation, avoiding round trips to the user agent during connection setup, - simplifies transparent on-the-wire credential replacement because we control both handshake and record protection. ### On-the-wire credential injection We need to inject or replace authentication headers (for AWS, GCP, etc.) so workloads never hold long-lived cloud credentials. To do this transparently we must: - reliably attribute a connection to a specific process (the principal authorized to obtain/consume credentials), - access plaintext application bytes before encryption (or terminate/reinitiate when encrypted), and - compute any provider-specific signatures (AWS V4 signing, etc.) and rewrite headers. I am not going to dive into all the details if you are interested check out our [blogpost](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) about credential injection. Implementation highlights: - the module inspects outbound buffers, since we manage TLS end-to-end we can peek at plaintext before encryption. - if an application already encrypted its data in userspace (so we only see ciphertext), Riptides terminates the connection and reinitiates it using ephemeral credentials under our control, then performs the injection. This is transparent to the client. - header rewriting requires robust HTTP parsing and stateful handling because application payloads may span multiple packets/segments. The kernel module keeps the necessary per-connection state to reconstruct requests where needed. - for provider flows that require extra signing (e.g., AWS signed headers), we compute the signatures dynamically in the kernel and replace the client's header with a freshly signed one. ![userspace-enc](../../assets/ebpf-modul/userspace-enc.jpg) >Data is encrypted in user space. ![plaintext-data](../../assets/ebpf-modul/plaintext-data.jpg) >Plaintext data from user space This is complex, but feasible in kernel space because we control handshake, keys, and send hooks. **Credential exposure via sysfs**: Acourding to linux manual page, sysfs is a kernel pseudo-filesystem exposing kernel objects. Riptides uses sysfs to present short-lived credentials just-in-time to a workload: the kernel module creates sysfs nodes with appropriate access controls and serves credentials such that only the requesting process can open/read them at the allowed time window. Creating scoped sysfs entries and enforcing in-kernel access control is straightforward within a kernel module, let's see how all these problems could be solved from eBPF. ## eBPF-based solutions - what eBPF can and cannot do eBPF is powerful for observability and safe programmability, but it runs in a sandbox with limited helpers and no arbitrary filesystem / crypto capabilities. Below I keep the same problems and discuss eBPF options and gaps. ### Certificate generation eBPF cannot generate asymmetric keypairs or produce CSRs/certs. It lacks the math libraries, heap, and API surface required for big-integer crypto operations, and you cannot realistically port an SSL/TLS stack into eBPF. So certificate/key generation must remain in userspace, with keys shared to eBPF only in limited forms (if at all). ### TLS handshake eBPF cannot implement a TLS handshake. Handshake logic requires stateful complex crypto and network interactions far beyond eBPF's intended use. The kernel provides only the record-layer kTLS facilities, handshake must be performed by a TLS library (userspace or kernel). eBPF’s strength here is observation - not managing TLS state machines. ### On-the-wire injection With eBPF we can also observe user data sent over the wire — this is very much eBPF’s territory. It offers multiple options, the main difference between them is where the inspection happens. Unfortunately there is no silver-bullet: a program that can inspect encrypted data generally cannot modify it, so we often need to transfer context between eBPF programs or combine eBPF with other mechanisms. eBPF hooks differ significantly depending on where they attach in the stack: we move from as close to the hardware as possible up to the user application. #### XDP(eXpress Data Path) - earliest interception XDP runs at the network device driver level and processes each incoming packet before it reaches the kernel networking stack, so it provides the highest performance with minimal overhead. Its primary use cases are things like DDoS protection (for example, [Cloudflare](https://blog.cloudflare.com/defending-the-internet-how-cloudflare-blocked-a-monumental-7-3-tbps-ddos/) has used XDP eBPF programs for large-scale mitigation) and high-performance filtering. In a nutshell, XDP programs return one of several actions that indicate what the kernel should do next: - **XDP_PASS:** pass the packet up the networking stack (optionally after in-place modification) - **XDP_TX:** resend the packet back to the same network port where it arrived on. - **XDP_REDIRECT:** redirects the packet to a different location. The location could be a different networking interface or a different CPU, or to the user-space. - **XDP_ABORTED:** used for debugging, triggers an exception and a trace log entry. - **XDP_DROP:** drop the packet. Because XDP operates at such a low level, parsing application-layer protocols (essential for HTTP header injection) is error prone and difficult. XDP runs on raw frames and processes packets individually, so reconstructing and parsing a complete HTTP request often requires complex, manual TCP reassembly and state handling when the data spans multiple segments. ![xdp](../../assets/ebpf-modul/xdp.jpg) Theoretically, if parsing succeeds, we then need to modify the HTTP header — which could be done in an XDP program. However, XDP provides no checksum or socket helpers, so everything must be implemented by hand. Implementing a certificate-signing mechanism here (essential for providers like Amazon) is effectively impossible in practice. For these reasons, performing L7 data processing at the XDP level is not recommended. #### Socket Filter - Application-level interception Moving up the stack, socket filters run after the kernel networking stack has processed packets, so they see complete packet data and are a much better fit for HTTP parsing and header modification. We can distinguish four relevant eBPF socket attach types: - **Socket filter:** the original Linux Socket Filter (derived from BSD BPF). A filter is attached to a specific socket via `setsockopt`. It runs on packets delivered to that socket and decides how many bytes to accept or drop before data reaches the socket’s receive queue. Think of it like an in-kernel tcpdump-style filter. - **SK_SKB** (socket buffer programs on SOCKMAP/SOCKHASH): They are sit on TCP data streams to parse application level(L7) messages and with that make decision wheter it is allowed or blocked or redirected. It works on SOCKMAP/HASH which is an unique map type which holds network sockets as values. SKP programs have different purposes based on their attach type. It could be Parsers or Verdict programs. - **SK_MSG** (send-path verdict on SOCKMAP/SOCKHASH): This is a verdict program for outbound messages on the socket sits inside the SOCKMAP/SOCKHASH. It inspects application data before it leaves the socket and can pass/drop or redirect to another socket via map helpers. Complements SK_SKB for receive-side control. - **SOCK_OPS:** Lifecycle and event callbacks for TCP sockets (e.g., state changes, established, timeouts) typically attached at a cgroup, great for enrolling/removing sockets in SOCKMAP/SOCKHASH and tuning per-connection behavior. It is not suitable for filter purposes. or HTTP parsing and reliable header modification the best eBPF approach is a combination of **SK_SKB** (receive-side parsing) and **SK_MSG** (send-side verdicts/rewrites). Because these hooks run after TCP reassembly, they can operate on higher-level application payloads and redirect to a peer socket that performs full header rewrites or injection. One important limitation: these hooks only see **plaintext** if encryption happens after the hook point. If the application performs TLS entirely in user space, the eBPF program will only see ciphertext and cannot decrypt it. Fortunately, many common user-space TLS libraries support **kTLS** , which moves record protection into the kernel so eBPF hooks placed before the kernel’s record-layer encryption can peek at plaintext. ![socket](../../assets/ebpf-modul/socket.jpg) #### KProbes/UProbes In our [tracing blogposts](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing) we explained how Kprobes work, here we’ll focus on Uprobes because they offer a unique approach for tracing encrypted messages. Uprobes are dynamic, per-process instrumentation points that attach to user-space function entry and exit without modifying binaries. They let you observe function arguments, return values, and relevant memory in the probed process with minimal overhead. This makes Uprobes ideal for parsing application-layer data at functions such as `SSL_write` and `SSL_read`. `SSL_write` sees plaintext application bytes before encryption, and `SSL_read` sees plaintext after decryption, so an attached uprobe can reliably parse HTTP headers, URIs, or custom protocols. Using Uprobes bypasses the limitation of kernel-level socket hooks - which only observe ciphertext when encryption happens in user space — and provides deep observability with precise process attribution. ![uprobe](../../assets/ebpf-modul/uprobe.jpg) Relying on Uprobes lets us parse payloads in a reliable way, but there are caveats. Uprobes typically depend on specific function names and calling conventions, so they work only for particular library versions (e.g., a given OpenSSL release). Other TLS libraries (LibreSSL, BoringSSL, GnuTLS, custom vendored stacks) expose different functions and memory layouts that require different Uprobe locations and handling. Supporting every library and every version is therefore non-trivial and becomes a long-term maintenance effort, but Uprobes remain a powerful additional tool when you can target a known runtime. **Credential exposure via sysfs**: An eBPF program cannot create files in `sysfs` (or anywhere in the filesystem). eBPF runs in a heavily restricted, sandboxed environment with no direct ability to perform arbitrary filesystem operations like creating files, opening paths, or writing to virtual filesystems such as `sysfs`. It can only interact through approved helpers, maps, and event output mechanisms, none of which allow creating `sysfs` nodes. ## Conclusion Riptides’ mission requires tight integration between cryptography, policy enforcement, and workload identity — all within the Linux kernel’s trust boundary. eBPF gives developers safe, dynamic programmability for observability, filtering, and lightweight policy logic, but its sandbox stops at the edge of cryptographic and filesystem operations. In contrast, kernel modules give us: - direct access to the crypto stack and socket lifecycle, - process-bound identity binding, - transparent credential orchestration, - and low-latency, in-kernel TLS activation. In short: eBPF is perfect for watching and shaping packets, but when you need to create, sign, and protect the identities behind them, you need the kernel itself. --- ## Why Riptides Embraces SPIFFE But Not SPIRE - URL: https://blog.riptides.io/why-riptides-embraces-spiffe-but-not-spire - Published: 2025-10-22 - Author: Janos Matyas - Category: Kernel - Tags: vision, spiffe, identity, kernel, linux At Riptides, we believe in standards. **SPIFFE** provides an elegant, open framework for defining workload identity — a lingua franca for secure, verifiable communication between non-human entities. It’s the right abstraction for the modern distributed world, where ephemeral workloads, containers, and functions all need to prove *who they are* before they can speak securely. But while SPIFFE defines the “what,” **SPIRE** — its reference implementation — defines one possible “how.” And that’s where our paths diverge. ## Where SPIRE Stops SPIRE is a valuable project, but it was designed for a very different operational model. It runs in user space, expects applications to participate in certificate handling, and leaves much of enforcement to other layers. Applications must request, load, and renew their certificates. Migration often means updating code or sidecar integration. In SPIRE’s approach: - Certificates live in userspace and are accessible to the application. - Lifecycle management (request, renewal, rotation) requires coordination with the workload. - There’s no inherent network or process-level enforcement, identity is issued, but not actively guarded. - Operational cost is extremely high. This leaves a **gap between identity issuance and enforcement**, which made us uncomfortable. You can *(almost)* know *who* something is — but not control *what* it does in real time. Another concern for us was that with SPIRE, the **operational cost** for teams is extremely high: - **Application-level integration** pushes identity logic into workload code. It fragments implementation, burdens developers, and breaks separation of concerns. - **Sidecars and service meshes** externalize identity but add weight and indirection. They depend on Kubernetes, multiply processes, and weaken identity fidelity — the certificate **belongs to the proxy**, not the process/workload. Kubernetes isn’t everywhere. Proxies aren’t everywhere. **The Linux kernel is.** Every workload touches it — opening sockets, issuing syscalls, launching processes. Read our post [Beyond Sidecars and Proxies: Why the Linux Kernel Is the Future of Non-Human Identity](/blog/rethinking-workload-identity-at-the-kernel-level) for more details. Another post dives into the risks of offloading identity to a proxy instead of the workload itself, [The Hidden Risk in Service Mesh mTLS: When Your Sidecar Becomes a Trojan Horse](/blog/the-hidden-risk-in-service-mesh-mtls-when-your-sidecar-becomes-a-trojan-horse). ![Where SPIRE Stops](../../assets/riptides-love-spiffe/illustration1.png) ## And Riptides Begins Riptides takes SPIFFE’s principles and drives them deeper, [right into the Linux kernel](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). In our model, **the private key never leaves kernel space**. The entire certificate lifecycle — issuance, rotation, revocation — happens [transparently and independently of the application](/blog/rethinking-workload-identity-at-the-kernel-level). **No changes, no SDKs, no sidecars, no user-space leakage**. For workloads, identity just *exists*. Secure communication happens by design, not by orchestration. This transparency is more than convenience; it’s a **security boundary shift**. It eliminates an entire class of attacks targeting userspace keys or misconfigured agents. It also enables something SPIRE was never built to do: **real-time policy enforcement and behavioral attestation** at the point of communication. Even when communicating with third-party systems that don’t speak SPIFFE, the workload’s SPIFFE SVID can be [exchanged to a credential, token, or secret](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) recognized by the external service. This credential is [injected directly on the wire](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example), transparently and securely, within the Linux kernel, so the workload can interact with third-party systems without ever exposing its private key or requiring manual handling. In addition, we also leverage SPIFFE to [federate identities across multiple clouds](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure), enabling secure, consistent workload authentication and authorization across AWS, GCP, Azure, Oracle and hybrid environments, without relying on cloud-native federation alone. ![and Riptides Begins](../../assets/riptides-love-spiffe/illustration2.png) ## Identity Is Only the Start Riptides extends far beyond SPIRE’s identity issuance model. We combine **granular workload attestation**, **policy enforcement**, and **observability** into one continuous trust fabric. - In [*Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE*](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), we delved into how kernel-level TLS integration allows workloads to automatically present their SPIFFE identities for authentication and secure communication, removing the need for application-level/userspace certificate handling and enabling transparent, end-to-end encrypted interactions. - In [*Workload Attestation and Metadata Gathering*](/blog/workload-attestation-and-metadata-gathering-building-trust-from-the-ground-up), we showed how our attestation pipeline collects process-level evidence, from binaries and environment metadata to node posture before a workload earns its identity. - In [*On-the-Wire Credential Injection*](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example), we demonstrated how secrets and credentials can be injected directly on the wire, removing the need for applications to ever handle them. - In [*Why Cloud-Native Federation Isn’t Enough*](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure), we explored how multi-cloud identity trust must go beyond federation into continuous kernel-level verification. - And in our [*Observability and Telemetry series*](https://riptides.io/blog?hashtags_equal=%5B%22telemetry%22%2C%22tracing%22%2C%22ebpf%22%5D), we explored how deep kernel-level telemetry, observability and tracing provide continuous visibility into workload behavior, enabling real-time attestation, policy enforcement, and auditing across both internal and third-party communications. In other words, **Riptides doesn’t just issue identities — we enforce them**. Every packet, every syscall, every credential handshake can be bound to verified identity and policy, in real time. ## The Vision: SPIFFE Realized, Not Just Implemented SPIFFE remains one of the most forward-looking identity standards in security. We use it as our trust foundation. But our mission at Riptides is to make that standard ***operationally invisible*** — to let identity, attestation, and enforcement flow together, **without touching the app, without trusting the user space, and without manual intervention.** Where SPIRE stops at issuance, Riptides continues to enforcement. Where SPIRE runs in user space, Riptides lives in the kernel. Where SPIRE defines identity, Riptides secures communication. That’s why we say: we don’t just implement SPIFFE — we **evolve** it. --- ## Beyond the Limits: Scaling Our Kernel Module Build Pipeline Even Further - URL: https://blog.riptides.io/beyond-the-limits-scaling-our-kernel-module-build-pipeline-even-further - Published: 2025-10-20 - Author: Peter Balogh - Category: Kernel - Tags: kernel, linux, build, automation ## Evolving our build system for multi-distro, multi-kernel automation Riptides issues [SPIFFE-based identities directly from the Linux kernel](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), securing workload communication at the system’s lowest boundary. To make that possible across diverse environments, our kernel modules must build fast and reliably for a wide range of distributions and versions. In our previous blog post, [Building Linux Driver at Scale: Our Automated Multi-Distro, Multi-Arch Build Pipeline](/blog/building-linux-driver-at-scale-our-automated-multi-distro-multi-arch-build-pipeline), we shared how we have built a fully automated pipeline to compile our kernel module across multiple distributions and architectures. As we expanded our supported kernels and distributions, we quickly hit GitHub Actions' scaling limits. This post dives into how we overcame those challenges and pushed our build automation even further. ## The First Challenge: Kernel Headers and Scale Building a kernel module is never "one-size-fits-all." Each distribution and kernel version requires its own `kernel-headers` or `kernel-devel` package. That means the same code must be compiled hundreds of times, and each time inside the matching environment for that kernel. **Over time, our build matrix exploded:** - Multiple distributions (Ubuntu, Fedora, Debian, CentOS, Amazon Linux, etc.) - Multiple versions per distribution - Multiple architectures (x86_64, aarch64) - Multiple kernel versions for each combination **This quickly exceeded GitHub Actions' limits:** - [Maximum 256 matrix jobs per workflow run](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idstrategymatrix) - [Maximum 1 MB output per job](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idoutputs) - [Maximum 50 MB total outputs per workflow run](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idoutputs) For a project that compiles **hundreds of kernel modules**, these limits became a real bottleneck. ### Our First Attempt: One Giant Matrix Initially, we generated a single large build matrix containing every possible combination. This worked well until we crossed 256 jobs, GitHub Actions simply refused to run more. Even when we tried to split by architecture or distribution, outputs from large jobs (such as build metadata and artifact paths) began hitting the 1 MB per-job output limit. ### Breaking the Barrier: Matrix Chunking + Workflow Dispatch To overcome these limitations, we rearchitected our pipeline into two workflows. **1. The "Matrix Generator" Workflow**: - Generates the full build matrix dynamically (based on available kernel versions and distributions) - For incremental builds, diffs the newly generated matrix with the previously used one (downloaded from an S3 bucket) - Chunks the effective matrix into smaller JSON segments (e.g., batches of 50) - Triggers multiple child workflows via `workflow_dispatch`, each with one matrix chunk Each chunk represents a manageable set of builds that stay well within GitHub's job limits. ![Matrix Generator Workflow](../../assets/kmod-build-beyond-limits/generate-build-matrices-workflow.png) **2. The "Matrix Build" Workflow**: The second workflow is invoked by the "Matrix Generator." It reads the matrix chunk, creates the `strategy.matrix` dynamically, and performs the actual kernel module builds. This distributed approach scales virtually without limit. We can trigger up to 256 child workflows, which means the theoretical maximum number of kernel module builds is `256 x 256 = 65536` — far beyond what we currently need. ![Matrix Build Workflow](../../assets/kmod-build-beyond-limits/matrix-build-workflow.png) ## The Second Challenge: Container Registry Rate Limits During large-scale builds, we also encountered rate limits from container registries. Fetching hundreds of base images in parallel is both expensive and slow, especially when each job pulls the same image layers. ### Breaking the Barrier: Custom Container Image Caching Layer - The "Matrix Generator" workflow builds our own base images (for each distro and architecture) if an image hasn't already been built and uploaded to the S3 bucket. - These base images are saved as tarballs and uploaded to the S3 bucket. - The "Matrix Build" workflows then download the tarballs from S3 only once and upload them as GitHub Artifacts. - Every parallel build job restores its image from the local artifact (no S3 traffic, no rate limiting). This drastically reduced build startup time and made the system much more reliable. ![Architecture](../../assets/kmod-build-beyond-limits/blog-og.jpg) ## +1 Challenge: Missing Kernel Versions While building for multiple distributions, we realized that some kernel versions weren't readily available through standard package repositories. Tools like [Falco's kernel-crawler](https://github.com/falcosecurity/kernel-crawler) are great for identifying available kernels, but they don't cover every version we need—especially the latest live-patch kernels from Amazon Linux or Fedora. ### Filling the Gaps: Fedora Koji & Amazon Linux Live Patch Packages To fill these gaps, we started leveraging: - Fedora Koji build system that provides official Fedora kernel packages for versions that aren't published in standard repos. - Amazon Linux live patch packages which is official kernel updates for Amazon Linux 2 and Amazon Linux 2023, including critical security patches. By pulling the kernel headers and modules directly from these sources, we ensure that our builds cover all supported kernel versions, including those missing from Falco's kernel-crawler. This step was essential for maintaining full coverage across distributions and staying up-to-date with security updates. ## Conclusion Scaling kernel module builds across dozens of Linux distributions and kernel versions isn't just about compute power, it's about architecture. By breaking past GitHub Actions' native limitations, we built a flexible, distributed system capable of compiling hundreds of kernel variants in parallel, without overloading our CI platform or container registries. The combination of **matrix chunking**, **workflow chaining**, and **prebuilt base image caching** gave us the control and reliability needed to handle large-scale builds efficiently. This approach now allows our engineering team to iterate faster, validate new kernel releases quickly, and deliver tested drivers for every major Linux environment automatically. **Benefits of the New Design**: | Challenge | Solution | Result | |-------------------------|------------------------------------|-------------------------------------| | 256 job workflow limit | Chunked matrix generation | Unlimited scalability | | Output size limits | Pass JSON chunks | Reliable data exchange | | Registry rate limiting | Cached tarballs + GitHub artifacts | Faster, stable builds | | High startup time | Prebuilt base images | Minimal job initialization overhead | We can now build Riptides kernel modules for hundreds of kernel versions across all major distributions in parallel, using self-hosted runners optimized for kernel compilation. **Key Takeaways**: - Chunk large matrices into smaller workflows to bypass GitHub Actions' 256 job limit. - Use `workflow_dispatch` or `repository_dispatch` events to trigger child workflows dynamically. - Keep workflow outputs minimal to avoid the 1 MB per-job and 50 MB per-run output caps. - Cache and reuse base images (as `.tar` files) to eliminate registry rate-limit issues. - Leverage GitHub Artifacts for fast, local image distribution in parallel jobs. - Design for scale from the start. A small architectural decisions early on prevent major pipeline bottlenecks later. Interested in how our Linux kernel journey unfolded and what we learned along the way? - [Rethinking Workload Identity at the Kernel Level](/blog/rethinking-workload-identity-at-the-kernel-level) - [Riptides: Kernel-Level Identity and Security Reinvented](/blog/riptides-kernel-level-identity-and-security-reinvented) - [From Breakpoints to Tracepoints: An Introduction to Linux Kernel Tracing](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing) - [From Tracepoints to Metrics: A journey from kernel to user-space](/blog/from-tracepoints-to-metrics-a-journey-from-kernel-to-user-space) - [Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) - [Linux kernel module telemetry: beyond the usual suspects](/blog/linux-kernel-module-telemetry-beyond-the-usual-suspects) - [From Tracepoints to Prometheus: the journey of a kernel event to observability](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability) If you enjoyed this post, follow us on [LinkedIn](https://www.linkedin.com/company/riptidesio/) and [X](https://x.com/riptidesio) for more updates. If you'd like to see Riptides in action, [get in touch with us for a demo](https://riptides.io/request-a-demo). --- ## Workload Attestation and Metadata Gathering: Building Trust from the Ground Up - URL: https://blog.riptides.io/workload-attestation-and-metadata-gathering-building-trust-from-the-ground-up - Published: 2025-10-13 - Author: Zsolt Varga - Category: Kernel - Tags: vision, spiffe, identity, zero-trust, kernel, linux Before any workload can speak securely, it must first earn the right to be trusted. At Riptides, we use [SPIFFE-based identities, certificate-based authentication, and TLS](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) to secure every connection directly from the Linux kernel, but those mechanisms depend on a deeper layer of truth. **Workload attestation** is how we verify that a process truly is what it claims to be, before granting it an identity. ## What is Workload Attestation? ### What is a _workload_? At its core, every workload reduces to **a running process** (or a process tree) inside a Linux system. Whether you call it a _container_, _pod_, _function_, or _microservice_, underneath it all is a process executing a binary with a particular environment and set of kernel attributes. However, a **workload** is not _one_ process — it’s a _logical unit of work_ represented by one or more **process instances** that share the same operational and semantic identity. For example: - The same container image running on 10 Kubernetes nodes produces 10 processes, but they _represent one workload_ in the scheduling and identity sense. - Each process instance has a unique runtime footprint (PID, namespace, start time, executable hash, etc.), but they all map to the same **workload class** (same image, same service account, same namespace). > **In short:** > **process = instance** > **workload = class of instances that perform the same logical function** ### What is workload attestation? **Workload attestation** is the process of collecting _verifiable facts_ about a **specific running process instance** and presenting those facts as **evidence** to a policy or issuing component. The attester answers the question: > “Can I prove, using data the process cannot forge, what this process actually is and where it came from?” If that evidence matches expected properties (image digest, service account, node posture, etc.), an issuer can confidently bind the workload instance to a **short-lived credential** (whose subject may later correspond to a SPIFFE ID). For more on how these credentials and certificates are the foundation of trust and fit into our broader identity model, see [X.509 Certificates in the Age of SPIFFE and Zero Trust](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust). ### The core insight Even though orchestration frameworks operate at higher layers (pods, deployments, jobs), the **root of truth is always process-level evidence**, because: - It’s what the kernel enforces (exec, namespaces, capabilities). - It’s where identity is realized (socket ownership, TLS sessions). - It’s the narrowest, most trustworthy observable boundary that can be measured and attested. You can think of attestation as _lifting process-level facts_ into _workload-level context_. ## Design Goals and Constraints ### Why design goals matter Workload attestation sits at the intersection of **systems introspection** and **trust establishment**. It must collect facts from the OS in a way that is **reliable, non-forgeable, and reproducible** across thousands of process instances. The challenge is to balance **fidelity** (accuracy of evidence) and **feasibility** (how safely and efficiently you can collect it). ### Determinism: same workload, same evidence A given workload should produce **consistent evidence** across its instances — assuming they run the same binary, image, and configuration. Even small runtime differences (binary hash, UID, node) must yield **distinct** evidence. This enables: - Predictable identity derivation (same workload → same logical ID). - Simple verification (expected vs. actual digest sets). - Stable trust chains across reproducible deployments. ### Collision resistance: prevent false equivalence Different workloads must not collapse into the same attested identity. Evidence should include entropy from: - the binary hash, - the image digest, - the namespace or service account, - and the node identity. Even if two workloads run the same binary, attestation should preserve their **contextual distinctness**. > Logical isolation must translate into **identity isolation**. ### Minimal privilege, maximal visibility The attester must observe workloads **without becoming an attack surface**. Principles: - Collect only what’s needed (no raw environment, no secrets, sanitize data). - Use kernel-exposed views (procfs, sysfs, cgroup, CRI/kubelet). - Separate collection privileges from policy and issuance. - Operate in _read-only_ mode whenever possible. > In short: **least privilege + read-only + host-level visibility**. ### Portability and abstraction The mechanism must generalize across: - Bare-metal and VM nodes - Container runtimes (Docker, containerd, CRI-O) - Orchestrators (Kubernetes, Nomad, ECS) - Public clouds (AWS, GCP, Azure) Evidence sources may differ, but the **schema** — the shape of the collected metadata — must remain stable. This allows one attestation framework to operate uniformly across a heterogeneous fleet. ### Operator visibility and auditability Operators must be able to **inspect** attestation outcomes: - Which workloads were attested, when, and on which node. - What evidence contributed to each decision. - Why a particular workload failed attestation. Good observability turns attestation from a black box into a transparent part of the platform’s trust fabric. At Riptides, our [kernel-level observability pipeline](https://riptides.io/blog?hashtags_equal=%5B%22telemetry%22%2C%22metrics%22%2C%22ebpf%22%5D) built on eBPF provides the fine-grained process and network insights that make such transparency possible. ## Identifying a Specific Linux Process Instance ### Why this matters At the heart of workload attestation is the ability to say with confidence: > “This evidence corresponds to _this exact process instance_, and not a recycled or spoofed one.” Linux reuses PIDs constantly, workloads `exec()` new binaries in place, and containers mask process hierarchies. Attestation therefore needs **stable, kernel-level identifiers** that remain valid across these transitions. ### Process instance vs. workload class A **workload** represents a logical unit of work that may have many running instances. Each **process instance** is one concrete execution of that workload — a live entity inside a PID namespace with its own start time and binary. Attestation always begins with the **process** because it’s the smallest, verifiable boundary the kernel exposes. Every higher-level concept (container, pod) ultimately maps back to a process. ### Stable, non-forgeable identifiers The goal is to derive a small set of **immutable facts** that uniquely identify one process instance even if its PID is reused later. | Category | Source | Example / Field | Purpose | | ----------------------- | --------------------------------- | ------------------ | --------------------------------------------- | | **Namespace identity** | `/proc//ns/pid` | inode number | Distinguishes processes across PID namespaces | | **Start time** | `/proc//stat` field 22 | jiffies since boot | Differentiates reused PIDs | | **Executable identity** | `/proc//exe` (opened fd) | SHA-256 of ELF | Code identity, stable even if file unlinked | | **User context** | `/proc//status` | uid, gid, name | Links workload to execution context | | **Command shape** | `/proc//cmdline` (sanitized) | argument vector | Optional — helps correlate replicas | ### Additional discriminators (optional) - **Cgroup path** (`/proc//cgroup`) – maps the process to its container or pod context. - **Mount namespace inode** (`/proc//ns/mnt`) – distinguishes isolated filesystem views. - **Capabilities** (`CapEff` in `/proc//status`) – captures runtime privilege set. These can enrich the evidence set. ### Interpreted languages and dynamic runtimes Hashing an executable ELF file works perfectly for statically linked or compiled binaries, but not for **interpreted languages** such as Python, Node.js, Ruby, or Java. In these cases, the ELF you hash is only the **interpreter**, not the code actually being executed. To improve accuracy: - **Include script or bytecode hashes** when the file is accessible on disk. - **Record the interpreter command line** (e.g., `/usr/bin/python3 app.py`) in sanitized form. - **Observe loaded modules or JARs** when feasible; their digests often carry semantic identity. - **Treat the container or image digest** as the stronger code anchor when the runtime dynamically loads code. Interpreted workloads therefore rely more on **environmental evidence** (image digest, orchestrator metadata) than on the ELF hash itself. Attestation systems must recognize this distinction and adjust policy accordingly. ## Collecting Environmental Evidence ### From process to environment A process on its own tells you _what_ is executing, but not _where_ or _under what circumstances_. To transform low-level process identity into a **workload identity candidate**, we must capture **contextual evidence**, the environmental signals that surround the process. This includes: - **Container and image information** (what packaged artifact is running), - **Orchestrator metadata** (what the scheduler declared), - **Node and OS provenance** (where it runs physically or virtually), - and optionally **Cloud instance metadata** (which management domain or project it belongs to). ### Container and runtime evidence Containers are the most common abstraction boundary between the orchestrator and the process. They provide deterministic packaging, but also hide process details. Therefore, an attester must be able to map a process → container → image digest relationship. | Source | Example | Notes | | ----------------------------------------------- | -------------------------------------------- | --------------------------------------------------- | | `/proc//cgroup` | `0::/kubepods.slice/.../docker-abcdef.scope` | Identify container runtime and ID | | Container runtime API (Docker, containerd, CRI) | `container.inspect` | Retrieve container ID, image digest, runtime config | - Record **image digest** (e.g., `sha256:...`), not just mutable tags. - Record **container ID**, but treat it as ephemeral — it’s not globally unique. - Minimize dependency on in-container paths; collect evidence from the host. > **Outcome:** Evidence linking the process instance to its immutable artifact — the container image digest. ### Orchestrator evidence (e.g., Kubernetes) The orchestrator adds intent: why this workload exists and under what identity it should run. | Key | Source | Purpose | | ------------------------ | -------------------- | ------------------------------------- | | `k8s:pod:namespace` | Kubelet / API | Defines administrative domain | | `k8s:pod:serviceaccount` | Pod spec | Maps workload to its logical identity | | `k8s.pod:uid` | Pod metadata | Unique immutable ID per Pod instance | | `k8s:node:name` | Kubelet registration | Links workload to node | Collecting this data requires querying either: - the **kubelet API** or **CRI socket** on the node (preferred for trust), or - the **Kubernetes API** (if securely authenticated). > Attesters should avoid trusting in-container environment variables or downward API files, as those can be manipulated by the workload itself. ### Node and OS provenance The node provides the execution substrate. Establishing node identity ensures that attestation chains tie workloads to trusted infrastructure. | Evidence | Source | Purpose | | -------------- | -------------------------------- | --------------------------------- | | Hostname | `/proc/sys/kernel/hostname` | Node-level label | | Kernel version | `uname -r` | Platform version tracking | | OS version | `/etc/os-release` | Distribution fingerprint | | Hardware UUID | `/sys/class/dmi/id/product_uuid` | Physical or virtual node identity | Combine these facts to verify node fleet membership or hardware trust anchors. ### Cloud instance metadata (optional) When running in public clouds, host-level provenance can be enriched with cloud metadata. These APIs provide verifiable information about the surrounding infrastructure, often cryptographically signed. | Cloud | Endpoint | Notable fields | | --------- | ----------------------------------------------------- | --------------------------------------------- | | **AWS** | `http://169.254.169.254/latest/meta-data/` | `instance-id`, `region`, `ami-id`, `vpc-id` | | **GCP** | `http://metadata.google.internal/computeMetadata/v1/` | `instance/id`, `zone`, `project-id` | | **Azure** | `http://169.254.169.254/metadata/instance` | `vmId`, `subscriptionId`, `resourceGroupName` | Access should be **controlled and authenticated** (IMDSv2 headers, Metadata-Flavor tokens). Evidence from IMDS proves the node’s identity, not the workload’s. ### Bringing it together Each process-level fingerprint expands into a richer **contextual identity graph**: ``` Process ├── Container → Image digest ├── Orchestrator → Namespace, ServiceAccount, Pod ├── Node → UUID, Kernel, OS └── Cloud → Instance ID, Region, Project ``` Together, these layers form a structured representation of both intrinsic and environmental evidence, suitable for signing, evaluation, and identity issuance. ## Dynamic Metadata Collection and Aggregation ### From theory to implementation Collecting attestation evidence sounds straightforward — read a few files from `/proc`, maybe query the container runtime, but in practice, workloads run in wildly diverse environments. An attester must dynamically adapt its collection strategy to the runtime it’s observing while maintaining a consistent output format. This requires an architecture built around **modular metadata collectors**, each specialized for a particular signal source: process, host, orchestrator, cloud, or hardware. ### Collector architecture The attester maintains a **registry of collectors**, each responsible for one logical source of evidence. Each collector implements two key methods: 1. determine if the environment or data source is present. 2. gather relevant metadata key–value pairs from that environment. Collectors run **concurrently** and merge their outputs into a single, flattened metadata document. ### Process-level collector Works directly with the Linux kernel’s process introspection interfaces (`/proc`, `sysfs`) to extract immutable facts: executable hash, cmdline, UID/GID, and capabilities, etc. ### Environment collectors - **Container collector:** maps process → container → image digest via cgroups and runtime sockets. - **Orchestrator collector:** queries kubelet/CRI to attach namespace, service account, and pod related information. - **Node collector:** reads DMI, hostname, OS release, and kernel version for node provenance. - **Cloud collectors:** query metadata endpoints for instance IDs and region info. Each collector adds context only when relevant and detected. ### Aggregation pipeline 1. Enumerate all collectors. 2. Run detection concurrently. 3. Collect evidence from applicable collectors. 4. Merge outputs into a **flattened metadata map** with deterministic precedence. ### Flattened metadata structure ```shell ec2:ami:id=ami-071d634346d669118 ec2:instance:id=i-0f700fba6dcff977b ec2:instance:type=t3.xlarge ec2:placement:region=eu-central-1 ec2:security-groups=eks-remoteAccess-04cbfacd-b64b-993d-1e18-b5bcb500566b ec2:services:domain=amazonaws.com ec2:services:partition=aws eks-cluster-sg-riptides-dev-eks-515310092 k8s:container:name=redis k8s:pod:image:name=redis:alpine@sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952 k8s:pod:init-image:count=0 k8s:pod:name=redis-cart-6f6d4b5589-vmcgn k8s:pod:namespace=riptides-demo k8s:pod:serviceaccount=redis-cart node:hostname=ip-10-0-1-59.eu-central-1.compute.internal node:kernel:arch=x86_64 node:uuid=ec29a4e4-a520-f27d-a783-fbbf46a973d7 process:binary:hash=sha256:2bb8368cb2e83a73c5d2bad949702452987c570627de41d269997f99455473b2 process:binary:path=/usr/local/bin/redis-server process:cmdline=redis-server *:6379 process:gid=1000 process:name=redis-server process:uid=1000 ``` - Every key is globally unique, machine-readable, and human-auditable. - Collectors operate independently; one failure does not stop the pipeline. - Keys are sorted lexicographically, values normalized. - The outcome is deterministic and portable. ## Summary and Conclusion Modern distributed systems depend on workloads that can authenticate securely, but **secure authentication is impossible without trusted attestation**. Before any workload receives an identity, the system must first _prove what it is_ and _where it runs_. That proof comes from **evidence**, not configuration. Across this post, we explored the foundations of workload attestation: 1. **Defining the problem** — A workload is ultimately a process executing within an environment. 2. **Design goals** — Determinism, collision resistance, least privilege, and portability. 3. **Process identity** — Unique fingerprints from immutable process level information like uid/gid, executable hash. 4. **Environmental context** — Container, orchestrator, node, and cloud metadata providing full context. 5. **Dynamic metadata gathering** — Modular collectors combining these layers into a canonical evidence set. Without attestation, every identity system is built on assumption. With it, identity becomes measurable, auditable, and revocable. Workload attestation is not just about security, it’s the **foundation of verifiable trust**. Every certificate, every SPIFFE ID, every cryptographic handshake ultimately depends on this ground truth. --- ## Zero Trust: From Perimeter to Kernel — How Riptides Pushes the Boundary - URL: https://blog.riptides.io/zero-trust-from-perimeter-to-kernel---how-riptides-pushes-the-boundary - Published: 2025-10-09 - Author: Janos Matyas - Category: Vision - Tags: vision, spiffe, identity, zero-trust, kernel, linux In the evolving threat landscape, “zero trust” is hardly a new phrase — but it is more important than ever. As attackers become more sophisticated, identity-based, lateral movement, and subtle persistence-based threats are no longer edge cases; they’re part of the assumed baseline. This post will walk through the general principles of zero trust (drawing on sources like John Kindervag’s original model, Google’s BeyondCorp/BeyondProd, Cloudflare’s implementation), then introduce how [Riptides takes zero trust further](/blog/rethinking-workload-identity-at-the-kernel-level), from the perimeter, to the node, to the kernel and why that matters for modern infrastructure security. **TL;DR:** We achieve this **kernel-level zero trust** using [strong attestation, SPIFFE for strong workload identity and kTLS](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) for secure, identity-bound communication. ## The General Principles of Zero Trust Let’s begin with what zero trust *means in practice*, drawing from the original “never trust, always verify” model and how major players have implemented it. ### Origins: Kindervag & the “Perimeter is Dead” - **John Kindervag** (Forrester analyst) popularized zero trust ~15 years ago, criticizing traditional perimeter-based defenses that implicitly trust everything inside a firewall. The idea: just because traffic is “inside” doesn’t mean it should be trusted. - The core mantra: *Assume breach.* Trust nothing by default. Reduce the blast radius of any intrusion. Enforce strong segmentation, least privilege, continuous verification. ### Key Concepts (as distilled over time) Established guidance from Kindervag, NIST, and Google shows that robust zero trust architectures typically include: 1. **Strong Identity & Authentication** Every user, every device, every service must prove who they are. Credentials, device posture, attestation, identity signals: these are continuously evaluated. 2. **Least Privilege Access / Principle of Minimal Privilege** Access granted only to what is needed, for as long as it is needed. Permissions are narrowly scoped, ephemeral where possible. 3. **Micro-Segmentation** Even within the data center or network, break up the environment so that lateral movement is constrained. Each boundary (service-to-service, process to process) is guarded. 4. **Continuous Verification / Context Awareness** Trust decisions are not static. They consider context: device health, recent behavior, time, location, the system’s state. 5. **Monitoring, Auditing, Feedback, and Automation** You need visibility into what is going on (flows, anomalies), to enforce policy, and to adjust dynamically. 6. **No Implicit or Perimeter Trust** Simply being “inside” a network doesn’t confer trust. All traffic (east-west, north-south), all processes, all workloads must be verified. ## Perspectives: Google BeyondCorp / BeyondProd & Cloudflare ### Google’s BeyondCorp & BeyondProd - **BeyondCorp** is Google’s internal zero trust model for user access: decoupling trust from network location. Whether you’re in the office or remote, access is granted based on identity, device posture, and context. - **BeyondProd** extends those ideas into production / services: no service is assumed to trust any other by default; workloads run on trusted machines, code provenance is verified; internal services enforce mutual distrust unless explicitly authorized. ### Cloudflare’s Zero Trust & “Access” - Cloudflare has built zero trust features into its access product: e.g., **Cloudflare Access** allows enforcing identity / device posture / location / additional signals before letting users reach applications. - They also apply zero trust to SaaS applications (not just internal apps) by aggregating identity signals, enforcing rules based on multiple criteria, generating JWTs, etc. - Their model emphasizes shifting checks away from assuming trust at the network boundary and instead making access decisions earlier, with many signals. ## The Riptides Narrative: Zero Trust at the Kernel ![The Riptides Narrative: Zero Trust at the Kernel](../../assets/zero-trust-at-riptides/poster.jpg) ### From Perimeter → Node → Kernel Many zero trust implementations focus on either: - securing the **perimeter** (user access, VPN replacement, access proxies) - or focusing on the **node / workload** (services talking to other services, microservice authentication, service mesh levels) At Riptides, we believe zero trust should start even deeper: at the **kernel**. - We adopt a posture of *default distrust*: nothing in userspace is trusted by default. Every workload, every process, every piece of code that runs in userspace must undergo **attestation**. - We leverage **SPIFFE** (Secure Production Identity Framework for Everyone) for workload identity, issuing cryptographic identities (SVIDs) to workloads only after attestation. If a process doesn’t have a trusted identity (SPIFFE id), it’s not implicitly trusted. So: for Riptides, it’s not enough that “access came from a trusted host” — the code itself, its provenance, its execution context, its behavior must be verified, attested, and assigned identity. That identity becomes the basis for further trust decisions. ### Getting Trust Once Identity is Established Once a workload/process has passed attestation and obtained a SPIFFE identity (an SVID), that becomes the key to building more complex trust relationships: - **Within a data center**: workloads can verify each other via SPIFFE mutual TLS (mTLS), with fine-grained policies based on SPIFFE IDs. - **Across clouds**: we support **workload identity federation** and **on-demand credential injection** to enable trust among workloads running in heterogeneous environments. - Trust expansion is governed by policy: e.g. “this SPIFFE identity (workload/service/process A) can talk to workload/service/process B under certain conditions”, always under least privilege, and policy based. To dive deeper into how we handle **workload identity federation** and **on-demand credential injection**, take a look at these posts: - [Federating non-human identities with external IdPs using ID tokens in AWS, GCP, and Azure](/blog/federating-non-human-identities-with-external-idps-using-id-tokens-in-aws-gcp-and-azure) - [Why Cloud-Native Federation Isn’t Enough for Non-Human Identities in AWS, GCP, and Azure](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure) - [Introducing libsigv4: AWS SigV4 Signatures in Portable C with Kernel Compatibility](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility) - [On-the-Wire Credential Injection: Secretless AWS Bedrock Access example](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) - [On demand credentials - Secretless AI assistant example on GCP](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp) - [Workload Identity Without Secrets: a Blueprint for the Post-Credential Era](/blog/workload-identity-without-secrets-a-blueprint-for-the-post-credential-era) - [Introducing tokenex: an open source Go library for fetching and refreshing cloud credentials](/blog/introducing-tokenex-an-open-source-go-library-for-fetching-and-refreshing-cloud-credentials) ## Why Starting in the Kernel Matters - **Reduces “implicit trust” risks**: Even trusted nodes / hosts can be compromised. Attackers may use privilege escalation or code injection. By enforcing that userspace code itself must be verified, you shrink the attack surface. - **Better granularity**: Trust decisions are made not just at the host boundary, but per process, per workload. Lateral movement and privilege escalation are harder. - **Uniform identity basis**: Using a standard like SPIFFE means consistency: policies are uniform, identities portable, credential issuance / revocation more manageable. - **Cross-environment and cross-cloud coherence**: As organizations use hybrid/multi-cloud, having a kernel-based identity/attestation basis helps ensure that trust is meaningful across environments. Interested in how our Linux kernel journey unfolded and what we learned along the way? - [Rethinking Workload Identity at the Kernel Level](/blog/rethinking-workload-identity-at-the-kernel-level) - [Riptides: Kernel-Level Identity and Security Reinvented](/blog/riptides-kernel-level-identity-and-security-reinvented) - [From Breakpoints to Tracepoints: An Introduction to Linux Kernel Tracing](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing) - [From Tracepoints to Metrics: A journey from kernel to user-space](/blog/from-tracepoints-to-metrics-a-journey-from-kernel-to-user-space) - [Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) - [Linux kernel module telemetry: beyond the usual suspects](/blog/linux-kernel-module-telemetry-beyond-the-usual-suspects) - [From Tracepoints to Prometheus: the journey of a kernel event to observability](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability) ## How Riptides Realizes This in Practice - **SPIFFE + Attestation** Every process/workload must undergo attestation (code origin, execution environment, etc) before being issued an SVID in kernelspace. No unverified userspace process is trusted. - **Mutual TLS & Policy Based on SPIFFE IDs** Once workloads have identities, communications between them are authenticated via mTLS using those identities. Policy engines decide what identities can communicate or access which services. - **Workload Identity Federation** For workloads outside our control, we support identity federation so they can present acceptable identities to integrate with our system. This allows cross-cloud trust without resorting to static credentials or VPNs. - **On-Demand Credential Injection** Instead of embedding long-lived secrets in workloads, credentials are injected in kernelspace only when needed, tied to identity, and revoked when no longer valid. - **Continuous Monitoring and Policy Enforcement** Even after identity issuance, processes and workloads are monitored and re-attested. If a process becomes non-compliant, its trust is revoked or restricted. ## Putting It All Together: The Riptides Zero Trust Stack (Kernel-First) | Layer | What It Covers | Trust / Identity / Controls | | --- | --- | --- | | Kernel / Process Attestation | Code origin, integrity, execution environment | SPIFFE IDs issued only after attestation; no userspace code is trusted by default | | Inter-process / Service Communication | Workload → workload, on same host or across hosts | mTLS with SPIFFE IDs, policy enforcement per identity | | Node / Host | The host’s posture, environment, security properties | Host attestation, secure boot, minimal trusted host invariants | | Data Center / Cloud | Networking, segmentation, environment boundaries | Trust federated via identity, policies enforce cross-workload access | | Multi-Cloud / Hybrid / Cross-Partner | Heterogeneous environments, partners, external workloads | Identity federation, on-demand credentials, consistent identity model across boundaries | ## Conclusion **Zero trust isn’t a product you buy**; it’s a mindset, a set of architectural commitments. From Kindervag’s early formulation through Google’s BeyondCorp/BeyondProd, Cloudflare’s identity-based enforcement, we’ve seen how zero trust can dramatically reduce risk, but also how most implementations still assume some level of implicit trust and/or static credentials. At Riptides, we push the boundary further: **kernel first:**trust nothing in userspace without attestation; only then issue identity; only then allow communication, based on fine-grained policy; extending trust across clouds via federation and credential injection. This gives us: stronger guarantees, smaller blast radius, and more defensible, resilient infrastructure. If you’re building systems today, especially cloud-native, distributed, hybrid or multi-cloud, it’s no longer sufficient to trust by node or network zone. The kernel is where the implicit trust still lingers — remove it, and you’re in a much stronger security posture. --- ## From Kernel WASM to User-Space Policy Evaluation: Lessons Learned at Riptides - URL: https://blog.riptides.io/from-kernel-wasm-to-user-space-policy-evaluation-lessons-learned-at-riptides - Published: 2025-10-06 - Author: Nandor Kracser - Category: Kernel - Tags: wasm, kernel, OPA, linux, identity ## Introduction At Riptides, we're building a platform that provides seamless [kernel-based non-human identity (NHI) with SPIFFE and kTLS](/blog/rethinking-workload-identity-at-the-kernel-level), delivering deep socket-level security and real-time policy enforcement [inside the Linux kernel](/blog/rethinking-workload-identity-at-the-kernel-level). One of our core challenges has been determining the optimal architecture for policy evaluation - specifically, where and how to run **Open Policy Agent (OPA) policies** that govern socket connections in real-time. This is the story of our journey from an ambitious **kernel-space WASM implementation** to a pragmatic user-space solution, and the hard-earned lessons we learned along the way. ## The Initial Vision: WASM in the Kernel When we first architected the Riptides platform, we had a bold vision: evaluate OPA policies directly in kernel space using a WebAssembly runtime. We thought this could work as an eBPF alternative of sorts, providing a more flexible and powerful way to run policy logic in kernel space. The reasoning seemed sound: - **Ultra-low latency**: Policy evaluation would happen at the kernel level without context switches - **Isolation**: WASM provides sandboxing for untrusted policy code - **Flexibility**: Policies could be updated without kernel module changes - **Performance**: No serialization overhead between kernel and userspace - **eBPF alternative**: More expressive than eBPF for complex policy logic while maintaining kernel-space execution To make this vision reality, we forked the [wasm3 WebAssembly runtime](https://github.com/wasm3/wasm3) and began the challenging work of porting it to run in Linux kernel space. For those interested in the technical details, our kernel port is available at [github.com/riptideslabs/wasm3-kernel](https://github.com/riptideslabs/wasm3-kernel). ## The Technical Journey ### Porting wasm3 to Kernel Space The wasm3 runtime was an attractive choice for kernel porting because: - It's designed to be lightweight and embeddable - Uses an interpreter rather than JIT compilation (safer and simpler in kernel context) - Has minimal external dependencies - Relatively small codebase However, porting any userspace runtime to kernel space presents significant challenges: ```c // Example of kernel-space WASM memory management static void* wasm_kernel_malloc(size_t size) { return kmalloc(size, GFP_KERNEL); } static void wasm_kernel_free(void* ptr) { kfree(ptr); } // Replaced standard library functions with kernel equivalents #define malloc(size) wasm_kernel_malloc(size) #define free(ptr) wasm_kernel_free(ptr) ``` We had to: - Replace all standard library calls with kernel equivalents - Implement custom memory management for WASM linear memory - Handle stack overflow protection in kernel context - Ensure atomic operations were kernel-safe - Remove floating-point operations (not allowed in some kernel contexts) The floating-point removal was particularly challenging. We had to modify wasm3 to handle the absence of floating-point operations, but many OPA-compiled WASM modules contained floating-point operations for numeric computations. While OPA's WASM compilation generally produces efficient code, certain Rego built-in functions like mathematical operations, JSON number parsing, or comparison functions could generate floating-point instructions. This incompatibility between our floating-point-free kernel environment and the OPA WASM compilation output became a significant source of policy evaluation failures, especially for policies that performed numeric operations or worked with JSON data containing decimal numbers. ### Integrating OPA Policies Once we had wasm3 running in the kernel, we integrated it with OPA policy evaluation. OPA provides the ability to compile Rego policies into WASM modules using `opa build -t wasm`, which creates standalone WASM executables that can be loaded into any WASM runtime: ```c // Simplified policy evaluation in kernel module static int evaluate_socket_policy(riptides_socket *s, const char *policy_wasm_data, size_t wasm_size) { wasm3_runtime *runtime = get_wasm_runtime(); // Load OPA-compiled policy WASM module wasm3_module *policy_module = wasm3_load_module(runtime, policy_wasm_data, wasm_size); // Prepare socket context as JSON input char input_json[512]; snprintf(input_json, sizeof(input_json), "{" "\"source_ip\":\"%s\"," "\"destination_ip\":\"%s\"," "\"destination_port\":%d," "\"process_name\":\"%s\"" "}", s->src_addr, s->dst_addr, s->dst_port, current->comm); // Allocate memory in WASM module and copy input uint32_t input_ptr = wasm3_call_function(policy_module, "opa_malloc", strlen(input_json) + 1); memcpy(wasm3_get_memory(runtime, input_ptr), input_json, strlen(input_json) + 1); // Parse JSON input in WASM module uint32_t parsed_input = wasm3_call_function(policy_module, "opa_json_parse", input_ptr, strlen(input_json)); // Create evaluation context and set input uint32_t ctx = wasm3_call_function(policy_module, "opa_eval_ctx_new"); wasm3_call_function(policy_module, "opa_eval_ctx_set_input", ctx, parsed_input); // Evaluate policy wasm3_call_function(policy_module, "eval", ctx); // Get result uint32_t result_addr = wasm3_call_function(policy_module, "opa_eval_ctx_get_result", ctx); uint32_t result_json = wasm3_call_function(policy_module, "opa_json_dump", result_addr); // Parse result - OPA returns [{"result": true/false}] for allow/deny policies const char *result_str = wasm3_get_memory(runtime, result_json); int decision = (strstr(result_str, "\"result\":true") != NULL) ? POLICY_ALLOW : POLICY_DENY; // Cleanup wasm3_call_function(policy_module, "opa_free", input_ptr); return decision; } ``` ### Early Success Initially, the system worked beautifully! The OPA WASM compilation workflow was straightforward: ```bash # Compile Rego policy to WASM opa build -t wasm -e riptides/socket/allow policy.rego # Load the resulting bundle into our kernel module echo "policy.tar.gz" > /dev/riptides_policy ``` We could: - Compile standard Rego policies to WASM using `opa build -t wasm` - Load the compiled WASM modules into the kernel via our device driver - Evaluate policies for socket connections with microsecond latency - Update policies by simply recompiling and reloading WASM modules - Maintain strong isolation between policy code and kernel code The performance numbers were impressive - policy evaluation took less than 10 microseconds in most cases, and the WASM modules were compact (typically 50-200KB for complex policies). ![outbound](../../assets/wasm-to-opa/imageinblog.jpg) ## The Reality Check: When WASM in Kernel Goes Wrong However, as we moved from proof-of-concept to production-ready code, we encountered a series of challenges that ultimately led us to reconsider our approach. ### Memory Management Nightmares WASM linear memory management in kernel space proved problematic: ```c // This became a source of kernel panics static int wasm_grow_memory(wasm3_runtime *runtime, uint32_t pages) { size_t new_size = pages * WASM_PAGE_SIZE; // vmalloc for large allocations, but this can fail under memory pressure void *new_memory = vmalloc(new_size); if (!new_memory) { return -ENOMEM; // This could panic the kernel } // Copy existing memory... potential for corruption here memcpy(new_memory, runtime->memory, runtime->memory_size); vfree(runtime->memory); runtime->memory = new_memory; runtime->memory_size = new_size; return 0; } ``` Issues we encountered: - **Memory fragmentation**: Large WASM linear memory allocations could fragment kernel memory - **OOM conditions**: Policy evaluation could trigger out-of-memory conditions that panic the kernel - **Memory leaks**: Complex WASM module lifecycle management led to subtle memory leaks - **Stack overflow**: Recursive policy evaluation could overflow kernel stacks ### Debugging Complexity Debugging WASM execution in kernel space was extremely challenging: - No standard debugging tools worked - Kernel panics provided minimal context about WASM execution state - Policy bugs could crash the entire system - Difficult to distinguish between wasm3 bugs, our porting bugs, and policy bugs ### Security Concerns While WASM provides isolation, running it in kernel space introduced new attack vectors: - Bugs in the WASM runtime could compromise kernel security - Malicious policies could potentially exploit kernel interfaces - The attack surface of the kernel was significantly increased - Fuzzing and security testing became much more complex ### Maintenance Burden Keeping our wasm3 fork in sync with upstream while maintaining kernel compatibility proved unsustainable: - Every wasm3 update required careful porting work - Kernel API changes affected our runtime integration - Supporting multiple kernel versions became a nightmare - The codebase became increasingly divergent from upstream wasm3 - Upstream wasm3 became basically unmaintained during the time ## The Pivot: Moving to User-Space After months of fighting these issues, we made the difficult decision to move policy evaluation out of the kernel and into our Go-based agent process. ### New Architecture The new architecture looks like this: ![outbound](../../assets/wasm-to-opa/imageinblog-1.jpg) ### Protocol Buffer Communication We use protocol buffers with nanopb for efficient kernel-userspace communication. The idea of using protobuf for kernel-userspace communication isn't entirely new - NetBSD explored this concept and demonstrated its feasibility in [their 2015 EuroBSDCon presentation](https://www.netbsd.org/gallery/presentations/riastradh/eurobsdcon2015/protobuf.pdf), showing that structured serialization protocols can work well in kernel contexts. ```protobuf // policy_request.proto message PolicyEvaluationRequest { string source_ip = 1; string destination_ip = 2; uint32 destination_port = 3; string process_name = 4; string hostname = 5; map labels = 6; } message PolicyEvaluationResponse { enum Decision { DENY = 0; ALLOW = 1; } Decision decision = 1; string reason = 2; uint32 cache_ttl = 3; } ``` ### Kernel-Side Implementation ```c // Kernel module policy evaluation static int evaluate_connection_policy(riptides_socket *s) { // Check cache first cached_decision *cached = lookup_policy_cache(s); if (cached && !cache_expired(cached)) { return cached->decision; } // Prepare protobuf request PolicyEvaluationRequest request = PolicyEvaluationRequest_init_zero; encode_socket_context(s, &request); // Send to user-space agent via device file PolicyEvaluationResponse response; int ret = send_policy_request(&request, &response); if (ret == 0) { // Cache the result cache_policy_decision(s, &response); return response.decision; } // Default to deny on communication failure return POLICY_DENY; } ``` ### User-Space Agent Implementation ```go // Go agent policy evaluation func (a *Agent) EvaluatePolicyRequest(req *PolicyEvaluationRequest) *PolicyEvaluationResponse { // Prepare OPA input input := map[string]interface{}{ "source_ip": req.SourceIp, "destination_ip": req.DestinationIp, "destination_port": req.DestinationPort, "process_name": req.ProcessName, "hostname": req.Hostname, "labels": req.Labels, } // Evaluate against OPA policies results, err := a.opa.Query(context.Background(), rego.Query{ Query: "data.riptides.socket.allow", Input: input, }) if err != nil { log.Errorf("Policy evaluation failed: %v", err) return &PolicyEvaluationResponse{ Decision: PolicyEvaluationResponse_DENY, Reason: "evaluation_error", } } // Process results if len(results) > 0 && results[0].Expressions[0].Value == true { return &PolicyEvaluationResponse{ Decision: PolicyEvaluationResponse_ALLOW, CacheTtl: 300, // 5 minutes } } return &PolicyEvaluationResponse{ Decision: PolicyEvaluationResponse_DENY, Reason: "policy_denied", } } ``` ## The Benefits of the New Architecture ### Reliability and Stability Moving policy evaluation to user-space immediately improved system stability: - **No more kernel panics**: Policy bugs can't crash the kernel - **Better error handling**: Graceful degradation when policies fail - **Easier debugging**: Standard Go debugging tools work perfectly - **Improved testing**: Unit tests, integration tests, and fuzzing all become straightforward ### Maintainability The new architecture is much more maintainable: - **Standard OPA**: No custom WASM runtime to maintain - **Pure Go**: Leverages Go's excellent ecosystem and tooling - **Simpler deployment**: Policy updates don't require kernel module changes - **Better observability**: Rich metrics and logging capabilities - **Reduced complexity**: Previously, we needed to augment processes in kernel space, which required additional message passing to user-space for policy evaluation. Now all process context gathering and policy evaluation happens in the agent, eliminating this complexity **Performance Characteristics:** While we lost the ultra-low latency of kernel-space evaluation, the performance is still excellent: - **Sub-millisecond evaluation**: Most policies evaluate in 200-500 microseconds - **Efficient caching**: Results are cached in kernel space for subsequent connections - **Batching optimization**: Multiple policy requests can be batched together - **Async evaluation**: Non-blocking policy evaluation for better throughput ### Security Improvements The security posture actually improved: - **Reduced attack surface**: Kernel module is much simpler and focused - **Principle of least privilege**: Policy evaluation runs in user-space with limited privileges - **Better isolation**: Policies are isolated from kernel and from each other - **Easier auditing**: Policy changes are easier to review and audit **Performance Characteristics:** The performance comparison revealed interesting insights: **Kernel WASM (wasm3 interpreter):** - Lower latency due to no context switching - But interpreted execution was slower than expected - Memory overhead from WASM linear memory allocation - Complex garbage collection in kernel space **User-space OPA (compiled Go binary):** - Higher latency due to kernel-userspace communication - But much faster policy execution due to compiled code - More efficient memory usage - Better CPU utilization Surprisingly, the compiled OPA evaluation in Go was significantly faster than the interpreted WASM execution, even accounting for the overhead of kernel-userspace communication. The user-space approach provides more than adequate performance for real-world workloads, especially with effective caching strategies. ## Lessons Learned ### 1. Complexity Has a Cost Running WASM in kernel space was technically impressive, but the complexity cost was enormous. The maintenance burden, debugging difficulty, and stability issues far outweighed the performance benefits. ### 2. The Kernel Should Stay Simple Kernels should focus on what they do best: resource management, scheduling, and hardware abstraction. Complex business logic like policy evaluation is better suited for user-space. ### 3. Performance Isn't Everything While the kernel WASM approach was faster, the user-space approach provides better overall system characteristics: reliability, maintainability, debuggability, and security. ### 4. Caching Changes Everything Effective caching in the kernel module means that the higher latency of user-space evaluation only affects the first request. Subsequent requests to the same destination are served from cache with microsecond latency. ### 5. Protobuf + nanopb Works Great The combination of protobuf for schema definition and nanopb for efficient kernel-space encoding/decoding provides an excellent balance of performance and maintainability. ## Living with the New Architecture Six months into production with our user-space architecture, we're genuinely happy with the decision. The system feels solid in ways that our kernel WASM implementation never quite achieved. From an operational perspective, policy deployments have become trivial. When we need to update a policy, we simply push new Rego files to our policy repository, and OPA picks them up automatically. No kernel module recompilation, no system restarts, no careful coordination between kernel and user-space components. Our security team can iterate on policies independently, testing them in development environments before rolling them out to production. The observability story has also dramatically improved. We now have rich metrics showing policy evaluation times, decision distributions, and error rates. When a policy behaves unexpectedly, we can trace through the decision logic with standard debugging tools rather than trying to decipher kernel logs. Our compliance team particularly appreciates the detailed audit trails - every policy decision is logged with full context, making security reviews straightforward. Perhaps most importantly, the system degrades gracefully under load or when things go wrong. If our agent process crashes or becomes unresponsive, the kernel module falls back to cached decisions or configurable default behaviors. With the WASM approach, a runtime error could potentially take down the entire kernel module. The development velocity gains have been substantial as well. New team members can contribute to policy logic using familiar Go tooling and testing frameworks. We can run comprehensive test suites against our policy logic, something that was nearly impossible with the kernel WASM approach. Policy authors and kernel developers can work independently, which has eliminated many coordination bottlenecks. Looking ahead, we're exploring some interesting optimizations enabled by the user-space approach. We're experimenting with machine learning models to predict policy decisions before they're requested, pre-warming caches for better performance. We're also investigating batch policy evaluation for workloads with predictable connection patterns. These kinds of sophisticated optimizations would have been extremely difficult to implement safely in kernel space. ## Conclusion The journey from kernel-space WASM to user-space OPA evaluation taught us that impressive technical achievements don't always make the best engineering decisions. While running WASM in the kernel was a fascinating technical challenge, the user-space approach provides better overall system characteristics for a production security platform. Key takeaways for anyone considering similar architectural decisions: 1. **Measure total cost of ownership**, not just raw performance 2. **Prioritize reliability and maintainability** over theoretical performance gains 3. **Use the right tool for the job** - kernels for kernel tasks, user-space for complex logic 4. **Design for operations** - consider debugging, monitoring, and updates from day one 5. **Caching can bridge performance gaps** between different architectural approaches The Riptides platform is now more reliable, maintainable, and secure than ever before. Sometimes the best technical decision is to choose the boring, well-understood solution over the exciting, cutting-edge one. Interested in how our Linux kernel journey unfolded and what we learned along the way? - [Rethinking Workload Identity at the Kernel Level](/blog/rethinking-workload-identity-at-the-kernel-level) - [Riptides: Kernel-Level Identity and Security Reinvented](/blog/riptides-kernel-level-identity-and-security-reinvented) - [From Breakpoints to Tracepoints: An Introduction to Linux Kernel Tracing](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing) - [From Tracepoints to Metrics: A journey from kernel to user-space](/blog/from-tracepoints-to-metrics-a-journey-from-kernel-to-user-space) - [Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) - [Linux kernel module telemetry: beyond the usual suspects](/blog/linux-kernel-module-telemetry-beyond-the-usual-suspects) - [From Tracepoints to Prometheus: the journey of a kernel event to observability](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability) --- ## Introducing tokenex: an open source Go library for fetching and refreshing credentials - URL: https://blog.riptides.io/introducing-tokenex-an-open-source-go-library-for-fetching-and-refreshing-cloud-credentials - Published: 2025-09-29 - Author: Sebastian Toader - Category: Credentials - Tags: federation, credentials ## Introducing [tokenex](https://github.com/riptideslabs/tokenex): an open source Go library for fetching and refreshing cloud credentials ## Why we built **tokenex** In modern cloud systems, **long-lived secrets are a liability**. They sprawl across environments, get baked into config files, and eventually leak. The industry’s response has been **short-lived, federated credentials** as AWS session tokens, GCP and Azure access tokens, OCI UPSTs—that reduce exposure and eliminate the need to persist secrets inside workloads. Even with provider SDKs offering automatic refresh, workloads spanning multiple clouds face a hidden challenge: each provider enforces its own configuration and credential exchange flow. Managing these in isolation quickly becomes cumbersome. This is especially true for **non-human identities**, where workloads must authenticate seamlessly across providers. The goal is clear, **secretless, automated access**: workloads should acquire short-lived credentials on demand, refresh them transparently, and never rely on long-lived secrets. As we’ve explored in earlier posts ([Why cloud-native federation isn’t enough](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure), and [Workload identity without secrets](/blog/workload-identity-without-secrets-a-blueprint-for-the-post-credential-era), the challenge isn’t just about **obtaining credentials once**, it’s about securely acquiring, refreshing, and distributing them automatically across providers in a world that’s rapidly shifting to the **post-credential era**.” Our broader architecture needed a **common building block** that could: 1. Take an identity token from an Identity Provider, 2. Exchange it for the appropriate credential in each cloud/provider, and 3. Continuously refresh that credential so workloads always have a valid one on hand. No such abstraction existed in the ecosystem. So we created **[tokenex](https://github.com/riptideslabs/tokenex)**: a Go library that abstracts away the messy details of token exchange and refresh behind a **single, consistent API**. With [tokenex](https://github.com/riptideslabs/tokenex), services can stay focused on their core functionality, without ever touching long-lived secrets or managing credential lifecycles themselves. ## What the tokenex library does - **Purpose:** [tokenex](https://github.com/riptideslabs/tokenex) is a modular Go library for fetching and refreshing cloud credentials and tokens. It abstracts credential acquisition, refresh, and configuration for AWS, GCP, Azure, OCI, OAuth2, and generic tokens from identity token providers. ## Features - **AWS:** Exchanges ID tokens for AWS temporary session credentials using AWS Workload Identity Federation. - **GCP:** Exchanges ID tokens for GCP access tokens using GCP Workload Identity Federation. - **Azure:** Exchanges ID tokens for Azure access tokens using Microsoft Entra ID Workload Identity Federation. - **OCI:** Exchanges ID tokens for OCI User Principal Session Tokens (UPST) using OCI Workload Identity Federation. - **Generic:** Returns the token provided by the identity token provider and refreshes it before expiration. - **K8sSecret:** Watches a Kubernetes secret that contains a token and publishes updates when the secret changes. - **OAuth2AC:** Obtains access tokens through the OAuth2 authorization code flow and refreshes them before expiration. - **OAuth2CC:** Obtains access tokens through the OAuth2 client credentials flow and refreshes them before expiration. ## Benefits - **Consistency:** Offers a consistent API for credential management, regardless of provider. - **Extensibility:** New providers or token types can be added with minimal friction by following the established provider pattern. - **Configurability:** Uses the Go "option" pattern, allowing users to configure providers with composable options (e.g., `WithClientID`, `WithScope`, etc.). - **Asynchronous & reactive:** Credentials are delivered via channels, supporting reactive and non-blocking workflows. ## How it works All credential providers in this library follow a consistent pattern for credential delivery: 1. The `GetCredentials` method returns a channel that receives credential updates. 2. For the first credential and each refresh, an `Update` event is sent. 3. If credentials are removed, a `Remove` event is sent. 4. In case of errors, the `Err` field is populated, `Credential` is nil, and the refresh loop exits. 5. When the refresh loop exits, the channel is closed. This design ensures that credentials are always up‑to‑date and that applications can handle refreshes or errors reactively. ### Graceful Shutdown For proper application shutdown, always: 1. Cancel the context when your application is terminating. 2. Wait for all credential handling goroutines to complete using a wait group. 3. Handle channel closure and context cancellation in your credential processing loops. This ensures that all resources are properly cleaned up and prevents goroutine leaks. ## Configurability - **Option pattern:** Each provider exposes a set of WithX functions (e.g., `WithClientID`, `WithIdentityTokenProvider`) to configure credentials at construction time. - **Provider construction:** Providers are created with `NewCredentialsProvider(...)`, accepting context, logger, and provider-specific configs. - **Custom token providers:** The token package allows injecting custom `IdentityTokenProvider` implementations. ## Extensibility - **Adding providers:** To add a new provider, implement a new subpackage with a struct that implements the `CredentialsProvider` interface. - **Custom options:** New options can be added by extending the option pattern in each provider. ## Summary table | Feature | Description | |-----------------|------------------------------------------------------------------------| | Providers | AWS, GCP, Azure, OCI, OAuth2 (AC/CC), Generic | | Config pattern | Go option pattern (`WithX` functions) | | Async support | Credentials delivered via channels | | Extensible | Via new subpackages and option pattern | | Use cases | Multi-cloud credential management, token exchange, secure service auth | ## Getting started This is a simple application that demonstrates how to use [tokenex](https://github.com/riptideslabs/tokenex) to obtain temporary credentials, called a **User Principal Session Token (UPST)**, from OCI. The UPST is then used to authenticate and list users in a tenancy via the OCI Go SDK. > This blog post uses **OCI** as an example. If you’re looking for **AWS, GCP, Azure, Kubernetes**, or **generic secret provider** integrations, check out the [**tokenex**](https://github.com/riptideslabs/tokenex) repository on GitHub. The application consists of two goroutines: 1. **UPST retrieval**: Responsible for exchanging an ID token (issued by an external IDP) for a UPST using [tokenex](https://github.com/riptideslabs/tokenex). 2. **Workload simulation**: A simple workload that uses the UPST to authenticate with OCI and list tenancy users. ### How UPST retrieval works The first goroutine leverages [tokenex](https://github.com/riptideslabs/tokenex) to retrieve UPSTs from OCI in exchange for an ID token from an external IDP. Under the hood, [tokenex](https://github.com/riptideslabs/tokenex) uses OCI's **Workload Identity Federation** to handle the token exchange. To enable this, you must configure trust between your IDP and OCI. For setup instructions, see the [OCI Workload Identity Federation guide](https://www.ateam-oracle.com/post/workload-identity-federation), specifically the section on **Identity Propagation Trust Configuration**. Without this setup, OCI will reject the ID token when [tokenex](https://github.com/riptideslabs/tokenex) attempts the exchange. This sample assumes: - OCI trusts the external IDP. - The trust configuration maps the ID token to a **service user** with privileges to list tenancy users. ### Important note on OCI Go SDK Support Currently, the OCI Go SDK does **not** natively support consuming UPSTs directly for authentication. To work around this, create an OCI SDK config file that uses the received UPST for authentication. Riptides' approach avoids persisting sensitive credentials (UPST and private keys) to disk. Instead, the UPST is injected **on the wire** during the `ListUsers` API call. This significantly strengthens security by eliminating the need for local credential storage. For a deeper dive into how this works, check out these related posts: - [On-the-Wire Credential Injection: Secretless AWS Bedrock Access Example](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) - [On-Demand Credentials: Secretless AI Assistant Example on GCP](/blog/on-demand-credentials-secretless-ai-assistant-example-on-gcp) ### Sample application ```go package main import ( "context" "crypto/md5" "crypto/rand" "crypto/rsa" "crypto/x509" "encoding/pem" "fmt" "log/slog" "os" "os/signal" "strings" "syscall" "time" "github.com/go-logr/logr" "go.riptides.io/tokenex/pkg/credential" "go.riptides.io/tokenex/pkg/oci" "go.riptides.io/tokenex/pkg/token" "github.com/oracle/oci-go-sdk/v65/common" "github.com/oracle/oci-go-sdk/v65/identity" ) func main() { ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() // clean up signal handler logger := logr.FromSlogHandler(slog.Default().Handler()) logger.Info("Press Ctrl+C to stop...") // setup credential provider to receive OCI credentials // create OCI credentials provider credProvider, err := oci.NewCredentialsProvider(ctx, logger) if err != nil { logger.Error(err, "failed to create OCI credentials provider") return } // under the hood the credential provider uses OCI workload identity federation to fetch user principal session tokens from OCI // the credential provider exchanges an input ID token for an OCI user principal session token // the input ID token can be obtained from any OIDC compliant IDP (e.g. Google, Microsoft, Auth0, Okta, etc.) // for this example, we use a static ID token provider that returns a hardcoded ID token issued by an OIDC compliant IDP // in a real application, you would implement the `token.IdentityTokenProvider` interface to create a dynamic ID token provider that fetches the ID token from an OIDC compliant IDP idTokenProvider := token.NewStaticIdentityTokenProvider("") // create RSA key pair for the application(workload) that is going to use the OCI user principal session tokens for authentication in order to be able to invoke OCI services privateKey, err := rsa.GenerateKey(rand.Reader, 2048) if err != nil { logger.Error(err, "failed to generate RSA key pair") return } privateKeyDer, err := x509.MarshalPKCS8PrivateKey(privateKey) if err != nil { logger.Error(err, "failed to marshal RSA private key to DER format") return } publicKeyDer, err := x509.MarshalPKIXPublicKey(privateKey.Public()) if err != nil { logger.Error(err, "failed to marshal RSA public key to DER format") return } hash := md5.Sum(publicKeyDer) parts := make([]string, len(hash)) for i, b := range hash { parts[i] = fmt.Sprintf("%02X", b) } fingerPrint := strings.Join(parts, ":") // currently the OCI SDK for Go v2 does not support using user principal session tokens for authentication directly // thus we either create a custom `common.ConfigurationProvider` that uses the user principal session tokens for authentication // or we use the `common.ConfigurationProviderForSessionToken` helper function // we use the later for this example and for this we need to create an OCI config file which will use the user principal session tokens we receive from the credential provider privateKeyFile, err := os.CreateTemp("", "private_key_*.pem") if err != nil { logger.Error(err, "failed to create private key file") return } defer os.Remove(privateKeyFile.Name()) privateKeyBlock := &pem.Block{ Type: "PRIVATE KEY", Bytes: privateKeyDer, } privateKeyPem := pem.EncodeToMemory(privateKeyBlock) if err := os.WriteFile(privateKeyFile.Name(), privateKeyPem, 0400); err != nil { logger.Error(err, "failed to write private key file") return } sessionTokenFile, err := os.CreateTemp("", "session_token_*") if err != nil { logger.Error(err, "failed to create session token file") return } defer os.Remove(sessionTokenFile.Name()) // create OCI config file that uses the session token file for authentication ociConfigFile, err := os.CreateTemp("", "oci_config_*") if err != nil { logger.Error(err, "failed to create OCI config file") return } defer os.Remove(ociConfigFile.Name()) // write OCI config file ociConfig := strings.Join([]string{ "[DEFAULT]", fmt.Sprintf("region=%s", "eu-frankfurt-1"), fmt.Sprintf("fingerprint=%s", fingerPrint), fmt.Sprintf("tenancy=%s", ""), fmt.Sprintf("key_file=%s", privateKeyFile.Name()), fmt.Sprintf("security_token_file=%s", sessionTokenFile.Name()), }, "\n") if err := os.WriteFile(ociConfigFile.Name(), []byte(ociConfig), 0400); err != nil { logger.Error(err, "failed to write OCI config file") return } // get OCI credentials creds, err := credProvider.GetCredentials(ctx, idTokenProvider, // supplies the ID token issued by an OIDC compliant IDP for the application(workload) that is going to use the OCI user principal session tokens for authentication. oci.WithClientID(""), // client ID of the application registered in OCI which is allowed to exchange ID tokens for OCI user principal session tokens. oci.WithClientSecret(""), // client secret of the application registered in OCI which is allowed to exchange ID tokens for OCI user principal session tokens. oci.WithIdentityDomainURL(""), // identity domain URL of the OCI tenancy where the application which is allowed to exchange ID tokens is registered. oci.WithRsaPublicKeyDer([]byte("")), // RSA public key in DER format of the application(workload) which is going to use the OCI user principal session tokens for authentication. ) if err != nil { logger.Error(err, "failed to get OCI credentials") return } // retrieve OCI credentials and updates before they expire for the identity that corresponds to the provided ID token go func() { defer stop() for { select { case <-ctx.Done(): return case credentialEvent := <-creds: if credentialEvent.Err != nil { logger.Error(credentialEvent.Err, "failed to get OCI credentials") return } token, ok := credentialEvent.Credential.(*credential.Token) if !ok { logger.Error(err, "failed to assert credential type") return } logger.Info("received new OCI credentials", "user principal session token", token.Token, "expires at", token.ExpiresAt.String()) // write session token to file; if the file already exists update it's content with a fresh session token if err := os.WriteFile(sessionTokenFile.Name(), []byte(token.Token), 0400); err != nil { logger.Error(err, "failed to write session token file") return } } } }() time.Sleep(5 * time.Second) // wait for initial credentials // use OCI credentials to call OCI services // in this example we use the OCI SDK for Go v2 to list the users in the OCI tenancy // the OCI SDK for Go v2 will use the OCI config file we created above which uses the user principal session tokens for authentication go func() { defer stop() // create OCI identity client configProvider, err := common.ConfigurationProviderForSessionToken(ociConfigFile.Name(), "") if err != nil { logger.Error(err, "failed to create OCI configuration provider") return } identityClient, err := identity.NewIdentityClientWithConfigurationProvider(configProvider) if err != nil { logger.Error(err, "failed to create OCI identity client") return } tenancyID, _ := configProvider.TenancyOCID() req := identity.ListUsersRequest{ CompartmentId: &tenancyID, // The OCID of the compartment (remember that the tenancy is simply the root compartment). } logger.Info("simulating application work...") listUsers := func() error { resp, err := identityClient.ListUsers(ctx, req) if err != nil { return err } for _, user := range resp.Items { logger.Info("user", "username", *user.Name) } logger.Info("----") return nil } // simulate application doing some work listUsers() for { select { case <-ctx.Done(): return case <-time.After(10 * time.Minute): err = listUsers() if err != nil { logger.Error(err, "failed to list OCI users") return } } } }() <-ctx.Done() logger.Info("context cancelled, exiting") } ``` ## Final thoughts With [tokenex](https://github.com/riptideslabs/tokenex), you no longer need to juggle cloud-specific SDKs or write custom refresh logic. It provides a **unified, extensible, and reactive** way to handle credentials across providers—out of the box. We’re excited to open source this library and invite the community to try it, give feedback, and contribute. 👉 Check out the code and documentation on GitHub: [riptideslabs/tokenex](https://github.com/riptideslabs/tokenex) --- ## Workload Identity Without Secrets: a Blueprint for the Post-Credential Era - URL: https://blog.riptides.io/workload-identity-without-secrets-a-blueprint-for-the-post-credential-era - Published: 2025-09-25 - Author: Janos Matyas - Category: Kernel - Tags: vision, spiffe, identity, zero-trust, kernel, linux ## The Safest Secret Is the One That Doesn’t Exist In our cloud-native world, secrets are liabilities: like API keys exposed in logs, vaulted credentials that never get revoked, or old tokens lying in backups. Each one is a ticking time bomb, waiting to be exploited. As [Felix Gaehtgens recently argued](https://www.linkedin.com/pulse/eliminating-nhis-how-spiffe-bootstraps-trust-without-chaos-gaehtgens-jsoaf), the biggest challenge isn’t just *storing* secrets, but the **bootstrap problem**: how do you securely give a workload its *first credential* without already having one in place? SPIFFE’s vision — eliminating credential chaos by issuing short-lived, verifiable identities at runtime — is the right answer to that decades-old chicken-and-egg dilemma. At Riptides, we share this vision. But we also believe that to make workload identity truly universal, it needs to be **seamless, short-lived or ephemeral, automatic**, and anchored where it can’t be bypassed: in the [kernel](/blog/rethinking-workload-identity-at-the-kernel-level). ## 1. The Hidden Cost of Credentials Every secret comes with baggage: - **Creation & injection** into environment variables, CI pipelines, metadata services. - **Storage** in vaults or config files. - **Rotation or revocation** a process rife with human error. - **Attack surface** across logs, backups, staging images, CI job definitions — the list is long. As Felix noted, cloud IAM roles and service accounts work well *within their boundaries*. But outside those silos, teams fall back to vaults and static credentials, multiplying risk with every workload. NHIM tools help organize the chaos, but they don’t eliminate it. Riptides flips the script: no secrets, no fragility, no leaks, just workload identity, [reimagined in the kernel](/blog/rethinking-workload-identity-at-the-kernel-level). ## 2. Cryptographic Trust Rooted in Identity, Not Treasures Instead of relying on “what you hold” (a password, a token), we look to “who you are” — and prove it continuously. - **SPIFFE IDs**: [Cryptographically rooted identities issued at runtime](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). - **Kernel-level attestation**: [Continuous proof of identity based on how and where the workload is running](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust). No vault. No tokens. Just cryptographic certainty tied directly to runtime provenance and purpose. Just [workload identity](/blog/rethinking-workload-identity-at-the-kernel-level) reimagined at the kernel level. ## 3. From SPIFFE to Seamless Adoption SPIFFE provides the standard foundation: short-lived SVIDs, attestation, and universal trust. But in practice, most deployments bolt identity onto sidecars and proxies. This adds complexity, weakens trust boundaries, and creates friction that slows mass adoption. In [our post *Rethinking Workload Identity at the Kernel Level*](/blog/rethinking-workload-identity-at-the-kernel-level), we argue that identity must be: - **Kernel-native** — bound directly to individual process instances. - **Ephemeral** — spun up with the workload and gone when it terminates. - **Process-isolated** — ensuring only the right code path receives identity. This makes workload identity invisible to developers, operationally effortless for teams, and universally enforceable. ## 4. A Foundation for the Post-Credential Era SPIFFE has shown us the way: workload identity without static credentials is possible. The next step is making it **automatic, seamless, and tied to the process itself** — so that it scales across clouds, VMs, bare metal, and edge environments without operational burden. Because in the post-credential era, identity isn’t something you distribute or rotate. It’s something you *prove* — **cryptographically, continuously, and without secrets**. --- ## “On demand credentials - Secretless AI assistant example on GCP” - URL: https://blog.riptides.io/on-demand-credentials-secretless-ai-assistant-example-on-gcp - Published: 2025-09-22 - Author: Mate Wolf - Category: Federation - Tags: federation, gcp, security, secret-injection Riptides [reimagines workload identity by embedding it directly in the Linux kernel](/blog/riptides-kernel-level-identity-and-security-reinvented), eliminating the need for sidecars, proxies, or application-level authentication logic. Built on [SPIFFE standards](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust), it enables per-process cryptographic identities that are portable, secure, and seamlessly federated across external systems without embedding secrets. Earlier posts detailed how SPIFFE Verifiable Identity Documents (SVIDs) are issued and enforced at the OS layer, providing strong runtime-scoped identities. Building on this, we introduced a [credential injection demo](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example), where AWS credentials were provisioned dynamically and just-in-time into outgoing requests—allowing workloads to securely access services like Amazon Bedrock without relying on static keys or tokens, addressing the [limitations of cloud-native federation](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure) alone. In this post, we take another step towards a world without stored credentials by exploring a demo application running on Google Cloud Platform that uses service account impersonation, credential files, and traditional access tokens to securely access GCP services. Through this example, we show how Riptides can provide the credential files and access tokens—two of Google Cloud’s standard authentication methods—and deliver them directly to managed workloads, all without storing a single secret. Our demo app is a compact chat assistant that showcases how a conversational UI can translate between geographic coordinates and postal addresses using a tool-enabled AI model hosted by Google, paired with a Geocoding API integration for retrieval-augmented lookups. The code repository contains a minimal implementation on top of assistant-ui and can be found [here](https://github.com/riptideslabs/geocode-assistant). Let's try to run the app! ## Run the app the standard way This demo requires two types of credentials: a **credential file** for the Vercel AI SDK for Google Vertex AI, which calls the LLM, and an **access token** for the tool that queries the Geocoding API. Both are referenced in `app/api/chat/route.ts` at `L41`, `L78` and `L126` and are sourced from environment variables via the `.env` file. Let's generate these credentials first. ### Setting up the GCP environment To get our demo app running, we need a service account with the right permissions. Specifically: - `roles/aiplatform.user` — required for calling **Vertex AI** - `roles/serviceusage.serviceUsageConsumer` — required for using the **Geocoding API** #### 1. Create a service account ```bash gcloud iam service-accounts create gcp-demo-svca ``` #### 2. Assign the roles Grant the service account the necessary permissions: ```bash gcloud projects add-iam-policy-binding \ --member="serviceAccount:gcp-demo-svca@.iam.gserviceaccount.com" \ --role="roles/aiplatform.user" ``` ```bash gcloud projects add-iam-policy-binding \ --member="serviceAccount:gcp-demo-svca@.iam.gserviceaccount.com" \ --role="roles/serviceusage.serviceUsageConsumer" ``` #### 3. Authenticate with the service account Next, download a JSON key for the service account and authenticate locally: ```bash gcloud auth login --cred-file= ``` Then retrieve an access token for the app: ```bash gcloud auth print-access-token ``` ### Running the demo app With authentication in place, set the environment variables in the project’s `.env` file: ```bash GOOGLE_CREDENTIALS_PATH= GOOGLE_MAPS_ACCESS_TOKEN= GOOGLE_PROJECT_ID= GOOGLE_PROJECT_REGION= ``` Start the application, and you’ll be able to query either **coordinates for an address** or an **address for coordinates**. ![Running the app the standard way](../../assets/credential-injection-gcp-demo/run-standard-way.png) At this point, the application is running successfully with the credentials we configured. But this setup raises an important question: is storing and injecting service account keys really the best approach?? ### The dark side of static credentials Static credentials and hardcoded service account keys may seem convenient, but they create brittle architectures, widen the blast radius of incidents, and work directly against zero-trust principles in modern cloud environments. Their key drawbacks are: - **Credential sprawl and copy-paste risk:** Keys proliferate across repos, laptops, and pipelines, making it difficult to inventory and revoke all copies after an incident or team change. - **Long-lived secrets increase blast radius:** A single leaked key enables persistent access until rotation, which is often slow and error-prone in multi-environment deployments. - **Weak auditability and provenance:** Access is attributed to a shared secret rather than a verifiable workload identity, hindering forensics and fine-grained policy enforcement. - **Operational drag from rotation:** Rotating static keys requires coordinated updates across services and environments, inviting downtime and configuration drift. - **Violates least-privilege over time:** Static keys tend to accumulate permissions and linger beyond their original purpose, especially when reused across pipelines or services. ### How does Riptides solve these problems? - Riptides issues [per‑workload identities and enforces them at the kernel layer](/blog/rethinking-workload-identity-at-the-kernel-level), eliminating the need to copy API keys into code, repos, or pipelines, which directly curbs proliferation across laptops and CI/CD systems. - Workloads [authenticate with short‑lived, automatically rotated identities rather than static keys](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), sharply limiting the usable window of any intercepted material and aligning with zero‑trust design. - Identities are [auto‑issued and rotated](/blog/the-critical-role-of-unique-workload-identity-in-modern-infrastructure) without coordinated app changes, avoiding mass redeployments or secret sync work across environments and pipelines. - Fine‑grained, [SPIFFE‑native workload identities](/blog/rethinking-workload-identity-at-the-kernel-level) and policies prevent the “one key used everywhere” pattern; privileges are scoped per workload and enforced continuously rather than accumulating over time. OK. But how can **[SPIFFE-native identities](/blog/rethinking-workload-identity-at-the-kernel-level)**, issued at the kernel level, be used on platforms that don’t yet support them natively, such as Google Cloud? This is where the **Riptides** [Identity Federation](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure) feature comes in. With this, it can translate [SPIFFE identities into an IDP token with its OIDC provider](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) and exchange this ID token for a credential that cloud providers understand, enabling secure workload authentication without relying on static credentials. ### Preparing GCP for Identity Federation To integrate SPIFFE identities into GCP, we need to set up **Workload Identity Federation**. This involves three steps: #### 1. Create a Workload Identity Pool ```bash gcloud iam workload-identity-pools create demo-pool \ --project= \ --location="global" \ --display-name="Demo Pool" ``` #### 2. Define a Workload Identity Provider ```bash gcloud iam workload-identity-pools providers create-oidc demo-provider \ --project= \ --location="global" \ --workload-identity-pool="demo-pool" \ --display-name="Demo Provider" \ --issuer-uri=https://lilia-consolidative-zayn.ngrok-free.app/oidc \ # Riptides IDP provider --attribute-mapping="google.subject=assertion.sub" ``` #### 3. Grant Service Account Impersonation Rights ```bash gcloud iam service-accounts add-iam-policy-binding gcp-demo-svca@.iam.gserviceaccount.com \ --project= \ --role="roles/iam.workloadIdentityUser" \ --member="principal://iam.googleapis.com/projects//locations/global/workloadIdentityPools/demo-pool/subject/spiffe://acme.corp/gcp-demo/reader" ``` In this step, we define which ID tokens, based on their subject, are authorized to assume the specified service account role. For example, tokens with the subject `spiffe://acme.corp/gcp-demo/reader` may be exchanged for credentials that impersonate the `gcp-demo-svca` service account. ### How does it work? The **Riptides Controlplane** can act as an identity provider, issuing OIDC tokens for workloads under its management. Thanks to the configuration above, GCP will trust these tokens for a specific workload identity (`spiffe://acme.corp/gcp-demo/reader`). Once trusted, GCP exchanges the Riptides-issued ID token for an access token that impersonates a chosen service account. Instead of dealing with downloaded key files and their lifecycle risks, you now get a **credential file** tied to workload identity federation—making authentication more secure, dynamic, and aligned with zero-trust principles. ### Define Riptides resources Riptides requires four resources to enable a fully secretless outbound call path to Google APIs. #### Define a Credential Source first ``` apiVersion: core.riptides.io/v1alpha1 kind: CredentialSource metadata: name: gcp-demo-acc-cs namespace: riptides-system spec: gcp: serviceAccount: "gcp-demo@.gserviceaccount.com" # the service account to impersonate oidcProviderId: "//iam.googleapis.com/projects//locations/global/workloadIdentityPools/demo-pool/providers/demo-provider" # the provider id that was defined earlier on GCP lifetime: "3600s" ``` The *CredentialSource* specifies how Riptides acquires short‑lived credentials from Google Cloud using Workload Identity Federation and service account impersonation, including the target OIDC provider and the service account to impersonate after token exchange. **Note:** Instead of service account impersonation, GCP provides the option to use direct resource access. In this case `serviceAccount` field should not be set and access should be granted for `principal://iam.googleapis.com/projects//locations/global/workloadIdentityPools/demo-pool/subject/spiffe://acme.corp/gcp-demo/reader` in the service we would like to access. #### Define a Service for the Geocoding API ``` apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: gcp-geocode-svc namespace: riptides-system spec: addresses: - address: geocode.googleapis.com # hostname that Riptides take care of port: 443 tags: ["geocode", "gcp"] labels: api: geocode # selector that can be used later to point out this service external: true # external says this service is outside Riptides-managed environment ``` The Service resource instructs Riptides to observe and, when configured, interpose on TLS connections targeting `geocode.googleapis.com` so that policy-driven credential injection can occur on egress. #### Define a Workload Identity for the demo app ``` apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: gcp-demo-reader namespace: riptides-system spec: selectors: - process:cmdline: "next-server (v15.5.2)" workloadID: gcp-demo/reader # this name appears in SPIFFE id scope: agent: id: # on which nodes this identity is defined and applied connection: tls: mode: PERMISSIVE # needed to let http connections to connect to the UI interface egress: - selectors: - api: geocode # it defines that sercices with this selector should be intercepted credentialName: gcp-demo-acc-cs-gcp-demo-reader-wc # this credential should be injected on the wire connection: tls: intercept: true # it tells injection can happen even if the connection to interrupt is TLS ``` WorkloadIdentity declares which running processes are recognized as the gcp-demo/reader workload and binds them to a SPIFFE-style identity (for example, spiffe://acme.corp/gcp-demo/reader) to be used as the subject in issued tokens and policies, aligning with Google’s impersonation subject expectations. The egress rule directs Riptides to inject the designated GCP credential on connections matching the geocode selector, enabling transparent, per-connection token delivery to `geocode.googleapis.com`. Because the Service above carries the same selector, every outbound connection to `geocode.googleapis.com` will receive a short‑lived GCP access token automatically at connection time. #### Define a Workload Credential ``` apiVersion: core.riptides.io/v1alpha1 kind: WorkloadCredential metadata: name: gcp-demo-acc-cs-gcp-demo-reader-wc namespace: riptides-system spec: workloadID: gcp-demo/reader # name of the workload identity we would like to be granted to reach the credential source credentialSource: gcp-demo-acc-cs # name of the credential source ``` WorkloadCredential binds the CredentialSource to the WorkloadIdentity and exposes ephemeral credential artifacts via `sysfs` for that workload, including a JSON file path that client libraries can reference for on-demand authentication. The path is built from a unique hash of the `WorkloadIdentity`'s ID and the `WorkloadCredential`'s name. ``` status: paths: - /sys/kernel/riptides/credentials/efa8626c-472c-5ac0-86d9-04a445c56c2b/gcp-demo-acc-cs-gcp-demo-reader-wc/gcp_credentials.json ``` Only processes authenticated as `gcp-demo/reader` can read the sysfs-backed credential file, enforcing least privilege at the identity boundary rather than via static secrets. ### Run the app secretless In practice, the demo app obtains the gcp-demo/reader identity, which authorizes secure read access to the sysfs-projected credential file and enables Riptides to inject short‑lived access tokens (via impersonation) on egress to `geocode.googleapis.com` without ever persisting credentials in the app or environment. Set environment variables as follows: ``` GOOGLE_CREDENTIALS_PATH=/sys/kernel/riptides/credentials/efa8626c-472c-5ac0-86d9-04a445c56c2b/gcp-demo-acc-cs-gcp-demo-reader-wc/gcp_credentials.json GOOGLE_MAPS_ACCESS_TOKEN=none ``` A placeholder value for `GOOGLE_MAPS_ACCESS_TOKEN` is acceptable because the real token is issued just-in-time and injected at the connection layer, rotated automatically per policy and lifetime settings defined in the CredentialSource. With this setup, the application can call the Geocoding API successfully, end-to-end, without storing or managing long‑lived secrets, relying instead on federated identity and short‑lived, policy-scoped tokens delivered at runtime. ![Run the app with Riptides](../../assets/credential-injection-gcp-demo/run-riptides-way.png) ### Final thoughts In this demo, Riptides delivered **secretless authentication** on GCP by enforcing [SPIFFE identities at the kernel](/blog/rethinking-workload-identity-at-the-kernel-level), federating them into [Workload Identity Federation](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure), and supplying two runtime paths: a federated credential file for Vertex AI and on‑the‑wire token injection to `geocode.googleapis.com` — **no downloaded keys, no app changes, no secret sprawl**. The result is concrete risk reduction: **static key elimination, automatic short‑lived impersonated tokens per connection, least‑privilege tied to a verifiable workload subject, and full auditability of who accessed what, when**. This isn’t just an implementation detail, it’s a security philosophy — a mantra of no static secrets, no blind trust, only SPIFFE based verifiable identities at runtime. --- ## Shai-Hulud and the Secret Hunters: How npm Installs Turn into Intrusions - URL: https://blog.riptides.io/shai-halud-and-the-secret-hunters-how-npm-installs-turn-into-intrusions - Published: 2025-09-18 - Author: Janos Matyas - Category: Security - Tags: credentials, supply-chain **TL;DR** A coordinated npm supply-chain campaign (self-branded *Shai-Halud*) trojanized popular packages to run a `bundle.js` payload at install time. The payload downloads a secret scanner (TruffleHog), validates discovered tokens (GitHub, npm, AWS, etc.), and repurposes them for persistence, exfiltration, or lateral movement (including planting GitHub Actions workflows). The root problem is static credentials living on disk or in CI - remove those and the attack collapses. Below: a tight forensic summary, Riptides’ operational stance, and two practical Riptides examples (GitHub + AWS). ## From Install to Intrusion: How npm Packages Became Secret Hunters In September 2025 a coordinated npm supply-chain campaign, calling itself **Shai-Halud**, began trojanizing popular packages — most visibly `@ctrl/tinycolor` — and rapidly expanded to hundreds of packages across multiple maintainers. Each infected package carried a hidden payload (`bundle.js`) that executed at install time, fetched a platform-specific TruffleHog binary, and immediately hunted the host for developer and CI credentials. ### Payload behavior (at a glance) - **Secret scanning** — downloads TruffleHog and scans local files, repo history, and environment variables. - **Token harvesting** — probes for `GITHUB_TOKEN`, `NPM_TOKEN`, `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, then calls provider endpoints (e.g., `/-/whoami`, GitHub API, instance metadata) to validate which tokens work. - **Persistence** — if GitHub tokens are valid, it can write a workflow file (e.g., `shai-hulud-workflow.yml`) into `.github/workflows` to regain access via CI. - **Exfiltration** — aggregates findings (e.g., `data.json`) and posts them to a hardcoded webhook; in some cases it created public repos under victims’ accounts. ### Why this campaign is especially dangerous It combines *scale* (popular, transitive packages) with *credential validation*. Once a single reusable secret is found, the attacker gains immediate foothold: unauthorized publishes, CI persistence, cloud lateral movement. Because the malicious logic runs at install time, developer laptops, CI runners, and build environments are all at risk. Supply-chain attacks like this are practically unavoidable: zero-days in dependencies will always surface, and even without them many developers don’t update packages regularly. The only realistic defense is to reduce the blast radius, which starts with removing static secrets that attackers can steal and reuse for lateral movement. ## Tokens as Treasure The Shai-Halud campaign makes one thing painfully clear: the single biggest enabler of modern supply-chain attacks is **static, long-lived credentials lying around in files, environment variables, or CI pipelines**. Once a trojanized package can read a filesystem, a build cache, or a CI runtime, an attacker doesn’t need to exploit anything else — they already have a “door key” into your systems. At Riptides, we frame this as a supply-chain symmetry problem: attackers only need **one reusable secret** to start [lateral movement](/blog/sharepoint-under-siege-lateral-movement-is-still-securitys-blind-spot) and persistence; defenders have to eliminate **every** reusable secret. That’s why we advocate a hard operational rule: **never store long-lived or static credentials on disk, in repositories, or in CI configurations.** ![How Riptides does it](../../assets/shai-halud-exploit/illustration1.png) The problem isn’t just that secrets are lying around. Modern supply-chain malware is smart enough to validate credentials in real time. TruffleHog, used by Shai-Halud, doesn’t just harvest keys; it checks whether they work. GitHub tokens are tested against the API, AWS keys are checked via instance metadata or STS calls. If the token works, it’s immediately leveraged: the malware can write GitHub Actions workflows, publish malicious packages, or assume cloud roles. This transforms a local compromise into account-wide or even cross-account access — and all it took was a secret sitting in a file. Static credentials aren’t just dangerous — they’re *high-value targets*. Every credential that persists beyond a single, scoped operation is a potential beachhead. Developer machines, CI runners, build agents — all of these environments are full of sensitive tokens and metadata. One compromised node, one scanned file, and an attacker has an actionable key in hand. Once that happens, lateral movement is almost inevitable. At Riptides, our philosophy is simple: **move identity off disk, limit its lifetime, and bind it to the workload that needs it**. We issue ephemeral tokens via OIDC or SPIFFE-backed flows and anchor non-human identity at the kernel level. Tokens are injected on-the-wire for network requests and never exposed on disk or in environment variables. Even if malware runs in the same node, it sees nothing it can reuse. Dependencies introduced via supply chain attacks operate strictly in **user-space**, where they cannot access credentials protected by Riptides; as [Riptides injects ephemeral credentials directly in kernel-space](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), malicious packages are unable to intercept or exfiltrate secrets, even if running on the same compromised node. This architectural boundary ensures that credential theft by supply chain malware is fundamentally blocked at the OS level. ![How Riptides does it](../../assets/shai-halud-exploit/illustration2.png) This approach changes the calculus for attackers. A stolen file becomes noise, not a beachhead. Replayed tokens expire quickly or are scoped to a workload that the attacker cannot control. By minimizing persistence and enforcing strict least privilege, the supply chain becomes far less fertile ground for exploitation. **In short:** static secrets are the Achilles’ heel of the software supply chain. Remove them, and the attack surface shrinks dramatically. At Riptides, [we replace them with ephemeral, verifiable credentials, injected on the wire](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example), ensuring that stolen files and environment dumps are useless to attackers. ## 3. Two examples from the current Shai-Halud exploit ### Example A — GitHub: install → secrets hunt **Typical exploit start** `bundle.js` runs during `npm install`, scans the filesystem and environment for GitHub tokens. Valid tokens are immediately reused to push workflows, publish packages, or exfiltrate secrets. **Riptides options for workloads that need GitHub access**: 1. **Static-but-isolated token (sysfs injection)** - The workload receives a static token *but it is never written to disk*. The kernel exposes it via a kernel-backed sysfs object readable only by that process; the kernel injects the token onto the wire when the process talks to GitHub. - **Why it helps:** a `postinstall` script or malicious process in `node_modules` cannot read that sysfs object and so cannot steal the token. 2. **Short-lived GH token (recommended)** - The workload is associated with a **Secret Source** and a GitHub **Service** in Riptides. Riptides mints ephemeral GitHub tokens on demand and [injects](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) them into outgoing requests at the kernel layer. Tokens never appear in process files, logs, or CI config; they are short TTL and rotated automatically. - **Why it helps:** even if the payload finds strings that look like tokens, those tokens are stale; live tokens exist only in-flight and only for tightly scoped operations. **Takeaway:** Move GitHub credentials off disk, prefer kernel injection and short TTLs. ### Example B — AWS access: secretless, on-the-wire access **Attack scenario** `bundle.js` searches for `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`. Valid keys let an attacker enumerate roles, list S3, create keys, assume roles, or spin up resources, and enabling cloud lateral movement. **Riptides secretless flow (on-the-wire SigV4 signing)**: 1. Riptides is registered as an OIDC IdP / trusted issuer in the AWS account and roles are created that accept Riptides-issued ID tokens. 2. A `CredentialSource` / `WorkloadCredential` declares which role a workload can assume and the allowed scope/TTL. 3. When the workload makes an AWS API request, the kernel interposes: Riptides mints short-lived STS credentials, performs SigV4 signing (or injects credentials into the wire), and the request leaves the host already signed — the client process never sees raw keys. **Why this breaks the exploit**: - No persistent keys to find on disk or in environment variables. - Temporary creds are short TTL, narrowly scoped, and bound to the workload context. - Kernel-anchored identity prevents trivial replay or cross-process misuse. **Takeaway:** Sign and inject AWS credentials on the wire; never bake keys into clients, images, or CI. You can read more about credential injection and federation done with Riptides in the following posts: - [On-the-Wire Credential Injection: Secretless AWS Bedrock Access example](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example) - [Why Cloud-Native Federation Isn’t Enough for Non-Human Identities in AWS, GCP, and Azure](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure) - [Introducing libsigv4: AWS SigV4 Signatures in Portable C with Kernel Compatibility](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility) ## Closing — why this matters and what to do next Shai-Halud is a clear reminder: **the weakest link in a software supply chain is often a credential sitting on disk**. The most effective defense isn’t another scanner, it’s changing where identity lives. Move credentials off disk, mint them just-in-time, scope them tightly, and keep them in memory or injected on the wire by an authority that can be audited and controlled. At Riptides, we’re strong believers in **[SPIFFE-based workload identities](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust)**, enabling fine-grained, zero-touch authentication with secure, ephemeral credentials and no secrets ever stored. This is the most robust and future-proof way to handle non-human identities, and it works seamlessly across multi-cloud environments. But where [SPIFFE adoption](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure) isn’t possible, Riptides provides a fallback: on-the-wire credential injection, delivering the right cloud or service credentials directly into the request stream without ever touching the application. **[On-the-wire credential injection](/blog/on-the-wire-credential-injection-secretless-aws-bedrock-access-example)** with Riptides enables secure, secretless invocation of external services. In this example, we demonstrated the approach with an AWS Bedrock agent, but Riptides provides the same seamless support across all major cloud providers — AWS, GCP, and Azure. Beyond cloud credentials, Riptides can also inject **OAuth2 tokens**, whether sourced from Kubernetes secrets or obtained via OAuth2 authorization code flows, directly into client requests. This pattern can be extended to agents, applications, and services in any environment, offering a unified, secure approach for managing non-human identities at scale. --- ## SPIFFE Meets OAuth2: Current landspace for Secure Workload Identity in the Agentic AI Era - URL: https://blog.riptides.io/spiffe-meets-oauth2-current-landscape-for-secure-workload-identity-in-the-agentic-ai-era - Published: 2025-09-15 - Author: Zsolt Varga - Category: SPIFFE - Tags: SPIFFE, Oauth2, AI, authn, authz Identity is the cornerstone of security. For human users, identity is well defined and standardized: single sign on, OpenID Connect, and multi factor authentication are widely deployed. For non human identities such as workloads, microservices, and increasingly AI agents, the situation is much less mature. Long lived API keys are copied into configuration files, secrets are scattered through CI pipelines, and revocation is either manual or ignored. At Riptides, we address this gap directly. Our approach is simple but powerful: every workload in the system [receives a unique SPIFFE ID](/blog/rethinking-workload-identity-at-the-kernel-level), anchored in [strong cryptography and tied to process metadata](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). This happens inside the [Linux kernel](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) and is completely transparent to the application. Applications do not need to be modified. They inherit a secure, short lived identity that proves who they are. Identity bootstrapping is the foundation. But workloads need to use this identity in existing protocols, and the one of the most important ones is OAuth2. Over the past decade OAuth2 has become the universal cross-authorization protocol for APIs and services. It is primarily used to let one service access resources from another on behalf of a human, with the user’s identity and consent central to the flow. As systems have evolved, the same model has been extended to cover non-human actors such as workloads, microservices, and autonomous agents, where no human is directly involved in every flow. Still, there is plenty of room for improvement, since secure workload identity and authorization are becoming essential. This post explores these specifications and drafts, how they connect to [SPIFFE](/blog/rethinking-workload-identity-at-the-kernel-level), and why they are essential in an agentic AI world. ## Why Unique Workload Identity Matters In human workflows we demand per user identity. Every login is linked to a person, access is logged, scoped, and revocable. Workloads rarely have this property. Instead, they often share a single API key or secret across many services. The risks are clear: - Credential sprawl: Keys and tokens get copied across code, configuration, and pipelines - No provenance: After an API call lands, it is hard to prove which workload generated it - Replay and lateral movement: A single compromise enables reuse of the same credential elsewhere - Operational burden: Secret rotation is brittle and disruptive With SPIFFE based workload identity, each workload has a unique short lived identifier `spiffe://trustdomain/workload/path`. That identifier is proven cryptographically and can be validated by any peer. In agentic AI, where autonomous agents instantiate dynamically and invoke APIs across multiple systems, this property becomes critical. Without unique workload identity, agent chains are opaque. With it, they become traceable, auditable, and accountable. ## OAuth2 and Workload Identity: The Standards Landscape OAuth2 was originally designed for user authorization, but it has steadily evolved to support machine and workload use cases. The following RFCs and drafts together form the building blocks for integrating SPIFFE based identities into OAuth2 flows. ### Client Onboarding and Dynamic Registration The first step is how a workload becomes a client of an Authorization Server. #### [RFC 7591](https://datatracker.ietf.org/doc/html/rfc7591) - Dynamic Client Registration Protocol Defines a standard way for clients to register themselves dynamically with an Authorization Server by sending configuration such as redirect URIs, grant types, and authentication methods. Instead of manual configuration, the client can request credentials programmatically. This enables scalable onboarding of new clients in distributed systems. #### [RFC 7592](https://datatracker.ietf.org/doc/html/rfc7592) - Dynamic Client Registration Management Protocol Extends RFC 7591 by allowing registered clients to read, update, or delete their registration. Management is secured with a `registration_access_token` issued during registration. This makes client lifecycle management more flexible and automated. #### [Dynamic Client Registration with Trusted Issuer Credentials](https://datatracker.ietf.org/doc/draft-kasselman-oauth-dcr-trusted-issuer-token/) - *draft* Enhances Oauth2 DCR by using tokens issued by trusted identity systems (such as SPIFFE) as software statements. Authorization Servers can validate these trusted issuer tokens automatically. This allows secure zero-touch client registration at scale. #### [Client Registration on First Use with SPIFFE](https://datatracker.ietf.org/doc/draft-kasselman-oauth-spiffe/) - *draft* Proposes extending OAuth2 DCR with SPIFFE. Workloads can register on first use by presenting a SPIFFE credential instead of a pre-provisioned secret. This eliminates manual setup for ephemeral workloads. ### Client Authentication and Proof of Possession Once registered, the workload must authenticate at the token endpoint. #### [RFC 8705](https://datatracker.ietf.org/doc/html/rfc8705) - Mutual TLS Client Authentication and Certificate Bound Access Tokens Introduces mTLS as a client authentication method at the token endpoint and binds issued access tokens to the client certificate used. This ensures that only the rightful holder of the private key can use the token. It significantly reduces the risk of token replay attacks. #### [RFC 9449](https://datatracker.ietf.org/doc/html/rfc9449) - Demonstrating Proof-of-Possession at the Application Layer (DPoP) Provides a way for clients to bind tokens to a public key using signed HTTP requests. Even if an access token is stolen, it cannot be reused without the private key. It is useful in contexts where TLS client authentication is not available. #### [SPIFFE Client Authentication](https://datatracker.ietf.org/doc/draft-schwenkschuster-oauth-spiffe-client-auth/) - *draft* Focuses on standardizing SPIFFE-based client authentication for OAuth2. Workloads present their SPIFFE SVID directly as proof of identity at the token endpoint. This provides a consistent method to integrate SPIFFE identities into OAuth2 flows. These mechanisms replace brittle client secrets with cryptographic proof of possession anchored in SPIFFE. ### Resource Scoping and Metadata Discovery Tokens should be scoped to the correct audience and discovered automatically. #### [RFC 8707 – Resource Indicators for OAuth 2.0](https://datatracker.ietf.org/doc/html/rfc8707) Defines the `resource` parameter, allowing clients to request tokens scoped to a specific API or resource server. This avoids issuing broadly scoped tokens valid for multiple services. It supports least-privilege principles in multi-service environments. #### [RFC 9728 OAuth 2.0 Protected Resource Metadata](https://datatracker.ietf.org/doc/html/rfc9728/) Standardizes how a resource server can publish metadata at a `.well-known/oauth-protected-resource` URI. Metadata includes supported scopes, authorization servers, and token presentation methods. This allows clients to discover how to interact with a resource dynamically. #### [RFC 8414 Authorization Server Metadata](https://datatracker.ietf.org/doc/html/rfc8414) Specifies how an Authorization Server publishes metadata about its endpoints and capabilities at a `.well-known/oauth-authorization-server` URI. Clients can discover token endpoints, authorization endpoints, signing keys, and supported features. This reduces manual configuration and improves interoperability. Together these provide dynamic discovery for both the Authorization Server and the Resource Server. ### Context Propagation and Workload Chains Agentic AI workloads rarely stop after one hop. Requests propagate across multiple services, each adding context. Preserving identity and context through these chains is critical. #### [Transaction Tokens](https://datatracker.ietf.org/doc/draft-ietf-oauth-transaction-tokens/) - *draft* Defines short-lived Transaction Tokens (JWTs) that capture identity and context when a request first enters a system. These tokens can be passed along or replaced as requests propagate between workloads. This enables verifiable provenance and prevents unauthorized internal calls. ## Why This Matters for Agentic AI AI agents orchestrate tasks across APIs and services at high speed and scale. Without unique workload identity, these agents are opaque and untraceable. Tokens are reused, provenance is lost, and the attack surface grows. With SPIFFE based identity integrated into OAuth2 flows, each agent is distinct, auditable, and accountable. Tokens are least privilege by default, scoped to resources, and bound to the workload that obtained them. Context can be propagated securely across chains of agents. This shifts the security model from managing static secrets to verifying dynamic, cryptographically provable identities. ## Conclusion The evolution of OAuth2 shows a clear path from user centric authorization to full support for non human identities. RFCs 7591, 7592, 8705, 8707, 9728, and 8414, together with drafts for transaction tokens, SPIFFE integration, and trusted issuer tokens provide some of the missing pieces. For this vision to take hold, these specifications must be adopted by the major OAuth2 Authorization Server implementations. When that happens, workload identity will no longer be a bolt-on feature but a native part of the authorization ecosystem. At Riptides, we are committed to supporting these standards in our platform, and we encourage existing OAuth2 platforms to adopt them as well, since only broad implementation will make workload identity truly interoperable and seamless across environments. This is the next frontier for OAuth2: moving beyond human-centric consent toward a universal model of secure workload identity. As agentic AI grows in scale and autonomy, these capabilities are not optional. They are the foundation for building systems that are secure, auditable, and trustworthy by design. --- ## From Keys to Handshakes: How Cryptography Powers Riptides - URL: https://blog.riptides.io/from-keys-to-handshakes-how-cryptography-powers-riptides - Published: 2025-09-08 - Author: Balint Molnar - Category: Security - Tags: kernel, ec, rsa, encryption, cryptography ## Cryptography at Riptides Modern infrastructure demands strong, scalable, and transparent identity, especially for non-human actors such as services, workloads, jobs, and agents. Riptides introduces a novel approach by anchoring non-human identity directly in the kernel, leveraging SPIFFE, kTLS, and in-kernel mTLS handshakes to deliver zero-intrusion security - combining identity and encrypted communication seamlessly for user-space applications. To realize this approach, Riptides builds on proven cryptography, specifically TLS. At its core, TLS relies on keys, certificates, encryption, digital signatures, and key exchange. In this post, we’ll walk through the fundamentals of these building blocks: symmetric cryptography, asymmetric cryptography (RSA and Elliptic Curve Cryptography), digital signatures, and key exchange. We’ll also share performance benchmarks from Riptides using different key types to highlight the trade-offs in practice. ## What is Cryptography? The word *cryptography* comes from the Ancient Greek *kryptós*, meaning “hidden” or “secret.” At its core, cryptography is the science of protecting information, allowing people (and machines) to communicate securely, even in the presence of adversaries. A simple analogy is a locked treasure chest: only someone with the correct key can open it. Cryptography underpins almost everything in the digital world, from online banking to AI agents exchanging data across networks. This post isn’t meant to be an exhaustive guide, but rather a focused introduction to the essentials. One of the central challenges is ensuring that data can be transmitted so only the intended recipient can read it and no one else. ## TLS Requirements Transport Layer Security (TLS) is the backbone of secure communication on the internet. At a high level, TLS guarantees three critical properties: - **Confidentiality**: Only authorized/intented recipients can read the data. - **Integrity**: Any tampering with the data is detectable. - **Authentication**: The parties involved can prove they are who they claim to be, and can verify each other’s identity. To deliver on these guarantees, TLS combines several cryptographic building blocks: - **Encryption**: Keeps the content of messages private (using both symmetric and asymmetric encryption). - **Digital Signatures**: Ensure authenticity of messages and certificates. - **Key Exchange**: Securely establishes/agrees a shared secret between parties. - **Certificates**: Bind public keys to verified identities, enabling trust. In the next sections, we’ll look at these components in more detail, starting with symmetric and asymmetric cryptography — the foundation of TLS. ### Symmetric Cryptography In symmetric cryptography, the same key is used for all operations. The main challenge is **securely sharing this key** between the sender and receiver without exposing it publicly. ![Symmetric Encryption](../../assets/ec-rsa-perf/symm.png) Key derivation functions (KDFs) help generate encryption keys from passwords, which are easier for humans to remember. Modern symmetric encryption schemes usually combine multiple algorithms into a single **encryption scheme (cipher construction)**. For example: ``` Key derivation algorithm + Symmetric cipher algorithm + Cipher block mode + Message authentication (MAC) algorithm ``` Examples of combined schemes include: `CHACHA20_POLY1305_SHA256` or `AES_GCM_256`. Typical shared key sizes are 128, 192, or 256 bits. Modern symmetric algorithms include **AES** and **ChaCha20**. We differentiate two modes of symmetric encryption: - **Block Ciphers (AES)**: Encrypt data in fixed-size blocks. - **Stream Ciphers (ChaCha20)**: Encrypt data byte by byte as a stream. For example, when you see `AES_GCM_256`, it includes multiple components: - **Block-to-stream transformation (GCM)**: Allows encrypting data of arbitrary size using a block cipher. - **Block Cipher (AES):** Encrypts the data blocks securely. - **Message Authentication (MAC):** Ensures the decrypted message matches the original and hasn’t been tampered with. To further enhance security, **authenticated encryption** combines encryption with a verification code to ensure message integrity. If this scheme is used, decryption success indicates both that the key is valid and the message is untampered. An even more advanced concept is **Authenticated Encryption with Associated Data (AEAD)**. AEAD binds additional data (associated data) to the ciphertext and context, helping detect attempts to cut and paste encrypted messages between different contexts. #### AES AES stands for **Advanced Encryption Standard**, also known as **Rijndael**. It is considered highly secure; although some attacks have been published, there are no known practical exploits. AES is one of the most widely used ciphers in TLS. Modern CPUs often include hardware enhancements to accelerate AES encryption and decryption. AES is a **block cipher** with a fixed block size of 128 bits, regardless of the key size. It usually requires an **initialization vector (IV)** a non-secret, random value used to add variability to the encryption. During encryption, AES takes the key and input data to produce the ciphertext. The IV is then combined with the ciphertext, and if authenticated encryption (AE) is enabled, a **Message Authentication Code (MAC)** is also included to ensure integrity. Decryption works in reverse. The AES decryptor first extracts the IV and uses it with the key to recover the original message. If AE is enabled, the MAC is verified to confirm the message has not been tampered with and that the key is valid. #### ChaCha20 **ChaCha20** is a highly secure, lightweight 256-bit **stream cipher**. It uses a 256-bit key along with a 96-bit nonce to encrypt data. The algorithm first derives an inner key from the user-provided key and nonce, then initializes the cipher and encrypts data blocks sequentially. The ciphertext is produced by XORing the plaintext with the output of the encryption step. Industry standards recommend using **ChaCha20-Poly1305** or **AES-256-GCM**, as both are highly secure and well-tested. ChaCha20 is roughly **three times faster than AES-GCM**, which is why it is the default choice in Riptides. ### Asymmetric Cryptography / Public Key Cryptography As mentioned earlier, **public key cryptography** uses separate keys for encryption and decryption: a **public key** and a **private key**, which are mathematically linked. Public key cryptography is not only used for encryption and decryption but also for **digital signatures** and **key exchange**. ![Asymm Encryption](../../assets/ec-rsa-perf/asymm.png) Encrypting data directly with asymmetric algorithms is computationally expensive compared to symmetric cryptography. Typically, a **hybrid approach** is used: - A symmetric key is generated to encrypt the message. - The symmetric key is then encrypted using the recipient’s public key. For decryption, the recipient first uses their private key to decrypt the symmetric key, and then uses that key to decrypt the message. Asymmetric cryptography can also be used “in reverse,” where data is encrypted with a private key. This proves the identity of the owner and is the basis for digital signatures. We distinguish two widely used asymmetric cryptosystems: #### RSA **RSA** (Rivest-Shamir-Adleman) is one of the earliest public key cryptosystems. It is based on **modular exponentiation** and the computational difficulty of factoring large integers. In RSA, the public and private keys are generated together. Typical RSA key lengths are 2048, 3072, or 4096 bits. Longer keys offer higher security but require more computation. Keys above 3072 bits are generally considered secure. RSA key generation involves finding three large integers: `e`, `d`, and `n`, such that: ``` (x^e)^d ≡ x (mod n) for all x in [0..n) ``` - `n` is the **modulus** and defines the key length (e.g., 3072 bits). - `(n, e)` is the **public key**, where `e` (commonly 65537) is the public exponent. - `(n, d)` is the **private key**, where `d` is the private exponent. #### ECC **Elliptic Curve Cryptography (ECC)** is a more modern public key cryptosystem based on the **elliptic curve discrete logarithm problem** a problem as hard as integer factorization. ECC provides strong security with much smaller keys: a **256-bit ECC key** is roughly equivalent to a **3072-bit RSA key**. Private keys are simply integers in the range of the curve’s field, making key generation extremely fast. Public keys are points on the curve, represented as coordinate pairs `{x, y}`. ECC supports different curves, which offer varying levels of security, performance, and key length. The most commonly used curves are: - `secp256k1` - `secp384r1` Both RSA and ECC can be used for: - Key exchange - Digital signatures - Encryption We’ve already covered encryption; next, let’s look at **key exchange** and **digital signatures**, which are also essential for TLS. ### Digital Signature A **digital signature** is a cryptographic scheme used to verify the authenticity of digital messages or documents. A valid signed document gives the recipient confidence that the message came from a known sender. Think of it like a handwritten signature—but for the digital world. ![Digital Sig](../../assets/ec-rsa-perf/digitalsig.png) A message is signed using a **private key**, and the signature can later be verified with the corresponding **public key**. Signed messages cannot be altered without detection, providing **authentication, message integrity, and non-repudiation**. Digital signatures can be implemented using **RSA** or **DSA**, while **ECC** provides an equivalent method called **ECDSA**. ### Key Exchange **Key exchange** is the process by which a secret key is securely shared between two parties. It ensures that no one else can access the key, which is essential for establishing secure communication in TLS. During the TLS handshake, the parties negotiate a secret key that will be used for encryption during the session. This handshake happens millions of times in web browsers every day. Common key exchange methods include: - **Diffie-Hellman Key Exchange (DHKE)**: one of the earliest widely used methods. - **Elliptic Curve Diffie-Hellman (ECDH)**: a more modern, efficient version using elliptic curves. With this, we now have a basic understanding of the cryptographic building blocks used in TLS. To maximize compatibility, **Riptides relies on certificates and TLS using SPIFFE**. ## ECC vs RSA with Riptides We recently introduced Elliptic Curve Cryptography (ECC) to Riptides and ran a brief performance test comparing TLS using RSA and ECC. For this comparison, we focus on the TLS handshake, as this is where the heavy computation happens. During data transfer, the same symmetric cipher is used, so the main difference in speed comes from the handshake itself. ### Setup For this benchmark, we used a typical developer laptop environment: a MacBook Air with an M3 processor and 16 GB of RAM. Since our benchmarks require Linux, we ran a Lima virtual machine with the default settings and the standard Ubuntu template. To measure TLS handshake performance, we used [Tempesta’s TLS perf tool]((). This lightweight tool stresses the **TLS handshake only** and quickly resets TCP connections after each attempt. The server part of the test is a simple **Go HTTP server**, with TLS provided using Riptides. For both RSA and ECC tests, we limited parallel connections to **100**, used **two threads**, and ran the benchmark for **10 seconds**. ### Results With **RSA cryptosystem,** we used a **4096-bit RSA key pair** with the `DHE-RSA-CHACHA20-POLY1305` cipher. Since the library terminates the connection immediately after the handshake, the `CHACHA20-POLY1305` symmetric cipher is not relevant here. For **certificate validation** and **key exchange**, the **Diffie-Hellman protocol** and RSA are used. With **ECC cryptosystem,** we used a **384-bit EC key pair** with the `ECDHE-ECDSA-CHACHA20-POLY1305` cipher. As before, the `CHACHA20-POLY1305` symmetric cipher can be ignored since the benchmark focuses on the TLS handshake. For **certificate validation** and **key exchange**, **Elliptic Curve Diffie-Hellman (ECDH)** and **ECDSA** are used. With **ECC,** we achieved roughly **6.5× faster** TLS handshake performance over **RSA**! Moreover, **ECC-384** provides better security than **RSA-4096**, as it is roughly equivalent to **RSA-7680** in terms of cryptographic strength. ## Conclusion In this blog post, we briefly introduced cryptography by exploring different cryptosystems, including **symmetric** and **asymmetric** cryptography. While discussing asymmetric cryptography, we compared **RSA** and **ECC**, and explained the roles of **key exchange** and **digital signatures**. With this foundation, we now understand the cryptographic building blocks behind TLS. Finally, we gave a rough estimate of Riptides’ performance using RSA and ECC for TLS, showing the significant speed and security benefits of ECC. There are many resources online if you want to dive deeper into cryptography, and we’re also preparing more blog posts on this topic so stay tuned! --- ## On-the-Wire Credential Injection: Secretless AWS Bedrock Access example - URL: https://blog.riptides.io/on-the-wire-credential-injection-secretless-aws-bedrock-access-example - Published: 2025-09-01 - Author: Sebastian Toader - Category: Federation - Tags: federation, non-human identity, aws, workload-id We’ve previously delved deep into how Riptides [reimagines workload identity by anchoring it directly in the Linux kernel](/blog/rethinking-workload-identity-at-the-kernel-level), removing reliance on sidecars, proxies, or application-level identity logic. Our approach is rooted in SPIFFE standards and designed for seamless federation, enabling per-process cryptographic identity issuance right at the OS layer. In earlier posts, we’ve explored how [SPIFFE Verifiable Identity Documents (SVIDs) are issued and enforced in the kernel](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), providing each workload with a strong, runtime-scoped identity that’s both portable and secure, while [federation lets external systems trust](/blog/federating-non-human-identities-with-external-idps-using-id-tokens-in-aws-gcp-and-azure) those identities without embedding secrets. In this post, we focus on another critical capability: **credential injection**. With Riptides, AWS credentials are injected dynamically into outgoing requests — **secretless and just-in-time** — allowing workloads to call services like Amazon Bedrock without ever touching long-lived keys or tokens. In our earlier post, [Why Cloud-Native Federation Isn’t Enough for Non-Human Identities in AWS, GCP, and Azure](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure), we outlined the limitations of relying only on cloud-native federation for non-human identities. One of the biggest challenges is how to provide credentials securely when workloads need to access cloud services. This post takes that discussion further by showing how **on-the-wire credential injection** with Riptides eliminates the need for stored secrets. To make the idea concrete, we’ll walk through a real-world example using the [Amazon Product Agent Review](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/use-case-examples/product-review-agent) sample application. The **Product Agent Review** app, built on **Amazon Bedrock**, uses LLMs to let clients query product reviews. It consists of two parts: - **Server-side**: an Amazon Bedrock agent running inside AWS. (Deployed via the provided [main.ipynb](https://github.com/aws-samples/amazon-bedrock-samples/blob/main/agents-and-function-calling/bedrock-agents/use-case-examples/product-review-agent/main.ipynb) notebook.) - **Client-side**: a Python application designed to run outside AWS using [Streamlit](https://streamlit.io/). (Logic available in [main.py](https://github.com/aws-samples/amazon-bedrock-samples/blob/main/agents-and-function-calling/bedrock-agents/use-case-examples/product-review-agent/main.py).) This client–server separation makes it a perfect case study. With Riptides, credentials are never embedded in the client environment. Instead, Riptides operates at the kernel level: as each HTTP request leaves the client, it transparently **injects short-lived AWS credentials** and re-signs the request with **[libsigv4](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility)**. The client never handles secrets — yet the request still arrives at AWS fully authenticated. > **Note:** Riptides’ [**libsigv4**](https://github.com/riptideslabs/libsigv4) library, which handles AWS SigV4 signing at the kernel level, is fully open-sourced. For a deep dive, see our post: [Introducing libsigv4: AWS SigV4 Signatures in Portable C with Kernel Compatibility](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility). ## The standard way: Authenticating and authorizing the client application Normally, for a client application to invoke an Amazon Bedrock agent, it needs to be wired into AWS IAM. This typically involves: - **An IAM role** with permissions to invoke the deployed *Product Review Agent*. - **A trust relationship**, which defines *who* is allowed to assume this role. That trust can be established in two common ways: - An **IAM user** that assumes the role, using long-lived AWS credentials (Access Key ID / Secret Access Key). - A **federated OIDC identity** that assumes the role, using short-lived credentials obtained after authenticating with an external identity provider (IDP) and retrieving an ID token. ### Creating and assigning a role for the client application To let a client application call the Bedrock agent, you need an IAM role with the right permissions and trust relationships. **Role permissions** (allowing the client to invoke a specific Bedrock agent): ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "bedrock:InvokeAgent", "Resource": [ "arn:aws:bedrock:eu-central-1::agent/", "arn:aws:bedrock:eu-central-1::agent-alias//" ] } ] } ``` **Role trust relationships:** The trust policy defines who can assume this role. Two common patterns are: - **IAM user** (using long-lived credentials): ```json { "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/" }, "Action": "sts:AssumeRole" } ] } ``` - **Federated identity** (via an external OIDC IdP and short-lived credentials). See: [Federating non-human identities with external IdPs using ID tokens in AWS, GCP, and Azure](/blog/federating-non-human-identities-with-external-idps-using-id-tokens-in-aws-gcp-and-azure) ```json { "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { ":sub": "", "oidc:aud": "sts.amazonaws.com" } } } ] } ``` ### Supplying credentials to the client application How the client obtains and uses AWS credentials depends on the SDK it uses. For example, with [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) (the SDK used by this sample app), credentials can be provided via: - Environment variables - AWS config and credentials files - Legacy Boto2 config file - Container credential provider (e.g. on Amazon ECS) - Custom logic that passes credentials directly to the client code ### The problem with storing credentials No matter which authentication method you choose, **credentials are always a liability**. - With **STS AssumeRole**, you need to manage and securely store the AWS Access Key ID and Secret Access Key. - With **federated authentication**, you must securely store not only the credentials used to connect to the external IDP, but also ensure the ID token is refreshed before expiration. On top of that, there’s no guarantee that every workload running in your environment can safely access these credentials. For non-human identities, handling secrets introduces risk, complexity, and potential points of failure. **Running the client application**: For demonstration purposes, we’ll run the `streamlit` client application using the **STS AssumeRole** authentication method, with credentials passed via AWS config file: **~/.aws/credentials**: ```ini [bedrock-pra-user] aws_access_key_id = aws_secret_access_key = ``` **~/.aws/config**: ```ini [profile bedrock-pra-user] region = eu-central-1 source_profile = bedrock-pra-user role_arn= ``` ```shell AWS_PROFILE=bedrock-pra-user streamlit run main.py -- --id --alias ``` By default, the Streamlit web interface is available at , where users can interact with the client application: ![Client application user chat](../../assets/credential-injection-aws/streamlit_std.png) As shown in the screenshot, the client successfully invoked the **Product Review Agent** with the input "Give me the last 2 reviews" and received a response from the agent, demonstrating that the setup works with standard AWS credentials. ## On-the-wire credential injection with Riptides Now let’s see how this works when the client application runs in a **Riptides managed environment**. Riptides solves the problem of stored secrets by injecting credentials **dynamically at runtime**, so the client never has to store or manage AWS credentials directly. ### Prerequisites #### Register Riptides Control Plane as an external IdP in AWS First, we need to register the Riptides Control Plane as an **OIDC identity provider** in AWS. This allows AWS to trust identity tokens issued by Riptides and exchange them for temporary credentials via STS: ```shell aws iam create-open-id-connect-provider \ --url "https://enjoyed-previously-llama.ngrok-free.app/oidc" \ --client-id-list "sts.amazonaws.com" ``` **Creating and assigning a role for the client application**: To allow the client application to call the Bedrock agent, we need an IAM role with the appropriate permissions and a trust relationship. **Role permissions** (allowing the client to invoke a specific Bedrock agent): *arn:aws:iam:::role/bedrock-pra-user-role-wif:* ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "bedrock:InvokeAgent", "Resource": [ "arn:aws:bedrock:eu-central-1::agent/", "arn:aws:bedrock:eu-central-1::agent-alias//" ] } ] } ``` **Role trust relationships:** The trust relationship must allow ID tokens from Riptides Control Plane to be exchanged for AWS temporary credentials, which will assume this role: ```json { "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/enjoyed-previously-llama.ngrok-free.app/oidc" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "enjoyed-previously-llama.ngrok-free.app/oidc:aud": "sts.amazonaws.com", "enjoyed-previously-llama.ngrok-free.app/oidc:sub": "spiffe://acme.org/streamlit" } } } ] } ``` Here, the *sub* field specifies the identity that Riptides will assign to the client application ### Setting up the client application with Riptides From Riptides’ perspective, the **Product Review Agent** is an external service that it doesn’t know about by default. To make Riptides aware of it, you need to register the service with the Riptides Control Plane. This is done by creating a Kubernetes custom resource like the following: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: Service metadata: name: bedrock namespace: riptides-system spec: addresses: - address: bedrock-agent-runtime.eu-central-1.amazonaws.com # AWS Bedrock service endpoint port: 443 # Port the client connects to labels: app: bedrock-pra # Label for matching this service external: true # Indicates this is an external service, not managed by Riptides ``` The **address** and **port** must match the AWS Bedrock Agent Runtime service endpoint, since the client application connects to the desired Bedrock agent via this AWS service. ### Defining How to Obtain Workload Credentials We define how the client workload obtains temporary AWS credentials using Riptides. ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialSource metadata: name: aws-cred-1 namespace: riptides-system spec: aws: # Temporary credentials sourced from AWS roleArn: arn:aws:iam:::role/bedrock-pra-user-role-wif # The IAM role to assume for temporary credentials audience: [] # Optional, defaults to sts.amazonaws.com. Must match the OIDC provider settings in AWS lifetime: "900s" # Optional, requested lifetime for AWS temporary credentials idTokenClaims: [] # Optional, additional claims to include in the ID token besides "sub" idTokenLifetime: # Optional, lifetime of ID tokens issued by Riptides apiVersion: core.riptides.io/v1alpha1 kind: WorkloadCredential metadata: name: streamlit-aws-cred-1 namespace: riptides-system spec: credentialSource: aws-cred-1 # Source of temporary credentials workloadID: streamlit # Workload ID to get AWS temporary credentials for ``` **How it works**: 1. **CredentialSource** defines how Riptides will obtain temporary AWS credentials for a workload. 1. **WorkloadCredential** links a specific workload ID (here, *streamlit*) to a credential source. Riptides Control Plane issues an ID token for the workload. The *sub* claim in the token follows this format: *spiffe:///*. In our demo, the trust domain is **acme.org**, so the *sub* claim becomes: **spiffe://acme.org/streamlit**. This matches the *enjoyed-previously-llama.ngrok-free.app/oidc:sub* we configured in AWS for the role trust relationship. The ID token is then exchanged via AWS STS for temporary credentials, which assume the specified role (*bedrock-pra-user-role-wif*) and allow the workload to invoke the Bedrock agent, all **without storing long-lived credentials.** The Control Plane automatically refreshes these temporary credentials when they expire, ensuring uninterrupted access for the workload. ### Assigning Workload IDs to processes We define **which processes can be assigned the `streamlit` workload ID, and under what conditions**: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: streamlit-id namespace: riptides-system spec: scope: agent: id: # Scope: node(s) where this workload identity can be assigned workloadID: streamlit # The workload ID assigned to matching processes selectors: - process:name: [streamlit] # Runtime process attribute that must match to get this identity connection: tls: mode: PERMISSIVE # Allows access to Streamlit's web interface on localhost via HTTP egress: # Egress rules for credential injection - selectors: - app: bedrock-pra # Service endpoint(s) targeted by this rule credentialName: streamlit-aws-cred-1 # Credentials to inject, referenced from WorkloadCredential connection: tls: intercept: true # Intercept traffic and inject credentials into HTTP requests ``` **How it works**: 1. **Riptides Agent** runs on nodes and acts as the bridge between the Control Plane and the Linux kernel module. 1. Workload IDs and credentials issued by the Control Plane are **restricted to processes on the node where the agent runs** — this is controlled by the *scope* field in the *WorkloadIdentity* CR. Multiple agents can also be targeted if needed. 1. The **Linux kernel module** monitors running processes and checks their runtime attributes against the *spec.selectors* values. Only matching processes are assigned the workload ID. 1. When a process with an assigned workload ID sends a request to a service referenced in the *egress* rules, **the temporary credentials from the specified WorkloadCredential are injected directly into the request.** In this example, any process named *streamlit* will receive **AWS temporary credentials** automatically when sending requests to the Amazon Bedrock Agent Runtime service endpoint. Note: A single workload can be assigned multiple credentials if it interacts with multiple services — for example, both AWS and GCP at the same time. ### AWS specific details: SigV4 signing Since this example uses AWS, on-the-wire credential injection requires **resigning HTTP requests** with the correct AWS SigV4 signature. Riptides accomplishes this using our [libsigv4](https://github.com/riptideslabs/libsigv4) library, implemented in portable C with Linux kernel compatibility. This enables credentials to be injected **and signed at the kernel level** just before the request is sent, ensuring full AWS authentication **without exposing keys to the client application**. For a deeper dive, see our previous post: [Introducing libsigv4: AWS SigV4 Signatures in Portable C with Kernel Compatibility](/blog/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility). **Running the client application**: For demonstration purposes, we’ll run the *streamlit* client application with placeholder AWS credentials: ```shell AWS_DEFAULT_REGION=eu-central-1 AWS_ACCESS_KEY_ID=none AWS_SECRET_ACCESS_KEY=none AWS_CA_BUNDLE=/sys/kernel/riptides/ca-certificates.crt streamlit run main.py -- --id --alias ``` Here, we intentionally provide an invalid AWS Access Key ID and Secret Access Key. *Riptides will automatically inject valid temporary credentials* into the requests sent by the client application. Now let’s ask the Product Review Agent: *"What are your capabilities?"* ![Client application user chat](../../assets/credential-injection-aws/streamlit_std.png) The client never sees real credentials, yet the request is fully authenticated and processed. ### Key points - No credentials are stored on the client machine or in configuration files. - Credentials are provided **just-in-time** for each request, minimizing the risk of leaks. - Requests are automatically signed or authorized according to the target service’s requirements. - The workflow of invoking external services remains unchanged from the client’s perspective. ## Benefits of on-the-wire credential injection Using Riptides provides several clear advantages over traditional approaches: | Traditional approach | Riptides approach | | ----------------------------------- |------------------------------------------- | | Store service credentials locally | No credentials stored on the client | | Rotate and secure secrets manually | Credentials injected dynamically | | Risk of credential leakage | Reduced attack surface | | Operational overhead | Minimal, automated | - Improved security posture for non-human clients. - Reduced operational complexity. - Works transparently with existing applications and cloud or service providers. ## Final thoughts At Riptides, we’re strong believers in **[SPIFFE-based workload identities](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust)**, enabling fine-grained, zero-touch authentication with secure, ephemeral credentials and no secrets ever stored. This is the most robust and future-proof way to handle non-human identities, and it works seamlessly across multi-cloud environments. But where [SPIFFE adoption](/blog/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure) isn’t possible, Riptides provides a fallback: on-the-wire credential injection, delivering the right cloud or service credentials directly into the request stream without ever touching the application. **On-the-wire credential injection** with Riptides enables secure, secretless invocation of external services. In this example, we demonstrated the approach with an AWS Bedrock agent, but Riptides provides the same seamless support across all major cloud providers — AWS, GCP, and Azure. Beyond cloud credentials, Riptides can also inject **OAuth2 tokens**, whether sourced from Kubernetes secrets or obtained via OAuth2 authorization code flows, directly into client requests. This pattern can be extended to agents, applications, and services in any environment, offering a unified, secure approach for managing non-human identities at scale. In upcoming posts, we’ll show how the same pattern applies to GCP and Azure services — making secretless, on-the-wire identity truly multi-cloud. --- ## Introducing libsigv4: AWS SigV4 Signatures in Portable C with Kernel Compatibility - URL: https://blog.riptides.io/introducing-libsigv4-aws-sigv4-signatures-in-portable-c-with-kernel-compatibility - Published: 2025-08-25 - Author: Nandor Kracser - Category: Kernel - Tags: kernel, aws When working with AWS Service APIs, like S3 from constrained or non-standard environments, generating SigV4 signatures is unavoidable. AWS provides an [official SigV4 for AWS IoT embedded SDK](https://github.com/aws/SigV4-for-AWS-IoT-embedded-sdk), but as we integrated it into our projects, we quickly ran into challenges: - **System header portability**: The official library assumes certain C library functions and headers will be available, which isn’t always true in kernel space or other restricted C environments. - **Redundant parsing**: We already parse HTTP headers in our stack, but the AWS SDK parses them again internally, leading to inefficiency. - **Flexibility**: We needed a library that could fit into both embedded systems *and* kernel-level code with tight integration requirements. That’s why we built [`libsigv4`](https://github.com/riptideslabs/libsigv4). ## What is AWS SigV4? [AWS Signature Version 4 (SigV4)](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) is the authentication protocol used by AWS services. Instead of sending credentials directly in a request, SigV4 computes a **cryptographic signature** over the request details, ensuring authentication, integrity, and time-bounded validity. At a high level, SigV4 works like this: 1. Collect request data (method, path, query string, headers, payload hash). 2. Create a *canonical request*: normalized string form. 3. Derive a signing key using HMAC-SHA256 and your AWS secret key. 4. Compute the final signature. 5. Attach the signature to the request (as an `Authorization` header or presigned URL). Whether you’re in userspace or kernel space, you need these steps to securely talk to AWS. ## Why Another SigV4 Library? ### Our Use Case In our setup, applications make ordinary HTTP requests without being aware of AWS SigV4. Underneath, in kernel space, we intercept and parse these designated requests. Using libsigv4, we then inject the required AWS authorization headers before forwarding the requests upstream. This approach allows applications to remain completely unchanged — they don’t need to know anything about AWS authentication or SigV4 signing. All of the complexity of parsing, signing, and securely managing credentials is handled transparently in the kernel. ### Current C SigV4 Landscape There are already several implementations of SigV4 in C: - The [AWS official embedded SDK](https://github.com/aws/SigV4-for-AWS-IoT-embedded-sdk) - [sidbai/aws-sigv4-c](https://github.com/sidbai/aws-sigv4-c), a lightweight but abandoned version - [libcurl’s SigV4 API](https://curl.se/libcurl/c/libcurl-sigv4.html) for userspace networking - Inspiration also came from [nanopb](https://jpa.kapsi.fi/nanopb/), which shows how C libraries can be minimal and portable But none of these struck the right balance for **kernel compatibility, portability, and avoiding duplicate parsing**. ## Design Goals of libsigv4 1. **Zero dynamic allocations** - Works on user-provided buffers. - Critical for kernel code or environments where `malloc` is unavailable or undesirable. 2. **Pluggable crypto backends** - Compatible with OpenSSL, mbedTLS, TinyCrypt, or even the **Linux Kernel Crypto API**. - Choose the right backend depending on your environment. 3. **System header portability** - Minimal reliance on libc. - Compiles cleanly in restricted header environments (including kernel space). 4. **No duplicate parsing** - Integrates with your existing HTTP request parsing. - Doesn’t re-parse headers you already parsed. ## Example: Signing a Simple Request Here’s a minimal example of using `libsigv4` with OpenSSL to sign a GET request to an AWS S3 endpoint: ```c #include #include #include #include "sigv4.h" int HMAC_SHA256(const unsigned char *data, size_t data_len, const unsigned char *key, size_t key_len, unsigned char *out, size_t *out_len) { unsigned int len = 0; char *ac = HMAC(EVP_sha256(), key, key_len, data, data_len, out, &len); *out_len = len; return (ac != NULL) ? 0 : -1; } int main() { aws_sigv4_params_t sigv4_params = { .access_key_id = aws_sigv4_string("your_access_key"), .secret_access_key = aws_sigv4_string("your_secret_key"), .method = aws_sigv4_string("GET"), .uri = aws_sigv4_string("/"), .query_str = aws_sigv4_string("encoding-type=url"), .host = aws_sigv4_string("riptides-sigv4.s3.eu-central-1.amazonaws.com"), .region = aws_sigv4_string("eu-central-1"), .service = aws_sigv4_string("s3"), .x_amz_date = aws_sigv4_string("20250815T071550Z"), .hmac_sha256 = HMAC_SHA256, .sha256 = (void *)SHA256, .sort = qsort, }; char auth_buf[AWS_SIGV4_AUTH_HEADER_MAX_LEN] = {0}; aws_sigv4_header_t auth_header = { .value = aws_sigv4_string(auth_buf)}; int status = aws_sigv4_sign(&sigv4_params, &auth_header); if (status == AWS_SIGV4_OK) printf("Signature: %s\\n", auth_header.value.data); else printf("Failed to sign request, status: %d\\n", status); return 0; } ``` This produces an `Authorization` header you can attach directly to your HTTP request. Because everything works on user-provided buffers, you control exactly how much memory is used. ## Get Started You can check out the source here: 👉 [https://github.com/riptideslabs/libsigv4](https://github.com/riptideslabs/libsigv4) We’d love feedback, especially from embedded developers running into the same challenges. ## Closing Thoughts For many IoT and embedded projects, **the hardest part isn’t AWS itself, but fitting AWS’s tools into constrained environments**. With `libsigv4`, we wanted to remove just enough friction to let developers stay focused on their applications. If you’re working on SigV4 in embedded C, give it a try - and let us know what environments you’re using it in. --- ## Practical Linux Kernel Debugging: From pr_debug() to KASAN/KFENCE - URL: https://blog.riptides.io/practical-linux-kernel-debugging-from-pr-debug-to-kasan-kfence - Published: 2025-08-18 - Author: Nandor Kracser - Category: Kernel - Tags: debug, kernel, linux Debugging kernel-space memory bugs is one of the most challenging tasks in systems programming. Fortunately, the Linux kernel comes with a rich toolbox of debugging features that can detect memory errors, race conditions, and invalid accesses - long before they cause a crash. At Riptides, the riptides-driver kernel module plays a central role in our architecture, enabling zero-trust communication and SPIFFE-based process identity. Because of its critical role, the module must be rock solid and free of subtle bugs. In this post, we’re sharing the debugging toolset we use to keep it reliable — and how you can apply the same techniques to your own kernel modules. In this post, we’ll take a beginner-friendly look at some of the most useful tools built into the Linux kernel for debugging memory issues and concurrency problems. This isn’t meant to be an exhaustive or deeply technical guide, but rather a practical overview that covers the essential, everyday debugging features commonly used by kernel developers. ## dyndbg: Dynamic Debug Printing For decades, `printk()` (much like `printf()` in user space) has been the go-to debugging tool for kernel developers. It's simple, effective, and universally understood - the first thing many developers reach for when troubleshooting. **Benefits of `printk()`:** - Works anywhere in kernel space - No extra tooling needed - Easy to trace logic and variable states **Downsides:** - Too much printk() output can flood logs and slow the system - Can't be turned off at runtime - Requires recompilation if you want to add or remove debug output The [dynamic debug framework](https://docs.kernel.org/admin-guide/dynamic-debug-howto.html) (dyndbg) is a smarter version of `printk()` via `pr_debug()`, allowing you to enable or disable specific debug messages at runtime - without changing your code or rebooting. ### How to Use 1. Insert `pr_debug("debug msg");` into your module. 2. Load your module. 3. Enable debug prints like this: ```bash echo 'module your_module +p' > /sys/kernel/debug/dynamic_debug/control ``` You can also use wildcards, function names, or file paths for fine-grained control. Useful for live debugging with minimal impact and easy filtering. ## 🔍 Kernel Debugging with a debug kernel Kernel bugs are harder to find and reproduce than user-space bugs. Stack traces are not guaranteed, no graceful exceptions - just a frozen system or a mysterious crash. That’s why we need instrumentation: features baked into the kernel that catch misbehavior before the system breaks. For this guide, we'll be using [Rocky Linux](https://rockylinux.org), which conveniently provides pre-built debug kernel packages that are easy to install and use. Before getting started, we need to install the debug version of the kernel and its matching headers package to enable and build against the debugging features: ```bash sudo dnf install kernel-debug kernel-debug-devel # Check the current debug kernel version: CURRENT_DEBUG_KERNEL=$(ls -1 /boot/vmlinuz-*+debug | head -1) # Set it as the default at boot: sudo grubby --set-default ${CURRENT_DEBUG_KERNEL} # Enable debug features at boot time, see details later: sudo grubby --update-kernel=${CURRENT_DEBUG_KERNEL} --args "kmemleak=on kasan=on kasan.multi_shot=1" # We need to reboot into the debug kernel now sudo reboot ``` Then boot into it. These kernels enable key debug configs like `CONFIG_KASAN`, `CONFIG_KFENCE`, and `CONFIG_LOCKDEP` that are compiled into this debug kernel. ## KASAN: Kernel Address Sanitizer **KASAN** (Kernel Address Sanitizer) is one of the most powerful memory debugging tools available in the Linux kernel. It works by maintaining a "shadow map" - a parallel memory region that tracks the state of every byte in kernel memory. For every 8 bytes of kernel memory, KASAN uses 1 byte of shadow memory to record whether that memory is valid, poisoned, or freed. When your kernel module performs any memory access (read or write), KASAN automatically checks the shadow map to verify that the access is legitimate. This happens transparently without any changes to your code. If KASAN detects an invalid access, it immediately triggers a kernel panic with a detailed report showing exactly what went wrong, where it happened, and the complete call stack that led to the violation. KASAN excels at catching three critical classes of memory bugs: **use-after-free** errors (accessing memory that has already been freed), **buffer overflows** (writing past the end of an allocated region), and **invalid pointer dereferences** (accessing uninitialized or corrupted pointers). These bugs are notoriously difficult to debug because they often don't cause immediate crashes - instead, they silently corrupt memory and cause mysterious failures much later. The shadow memory approach makes KASAN extremely thorough but also relatively expensive in terms of memory overhead (roughly 12.5% additional RAM usage). This makes it ideal for development and testing environments where you want to catch every possible memory error, but less suitable for production systems. Enable KASAN at boot time: ```bash kasan=1 ``` When KASAN detects a violation, detailed output will immediately appear in `dmesg` with stack traces showing both the violating code and the allocation/deallocation history. This makes it invaluable during fuzzing, stress testing, or any scenario where you're trying to trigger edge cases in your kernel module. ### Example: KASAN Catching an Out-of-Bounds Write Here's a simple example of buggy kernel module code that KASAN would immediately catch: ```c static void __out_of_bounds(int num_bytes) { char *buffer = kmalloc(num_bytes, GFP_KERNEL); // This writes past the end of our num_bytes-byte allocation! strcpy(buffer, "This string is definitely longer than %d bytes", num_bytes); kfree(buffer); } ``` Without KASAN, this bug might silently corrupt adjacent memory and cause mysterious crashes elsewhere in the kernel. With KASAN enabled, you'd immediately see output like this in `dmesg`: ```log [Tue Aug 5 12:00:30 2025] ================================================================== [Tue Aug 5 12:00:30 2025] BUG: KASAN: slab-out-of-bounds in debug_zoo_init+0x224/0x1000 [debug_zoo] [Tue Aug 5 12:00:30 2025] Write of size 47 at addr ffff0000e5f5cc40 by task insmod/5228 [Tue Aug 5 12:00:30 2025] CPU: 2 PID: 5228 Comm: insmod Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+debug #1 [Tue Aug 5 12:00:30 2025] Hardware name: Apple Inc. Apple Virtualization Generic Platform, BIOS 2075.120.2.0.0 04/18/2025 [Tue Aug 5 12:00:30 2025] Call trace: [Tue Aug 5 12:00:30 2025] dump_backtrace+0xac/0x130 [Tue Aug 5 12:00:30 2025] show_stack+0x1c/0x30 [Tue Aug 5 12:00:30 2025] dump_stack_lvl+0xac/0xe8 [Tue Aug 5 12:00:30 2025] print_address_description.constprop.0+0x84/0x2e8 [Tue Aug 5 12:00:30 2025] print_report+0x100/0x1e4 [Tue Aug 5 12:00:30 2025] kasan_report+0x80/0xbc [Tue Aug 5 12:00:30 2025] kasan_check_range+0xe4/0x190 [Tue Aug 5 12:00:30 2025] memcpy+0x54/0x90 [Tue Aug 5 12:00:30 2025] debug_zoo_init+0x224/0x1000 [debug_zoo] ``` This detailed report shows exactly where the violation occurred, what type of access it was, and the complete allocation history - making the bug trivial to fix. ![outbound](../../assets/linux-kernel-debug/out-of-bound.png) ## KFENCE: Kernel Electric Fence **KFENCE** (Kernel Electric Fence) takes a fundamentally different approach to memory debugging compared to KASAN. While KASAN checks every single memory access with significant overhead, KFENCE uses a sampling-based strategy that makes it suitable even for production environments. The core idea behind KFENCE is elegantly simple: instead of tracking all allocations, it randomly selects a small percentage of `kmalloc()` calls and places each selected allocation on its own dedicated memory page, surrounded by **guard pages**. These guard pages are marked as non-accessible in the page tables, so any attempt to read or write beyond the allocated object immediately triggers a page fault. When a page fault occurs on a guard page, KFENCE knows exactly what happened: either an out-of-bounds access (reading/writing past the end of the allocation) or a use-after-free (accessing memory that was previously freed). The page fault handler generates a detailed report showing the violating instruction, call stack, and allocation history. This sampling approach means KFENCE has very low overhead (typically less than 1% performance impact) since it only tracks a tiny fraction of allocations. However, it's probabilistic - it might miss bugs that happen in allocations that weren't selected for monitoring. The trade-off is worth it for production systems where you need some level of memory error detection without the heavy overhead of KASAN. KFENCE is particularly effective at catching temporal bugs (use-after-free) because it never actually returns freed memory to the allocator - instead, it keeps the guard pages in place indefinitely, ensuring that any future access to freed KFENCE-monitored memory will immediately fault. Enable KFENCE with a sampling interval in milliseconds (lower numbers = more frequent sampling): ```bash kfence.sample_interval=100 ``` **Note:** Unlike KASAN which requires a debug kernel, KFENCE is available in most standard kernel builds (including production kernels) since it has minimal overhead. You can enable it on your normal kernel without needing to install `kernel-debug`: ```bash # For a standard kernel, just add KFENCE parameters: sudo grubby --update-kernel=ALL --args "kfence.sample_interval=100" sudo reboot ``` This makes KFENCE particularly valuable for production debugging scenarios where installing a debug kernel isn't practical. ### Example: KFENCE Catching a Use-After-Free Here's an example of a use-after-free bug that KFENCE would catch: ```c static void __use_after_free(void) { char *leak = kzalloc(100, GFP_KERNEL); if (!leak) { printk(KERN_ERR "Memory allocation failed\n"); return; } strcpy(leak, "This is a use-after-free example"); kfree(leak); // Using the pointer after freeing it, which is undefined behavior printk(KERN_INFO "Using freed memory: %s\n", leak); // This is dangerous! } ``` If this allocation was selected by KFENCE's sampling, you'd see output like: ```log [Tue Aug 5 15:58:32 2025] ================================================================== [Tue Aug 5 15:58:32 2025] BUG: KFENCE: use-after-free read in string+0x50/0x100 [Tue Aug 5 15:58:32 2025] Use-after-free read at 0x00000000383f6e38 (in kfence-#121): [Tue Aug 5 15:58:32 2025] string+0x50/0x100 [Tue Aug 5 15:58:32 2025] vsnprintf+0x190/0x780 [Tue Aug 5 15:58:32 2025] vprintk_store+0xfc/0x4d0 [Tue Aug 5 15:58:32 2025] vprintk_emit+0x11c/0x3cc [Tue Aug 5 15:58:32 2025] vprintk_default+0x3c/0x44 [Tue Aug 5 15:58:32 2025] vprintk+0xc8/0x110 [Tue Aug 5 15:58:32 2025] _printk+0x64/0x8c [Tue Aug 5 15:58:32 2025] debug_zoo_init+0x128/0x1000 [debug_zoo] [Tue Aug 5 15:58:32 2025] do_one_initcall+0x4c/0x2e0 [Tue Aug 5 15:58:32 2025] do_init_module+0x5c/0x220 ... [Tue Aug 5 15:58:32 2025] kfence-#121: 0x00000000383f6e38-0x00000000c727eacc, size=100, cache=kmalloc-128 [Tue Aug 5 15:58:32 2025] allocated by task 5364 on cpu 0 at 51.206043s: [Tue Aug 5 15:58:32 2025] kmalloc_trace+0x228/0x270 [Tue Aug 5 15:58:32 2025] debug_zoo_init+0xe8/0x1000 [debug_zoo] [Tue Aug 5 15:58:32 2025] do_one_initcall+0x4c/0x2e0 [Tue Aug 5 15:58:32 2025] do_init_module+0x5c/0x220 ... [Tue Aug 5 15:58:32 2025] freed by task 5364 on cpu 0 at 51.206047s: [Tue Aug 5 15:58:32 2025] debug_zoo_init+0x11c/0x1000 [debug_zoo] [Tue Aug 5 15:58:32 2025] do_one_initcall+0x4c/0x2e0 [Tue Aug 5 15:58:32 2025] do_init_module+0x5c/0x220 ... [Tue Aug 5 15:58:32 2025] CPU: 0 PID: 5364 Comm: insmod Kdump: loaded Tainted: G B OE ------- --- 5.14.0-570.30.1.el9_6.aarch64 #1 [Tue Aug 5 15:58:32 2025] Hardware name: Apple Inc. Apple Virtualization Generic Platform, BIOS 2075.120.2.0.0 04/18/2025 [Tue Aug 5 15:58:32 2025] ================================================================== ``` Use KFENCE in long-running systems where reproducibility is hard but correctness is critical, or in production environments where you want some memory error detection without KASAN's overhead. ## kmemleak: Memory Leak Detector **kmemleak** is the kernel's equivalent of `valgrind` for user-space memory leak detection. It works by tracking all kernel memory allocations (via `kmalloc()`, `kzalloc()`, `vmalloc()`, etc.) and periodically scanning all kernel memory to find allocations that are no longer reachable from any pointer. The core principle is simple but powerful: kmemleak maintains a database of all active allocations along with their call stacks. During a scan, it treats all kernel data structures, CPU registers, and stack memory as potential "roots" and performs a mark-and-sweep garbage collection algorithm. Any allocation that can't be reached by following pointer chains from these roots is considered a leaked object. What makes kmemleak particularly valuable is that it catches **logical leaks** - memory that's technically still allocated but no longer accessible by your code because you've lost all references to it. These bugs are often subtle: a function returns early due to an error condition, forgetting to free memory it allocated, or a data structure is partially cleaned up but leaves some allocations dangling. Unlike KASAN and KFENCE which detect violations immediately when they occur, kmemleak is **passive** - it only reports leaks when you explicitly trigger a scan. This makes it ideal for regression testing and periodic health checks of long-running systems. kmemleak tracks several important pieces of information for each allocation: - **Size and address** of the allocated memory - **Complete call stack** showing where the allocation occurred - **Timestamp** of when the allocation was made - **References** from other kernel objects Enable kmemleak at boot time: ```bash kmemleak=on ``` Then trigger scans manually to check for leaks: ```bash echo scan > /sys/kernel/debug/kmemleak cat /sys/kernel/debug/kmemleak # you can use tail -f as well for continous observing ``` ### Example: kmemleak Catching a Memory Leak Here's an example of a memory leak that kmemleak would detect: ```c static void __leak(void) { char *leak = kmalloc(100, GFP_KERNEL); if (!leak) { printk(KERN_ERR "Memory allocation failed\n"); return; } strcpy(leak, "This is a memory leak example"); printk(KERN_INFO "Memory leak example: allocated %p and now leaking\n", leak); } ``` After loading this module and triggering a kmemleak scan, you'd see in the `dmesg` logs: ``` [Wed Aug 6 10:26:26 2025] Here comes the Debug Zoo! [Wed Aug 6 10:26:26 2025] Use 'dmesg' to see the debug messages. [Wed Aug 6 10:26:26 2025] Memory leak example: allocated 00000000bd22923f and now leaking [Wed Aug 6 10:28:56 2025] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) ``` And in `/sys/kernel/debug/kmemleak` you will find something like: ```log unreferenced object 0xffff00012968a800 (size 128): comm "insmod", pid 2287, jiffies 4294954674 hex dump (first 32 bytes): 54 68 69 73 20 69 73 20 61 20 6d 65 6d 6f 72 79 This is a memory 20 6c 65 61 6b 20 65 78 61 6d 70 6c 65 00 00 00 leak example... backtrace (crc 54832baf): kmemleak_alloc+0xb4/0xc4 kmalloc_trace+0x268/0x340 debug_zoo_init+0x11c/0x1000 do_one_initcall+0x178/0xad0 do_init_module+0x1dc/0x660 load_module+0x1034/0x1600 ``` This report shows the leaked object's address, size, age, and complete allocation call stack, making it easy to track down where the leak originated. **Pro tip:** To improve stack trace quality and make debugging easier, compile your modules with debug information: ```make ccflags-y += -g -fno-omit-frame-pointer -fno-optimize-sibling-calls ``` kmemleak is particularly useful for: - **Regression testing** - run scans before and after code changes - **Long-running system health checks** - periodic scans on development systems - **Error path validation** - ensuring cleanup code properly handles all allocations ## Lockdep: Deadlock & Lock Order Validator **Lockdep** is the kernel's sophisticated deadlock detection and lock validation system. It's designed to catch one of the most insidious classes of kernel bugs: **deadlocks** and **lock ordering violations** that can cause the entire system to freeze. The fundamental problem Lockdep solves is detecting potential deadlocks before they actually occur. A classic deadlock scenario happens when: 1. Thread A holds lock X and tries to acquire lock Y 2. Thread B holds lock Y and tries to acquire lock X 3. Both threads wait forever for each other But Lockdep goes beyond just detecting active deadlocks - it builds a **dependency graph** of all lock acquisitions throughout the system's runtime and analyzes this graph to detect scenarios that *could* lead to deadlocks, even if they haven't happened yet. **How Lockdep Works:** Lockdep monitors every `spin_lock()`, `mutex_lock()`, `read_lock()`, etc. call and builds a directed graph of lock dependencies. If thread A acquires lock X then lock Y, Lockdep records the dependency X → Y. Later, if it sees another code path trying to acquire Y → X, it immediately flags this as a potential deadlock because these two paths could execute simultaneously and deadlock. What makes Lockdep particularly powerful is its ability to catch **lock inversion bugs** across completely different code paths that might rarely execute at the same time. A bug might exist for years without manifesting until exactly the right timing conditions occur. Lockdep also validates other locking correctness properties: - **Lock class consistency** - ensuring the same lock is always used with the same semantic rules - **IRQ safety** - detecting when code that can run in interrupt context tries to acquire locks held in non-interrupt context - **Recursive locking** - catching attempts to acquire the same lock twice **Performance Impact:** Lockdep has significant overhead (~20-30% performance impact) and uses substantial memory to track the dependency graph, so it's only enabled in debug kernels and should never be used in production. Enable Lockdep at boot time: ```bash lockdep=1 ``` Alternatively, if you're already running a debug kernel, Lockdep is usually enabled by default through the `CONFIG_PROVE_LOCKING` kernel configuration option. ### Example: Lockdep Catching a Lock Ordering Bug Here's an example of problematic locking that Lockdep would catch: ```c static struct task_struct *lock_thread1; static struct task_struct *lock_thread2; static DEFINE_MUTEX(lock_a); static DEFINE_MUTEX(lock_b); static int lock_thread_fn1(void *data) { mutex_lock(&lock_a); msleep(100); mutex_lock(&lock_b); // A -> B msleep(100); mutex_unlock(&lock_b); mutex_unlock(&lock_a); return 0; } static int lock_thread_fn2(void *data) { msleep(50); // Ensure interleaving mutex_lock(&lock_b); msleep(100); mutex_lock(&lock_a); // B -> A -> should trigger lockdep mutex_unlock(&lock_a); mutex_unlock(&lock_b); return 0; } static void __lockdep(void) { printk(KERN_INFO "lockdep test loaded\n"); lock_thread1 = kthread_run(lock_thread_fn1, NULL, "lock_thread1"); lock_thread2 = kthread_run(lock_thread_fn2, NULL, "lock_thread2"); } ``` When Lockdep detects this lock ordering violation, you'd see detailed output in `dmesg`: ```log [Wed Aug 6 10:55:13 2025] Debug Zoo module unloaded [Wed Aug 6 10:55:17 2025] Here comes the Debug Zoo! [Wed Aug 6 10:55:17 2025] Use 'dmesg' to see the debug messages. [Wed Aug 6 10:55:17 2025] lockdep test loaded [Wed Aug 6 10:55:17 2025] ====================================================== [Wed Aug 6 10:55:17 2025] WARNING: possible circular locking dependency detected [Wed Aug 6 10:55:17 2025] 5.14.0-570.30.1.el9_6.aarch64+debug #1 Tainted: G OE ------- --- [Wed Aug 6 10:55:17 2025] ------------------------------------------------------ [Wed Aug 6 10:55:17 2025] lock_thread2/3005 is trying to acquire lock: [Wed Aug 6 10:55:17 2025] ffff8000096e3168 (lock_a){+.+.}-{3:3}, at: lock_thread_fn2+0x48/0x70 [debug_zoo] [Wed Aug 6 10:55:17 2025] but task is already holding lock: [Wed Aug 6 10:55:17 2025] ffff8000096e30a8 (lock_b){+.+.}-{3:3}, at: lock_thread_fn2+0x30/0x70 [debug_zoo] [Wed Aug 6 10:55:17 2025] which lock already depends on the new lock. other info that might help us debug this: [Wed Aug 6 10:55:17 2025] Possible unsafe locking scenario: [Wed Aug 6 10:55:17 2025] CPU0 CPU1 [Wed Aug 6 10:55:17 2025] ---- ---- [Wed Aug 6 10:55:17 2025] lock(lock_b); [Wed Aug 6 10:55:17 2025] lock(lock_a); [Wed Aug 6 10:55:17 2025] lock(lock_b); [Wed Aug 6 10:55:17 2025] lock(lock_a); [Wed Aug 6 10:55:17 2025] *** DEADLOCK *** [Wed Aug 6 10:55:17 2025] 1 lock held by lock_thread2/3005: [Wed Aug 6 10:55:17 2025] #0: ffff8000096e30a8 (lock_b){+.+.}-{3:3}, at: lock_thread_fn2+0x30/0x70 [debug_zoo] ``` **When to use Lockdep:** - During development and testing of any code that uses kernel locks - When debugging mysterious hangs or performance issues - For validating complex locking hierarchies in subsystems - Before submitting kernel patches that modify locking behavior **Pro tip:** Lockdep catches bugs that might take months or years to manifest in real systems. Always test your kernel modules with Lockdep enabled during development. ## Testing Your Module (The Right Way) **Our approach at Riptides:** We run a layered debugging strategy for the `riptides-driver` module. During active development, we test on debug kernels with the full suite enabled (KASAN + kmemleak + Lockdep) to catch every possible issue. For daily development on normal kernels, we keep KFENCE enabled as a lightweight background monitor, but skip Lockdep due to its performance overhead. This ensures that memory corruption bugs are caught immediately during code changes, while locking violations are caught during our periodic debug kernel testing sessions. We compile and run our test cases in CI on debug kernels as well and monitor `kmemleak` and `dmesg` for errors. It’s tempting to write a module, load it, see no crash, and assume it's fine. Don’t. - Test **unhappy paths**: failed allocations, edge values, misuse. - Intentionally leak memory or trigger races to ensure your module behaves. - Avoid relying only on printk debugging; use tracepoints and structured logging. Robust modules survive chaos - not just ideal conditions, don’t forget to test beyond the happy path. All the examples are available on [GitHub](https://github.com/bonifaido/kernel-debug-zoo). **Resources** - [KASAN docs](https://www.kernel.org/doc/html/latest/dev-tools/kasan.html) - [KFENCE docs](https://www.kernel.org/doc/html/latest/dev-tools/kfence.html) - [Lockdep guide](https://www.kernel.org/doc/Documentation/locking/lockdep-design.txt) - [kmemleak guide](https://www.kernel.org/doc/html/latest/dev-tools/kmemleak.html) - [Dynamic debug](https://www.kernel.org/doc/html/latest/admin-guide/dynamic-debug-howto.html) - [Debug Zoo module](https://github.com/bonifaido/kernel-debug-zoo) --- ## Why Cloud-Native Federation Isn’t Enough for Non-Human Identities in AWS, GCP, and Azure - URL: https://blog.riptides.io/why-cloud-native-federation-isnt-enough-for-non-human-identities-in-aws-gcp-and-azure - Published: 2025-08-11 - Author: Sebastian Toader - Category: Federation - Tags: federation, non-human identity, aws, gcp, azure ## 1. Quick recap: What we have today Before we delve into the core issues, let’s briefly review the landscape of non-human identity (NHI) federation among the big three cloud providers. We’ve previously covered how external identity federation using ID tokens works in AWS, GCP, and Azure in this [blog post](/blog/federating-non-human-identities-with-external-idps-using-id-tokens-in-aws-gcp-and-azure). ### In short - AWS, GCP, and Azure support external identity providers (IDPs) via OpenID Connect (OIDC). - Workloads running outside the cloud (e.g., on-prem or in another cloud) can authenticate to cloud APIs using short-lived ID tokens from an external IDP. - These ID tokens must be securely retrieved and rotated, usually by software running alongside the workload. Major cloud providers offer built-in mechanisms for this when workloads run natively on their infrastructure via instance metadata services. ### Cloud-specific examples #### **AWS** EC2 instances can use instance profiles to obtain temporary credentials with the permissions of an IAM role. These credentials are fetched via the AWS Instance Metadata Service (IMDS) and can be used by workloads running on the instance to request a signed token from AWS STS. This token can then be exchanged: - With GCP’s Workload Identity Federation endpoint for a GCP access token. - With Azure Entra's Workload Identity Federation endpoint for an Azure access token. This allows workloads running on AWS EC2 instances to authenticate to GCP or Azure using the IAM role of the instance. #### **Microsoft Azure** Azure VMs can be assigned managed identities. Workloads on such VMs can retrieve access tokens directly from the Azure Instance Metadata Service without requiring secrets. The VM’s managed identity backs these tokens and is automatically scoped to the environment. This token can then be exchanged: - With GCP’s Workload Identity Federation endpoint for a GCP access token. - With AWS STS for AWS temporary session credentials. This allows workloads on Azure VMs to authenticate to GCP or AWS using the managed identity of the VM. #### **Google Cloud Platform (GCP)** GCP VMs can obtain identity tokens from the GCP Instance Metadata Service (IMDS). These tokens represent the VM’s service account and can be retrieved by any workload running on the VM without requiring secrets. This token can then be exchanged: - With Azure Entra's Workload Identity Federation endpoint for an Azure access token. - With AWS STS for AWS temporary session credentials. This allows workloads running on GCP VM instances to authenticate to Azure or AWS using the VM’s identity. ### Limitations While these integrations enable multi-cloud identity federation, they have important limitations: - **Shared identity per instance**: All workloads on a VM share the same identity, which limits isolation and accountability. - **No fine-grained workload identity**: There's no built-in way to distinguish which process or container made a request. - **Credentials not scoped to workload**: If one workload is compromised, it can use the shared identity to impersonate others on the same host. ## 2. The hidden credential management burden Even with cloud federation mechanisms in place, **something still has to retrieve the ID token** in the first place. This often means: - Storing client secrets or service account keys securely. - Managing lifecycle and refresh logic of tokens. - Protecting tokens at rest and in memory. When workloads run **within a cloud provider’s infrastructure**, this burden is significantly reduced thanks to built-in identity mechanisms: - **AWS Instance Metadata Service (IMDS)** provides IAM role credentials to EC2 instances. - **Azure Managed Identities** allow token acquisition without storing secrets. - **GCP Instance Metadata Service** exposes identity tokens tied to VM service accounts. However, as discussed above, these cloud-native options come with important caveats, particularly regarding shared identity at the instance level. As a result, **they may not be desirable in scenarios requiring strict workload-level isolation**. For hybrid environments, on-prem infrastructure, or more fine-grained identity boundaries, you're still often left with the traditional complexity of securely provisioning and rotating credentials to the right process and keeping them out of reach from others. Solutions like **HashiCorp Vault**, or **AWS Secrets Manager** help, but they introduce their overhead as they require setup, access control, encryption configuration, and often come with latency and availability concerns. In the end, **secure credential distribution and isolation remain a hard problem**, even when federation mechanisms are available. ## 3. What ideal federation *should* look like Imagine a world where you don’t have to think about storing or rotating secrets, where every workload has its own cryptographic identity, and where credential issuance is automatic and secure. This vision includes: - **Identity-first architecture**: Workload identity is the foundation. Everything from access decisions to credential issuance is built on this verifiable identity. - **No stored secrets**: Workloads receive ephemeral credentials only when needed. - **Short-lived credentials**: Always fresh, scoped, and revocable. - **Automatic rotation**: Credentials are seamlessly renewed without developer intervention. - **Secure delivery**: Credentials are scoped so only the intended workload can access them. - **Verifiable workload identity**: Identities are assigned securely based on workload attributes such as binary path, command-line arguments, workload name, namespace, or deployment metadata. A workload cannot assume any identity other than the one intended for it. This enables: - Fine-grained, per-workload identity even on the same node. - Simpler, declarative policy management. - Stronger security guarantees that are harder to bypass or misconfigure. This is exactly what we’re building at Riptides: rooted in the [Linux kernel](/blog/rethinking-workload-identity-at-the-kernel-level) and [built on SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), that works seamlessly across on-prem, hybrid, and cloud-native environments. ## 4. How Riptides solves this Riptides acts as an external identity provider (IDP) to cloud platforms like AWS, GCP, and Azure, solving the credential delivery challenge at the [operating system level](/blog/rethinking-workload-identity-at-the-kernel-level). Here’s how it works: - **Kernel-level visibility**: A [Linux kernel module](/blog/securing-workloads-with-kernel-telemetry-and-metrics) observes all network activity of workloads (processes and containers) in real time. - **User-space coordination**: A user-space agent configures the [kernel module with policies](/blog/rethinking-workload-identity-at-the-kernel-level) that define identity assignment and communication rules. - **SPIFFE SVIDs as workload identities**: Each workload receives a [SPIFFE ID](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust) based on runtime attributes such as binary path, command-line args, workload name, namespace, or deployment metadata. - **On-demand cloud credential issuance**: Riptides generates a short-lived identity token (SVID) and [exchanges it for cloud-specific temporary credentials](/blog/federating-non-human-identities-with-external-idps-using-id-tokens-in-aws-gcp-and-azure) via standard federation mechanisms. - **Secure credential delivery**: Riptides supports two secure credential delivery options to fit different use cases and business requirements: 1. **Dynamic `sysfs` delivery** — Credentials are exposed just-in-time via `sysfs`, scoped so only the requesting workload can read them at the exact moment they call a cloud API. 2. **On-the-wire injection** — Riptides intercepts outbound cloud API requests and replaces authentication headers with valid, short-lived credentials. - **Zero code changes required**: Existing cloud SDKs (AWS, GCP, Azure) continue to work either with a simple config tweak to read credentials from the `sysfs` path, or completely transparently when using the on-the-wire injection. - **Multi-cloud ready**: No need for cloud-specific agents. Riptides works across AWS, GCP, and Azure out of the box. - **Centralized governance**: Identity policies and permissions are managed centrally, making governance, auditing, and policy enforcement consistent and scalable. The diagram below shows the high-level flow of the `sysfs` based solution. In a follow-up post, we’ll cover the **on-the-wire credential replacement** approach. ![Non-human identity federation with Riptides across cloud providers](../../assets/riptides-nhi-federation/riptides-nhi-federation.jpg) ### Demo: Using Riptides prepared GCP credentials from `sysfs` with the `gcloud` CLI In the following recording, you’ll see a simple demo showing how credentials prepared by Riptides in `sysfs` can be used with the `gcloud` CLI. ![Demo with gcloud CLI](../../assets/riptides-nhi-federation/demo.gif) 1. **Initial state, no credentials available** First, you can see that the `gcp_credentials.json` file is not accessible. This is because the credential source configuration and policies, which Riptides uses to generate credentials for the `gcloud` CLI, have not yet been created. 2. **Grant workload access to GCP resources** Next, we run the step labeled `# Allow workload access to GCP resources`. In this step, we create the required credential source configuration and policies. These are defined as Kubernetes custom resources, which are consumed by the Riptides Control Plane: ```yaml apiVersion: core.riptides.io/v1alpha1 kind: CredentialSource metadata: name: gcp-wif spec: gcp: serviceAccount: demo-56@deft-diode-457816-s2.iam.gserviceaccount.com oidcProviderId: //iam.googleapis.com/projects/432279690143/locations/global/workloadIdentityPools/demo/providers/demo2 --- apiVersion: core.riptides.io/v1alpha1 kind: WorkloadCredential metadata: name: gcp-access-token spec: workloadID: staging/demo/gcloud-cli credentialSource: gcp-wif --- apiVersion: core.riptides.io/v1alpha1 kind: WorkloadIdentity metadata: name: gcloud-cli spec: scope: agentGroup: id: riptides/agentGroup/demo selectors: - process:binary:path: /snap/google-cloud-cli/364/usr/bin/python3.10 process:gid: 1000 process:uid: 501 workloadID: staging/demo/gcloud-cli ``` We’ll cover the full configuration process in a separate post to avoid overloading this one with too much detail. Briefly: - WorkloadCredential specifies which credentials a workload identity should receive. - CredentialSource defines how and where to obtain those credentials. - WorkloadIdentity describes the attributes of a workload, which must match at runtime for it to assume the referenced workload identity. Once configured, Riptides securely obtains the credentials from the cloud provider and delivers them to `sysfs` on the relevant node. 3. **Access to credentials restricted to `gcloud` workload** In the step labeled `# Access is restricted so that only the designated workload can read its assigned credentials`, we verify that no other process can access these credentials. Only the process matching the configured `WorkloadIdentity` attributes can read them. 4. **Successful authentication** After that, the `gcloud auth login` command succeeds because the credentials are now available in `sysfs`. Note the UUID after `/sys/kernel/riptides/credentials/` in the path; this is derived from the workload’s SPIFFE ID and scopes the credentials to that workload. This ensures that no other workload can access this path. The Linux kernel module verifies workload attributes at runtime to determine the workload’s identity. Only if the UUID derived from this identity matches the UUID in the path is the process allowed to read its contents. 5. **Listing VM instances** Once logged in, we successfully list VM instances using the `gcloud` CLI. 6. **Revoking access** Finally, in the step labeled `# Remove GCP access and try again`, we delete the credential source configuration and policies for the `gcloud` workload. As expected, listing VM instances now fails because the credential file has been removed from `sysfs`. **In short:** you never have to handle GCP credentials manually; Riptides provisions, delivers, scopes, and refreshes them securely and automatically, exactly when needed. ## 5. Why do we advocate for SPIFFE IDs as workload identity We believe every workload deserves a **verifiable, unique, and trusted identity** and [SPIFFE](/blog/introduction-to-spiffe-secure-identity-for-workloads) provides exactly that. ### Key benefits of SPIFFE - **Cloud-agnostic identity format**: One consistent identity format across all environments, thus there is no need to adapt to cloud-specific credential formats. - **Identity-first trust model**: Credentials are issued *only after* workload identity is verified. In traditional models, credentials are distributed to a host, and any process on that host can use them, regardless of what it is. Riptides changes that: 1. A workload’s **SPIFFE ID is securely issued and verified**. 2. **Only then** are short-lived cloud credentials issued, scoped to that identity. 3. The **Kernel module enforces** which identity a workload can be assigned based on workload attributes. This enforces **strict credential isolation**, reducing the risk of lateral movement, privilege escalation, and credential leakage. Riptides brings **zero-secret, per-workload identity** to any Linux system, with **no application code changes** required. It eliminates manual credential distribution and ensures that each workload receives the right credentials, at the right time, in the most secure way possible. ## Final Thoughts Modern cloud-native federation is a step forward — but it still places a hidden burden on developers and operators. Cross-cloud federation helps, but it’s **not workload-aware, not granular, and certainly not zero-trust**. Riptides provides a truly secure, zero-touch identity solution: - Fine-grained, SPIFFE-based workload identities - Secure, ephemeral credentials - No secrets stored - Seamless multi-cloud integration If you're building secure cloud-native systems at scale, it's time to rethink how you manage non-human identity. Let Riptides handle it for you securely, automatically, and correctly. In our next post, we’ll dive deeper into the on-the-wire credential injection method and demonstrate how Riptides works in practice with real applications and cloud providers, all without secrets. Stay tuned. --- ## Securing Workloads with Kernel Telemetry and Metrics - URL: https://blog.riptides.io/securing-workloads-with-kernel-telemetry-and-metrics - Published: 2025-08-07 - Author: Janos Matyas - Category: Kernel - Tags: spiffe, identity, zero-trust, kernel, linux ## Securing Workloads with Kernel Telemetry and Metrics In modern infrastructure, workloads—not humans—make most of the decisions and connections. They spin up services, authenticate peers, and communicate across nodes faster than any human operator ever could. Yet, to secure and verify these interactions, we need visibility deep enough to *understand intent*, not just traffic. At Riptides, this journey started in the Linux kernel—with tracepoints. This post recaps our path from breakpoints to production ready Prometheus metrics and shows how kernel-level telemetry empowers our non-human identity platform with unmatched context and trust. ## From Debugging to Tracing: Listening to the Kernel Our journey began where most kernel work does: debugging. Traditional breakpoints offer a glimpse into execution but halt progress—fine for development, not for real-time systems. [In our first post](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing), we moved from breakpoints to **tracepoints**, opening a window into kernel events without blocking the system. Tracepoints are efficient, built-in hooks in the Linux kernel. They let us observe what's happening—packet reception, syscalls, context switches—without modifying the code or stopping execution. For observability and security alike, that non-intrusive visibility is gold. ![From Debugging to Tracing: Listening to the Kernel](../../assets/from-kernel-telemetry-to-identity/illustration1.jpg) ## Surfacing Events: Tracepoints to User-Space Metrics Tracepoints give us low level visibility, but events alone isn’t insight. In [our second post](/blog/from-tracepoints-to-metrics-a-journey-from-kernel-to-user-space), we detailed how we transform tracepoint events into **structured telemetry** consumable by our userspace systems. The key is efficient transport. We use `ringbuf` (not `perfbuf`) for minimal overhead and latency. We write custom kernel modules to handle filtering, sampling, and formatting near the source. This avoids swamping userspace with noise and allows us to focus on high value signals—like syscall patterns, packet anomalies, or handshake timelines. ![Surfacing Events: Tracepoints to User-Space Metrics](../../assets/from-kernel-telemetry-to-identity/illustration2.jpg) ## From Kernel Events to Prometheus Metrics With structured events flowing into user-space, the next step is surfacing them to the broader observability ecosystem. [In our third post](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability), we connected our kernel events to **Prometheus** metrics. Why Prometheus? Because it's the lingua franca of observability. Engineers and security teams alike already use it. By exposing metrics from kernelspace (e.g., handshake latencies, dropped packets, authentication attempts), we enrich dashboards and alerts with context that typically lives far beneath the surface. Instead of just *"500 errors increased,"* we can say, *"TLS handshake failed for pod A to B due to invalid identity, traced to `ksys_read` syscall path and confirmed by dropped SYN-ACK in `tcp_retransmit_skb`."* This is not just observability. It's **forensics in real time**. ![From Kernel Events to Prometheus Metrics](../../assets/from-kernel-telemetry-to-identity/illustration3.jpg) ## Beyond the Usual Suspects: Telemetry as a Security Primitive While most telemetry focuses on CPU, memory, and I/O, [our fourth post](/blog/linux-kernel-module-telemetry-beyond-the-usual-suspects) zoomed in on **communication patterns**. Because at Riptides, communication isn’t just performance data—it’s identity data. Our eBPF-based pipeline captures how services behave when they initiate or receive connections. What identities they use. Whether they comply with policy. If they retry, escalate, or fail silently. These behaviors form a **behavioral fingerprint**. By combining this with identity protocols like SPIFFE and enforcing policy through in kernel controls (like kTLS and in kernel mTLS), we don’t just observe; we *enforce trust* at the lowest possible level. ![Beyond the Usual Suspects: Telemetry as a Security Primitive](../../assets/from-kernel-telemetry-to-identity/illustration4.jpg) ## Why Metrics Matter for Non-Human Identity In human-centric security, we rely on context: location, device, login history. For workloads, the kernel is that context. - Did this pod establish mTLS with a peer inside the cluster or across a boundary? - Is this service making calls outside its declared dependency graph? - Was this binary spawned by a known parent process, or was it injected? These are **kernel-level questions**. And their answers emerge from kernel level metrics. That’s why we anchored our non-human identity platform in Linux telemetry. Metrics aren’t just a monitoring tool—they’re a **source of truth**. They let us prove that identity isn’t just declared, but **expressed** in how code behaves. ## Conclusion: Identity, Observability, and the Kernel Security starts with identity—but trust is built on behavior. At Riptides, we use Linux telemetry to tie identity to behavior in real-time, without relying on user-space agents or guesswork. By tracing from the kernel up, we deliver deep observability and robust non-human identity enforcement that scales across clusters, clouds, and container runtimes. The kernel is speaking. We're just making sure the right systems are listening. --- ## The Hidden Risk in Service Mesh mTLS: When Your Sidecar Becomes a Trojan Horse - URL: https://blog.riptides.io/the-hidden-risk-in-service-mesh-mtls-when-your-sidecar-becomes-a-trojan-horse - Published: 2025-08-04 - Author: Zsolt Varga - Category: Security - Tags: spiffe, x509, mTLS, Istio ## Introduction Service meshes like Istio promise stronger zero-trust security by automating mutual TLS (mTLS) between workloads. By offloading certificate management to a sidecar proxy (typically Envoy), mTLS becomes transparent to application developers. But this convenience hides a critical flaw, **rogue processes on the same node or inside the same pod can impersonate trusted workloads**. As we discussed in our blog post - [Rethinking Workload Identity at the Kernel Level](/blog/rethinking-workload-identity-at-the-kernel-level), sidecars and proxies simplify network security but introduce ambiguity around *who* is actually speaking on the wire. It's time to rethink how workload identity is established and enforced. At Riptides, we believe that true zero trust starts not at the edge, but **inside the kernel** — where [identity can be cryptographically anchored to the process itself](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). If a proxy can't distinguish between two processes behind the same socket, it's not enforcing zero trust — it's just forwarding packets. ## How Istio Implements mTLS In Istio (and similar meshes like Linkerd), secure communication between services is achieved through automatic mTLS, implemented via: - A dedicated **sidecar** injected into each pod to handle ingress and egress. - The Istio control plane component (`istiod`) issuing **X.509 certificates** to Envoy via SDS (Secret Discovery Service). - Certificates containing **SPIFFE-based identities**, e.g., `spiffe://cluster.local/ns/default/sa/myserviceaccount` - **iptables** (or similar) rules routing all pod traffic through Envoy. - Envoy handling mTLS handshakes and peer identity verification. ### Certificate Lifecycle - Istiod issues short-lived certificates (default TTL: 24h). - Certificates are rotated automatically via SDS pushes. - The **workload itself never sees the private key** — only Envoy handles it. This architecture makes strong cryptographic identity "invisible" to the application, and that’s where the problem begins. ## The Trust Assumption and Its Flaw The system **implicitly trusts** any connection that: > arrives over mTLS and presents a valid SPIFFE identity. While this is technically sound **at the network layer**, it is **not sufficient at the workload level**. Envoy authenticates other Envoys and not the actual application inside the pod. The application: - Does **not participate** in the TLS handshake. - Cannot **sign requests** or cryptographically assert its identity. - Relies entirely on Envoy to represent it. ## Attack Scenario: Rogue Process Impersonation Here’s how a process-level impersonation attack can happen, even in a pod with a single application container: 1. **Step 1: Initial Compromise** An attacker exploits a vulnerability in the application or breaks out of a container into the pod namespace. 2. **Step 2: Reconnaissance** They find Envoy listening on `127.0.0.1:15001` (outbound) and `127.0.0.1:15006` (inbound) — standard Istio ports. 3. **Step 3: Forged Requests** Using `curl`, `netcat`, or a custom application, the attacker sends arbitrary HTTP requests to Envoy’s outbound listener. 4. **Step 4: mTLS Proxying** Envoy forwards the request over mTLS using the pod’s legitimate certificate. The receiving Envoy verifies the SPIFFE ID and allows the request. 5. **Step 5: Impersonation Complete** The target service sees a valid, authenticated SPIFFE identity — and **has no way to tell** it wasn’t the real application. This is *not* a theoretical issue — it’s a byproduct of conflating **network identity** with **process identity**. ![istio-issue-illustration](../../assets/service-mesh-mtls-hidden-risk/illustration1.jpg) ## Technical Deep Dive ### Let’s inspect the Envoy configuration that enables this behavior Outbound cluster: ```yaml clusters: - name: outbound|8080||target-service.default.svc.cluster.local transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificate_sds_secret_configs: - name: default ``` Inbound listener: ```yaml listeners: - address: socket_address: address: 127.0.0.1 port_value: 15006 filter_chains: - filters: - name: envoy.filters.network.http_connection_manager ``` The critical detail: any process in the pod can access localhost:15001 or 15006. There is no enforcement at the socket or process boundary. Envoy assumes it’s proxying on behalf of the legitimate workload, but it has no way to prove that. ## Why Namespaces and Sidecars Aren’t a Security Boundary Even when containers run in separate namespaces, Kubernetes pods often share: - **Network namespace** (e.g., localhost is shared) - **PID namespace** (sometimes) - **Volumes and secrets** (occasionally) This means: - Even "unprivileged" containers can talk to the sidecar. - Rootless containers are not always enforced. - Sidecar boundary is not a security boundary. ## Towards Workload-Bound Identity Closing this gap requires cryptographic identity tied to the actual process, not just the pod or the network path. To close this gap, what we propose: - Using **identity attestation agents inside the process boundary**. - Leveraging **kernel based TLS** to bind process to cryptographic material. - Applying **SPIFFE** to issue identity per process using selectors (e.g., PID, Cgroup, etc). This results in: - **Cryptographic identity** that is **bound to the actual process**, not the network path. - **mTLS termination in-process**, removing reliance on external sidecar boundaries. ## Conclusion Sidecar-based mTLS brings convenience, but also risk. It allows any co-located process to speak as the workload, with no cryptographic attestation at the process level. In a true zero-trust architecture, you must **authenticate the process, not a proxy in front of it**. To build secure-by-default systems: - Treat sidecars as a *transport layer*, not a *security boundary*. - Move toward **cryptographic identities tied to processes** and **kernel enforcement**. - Design for **explicit attestation** and **auditable identity paths**. Non-human identity is too foundational and too sensitive to be bolted on as an afterthought. It deserves a native, first-class treatment. By operating directly in the Linux kernel, Riptides removes layers of complexity and guesswork, grounding identity in the one place all workloads truly run. No proxies, no sidecars, no credential sprawl just cryptographic trust, bound to the process itself. In an era of lateral movement and advanced threats, network identity isn’t enough. Process identity is the new perimeter. The future of identity isn't just more secure. It's leaner, simpler, and built in from the start. And with Riptides, it’s already here. --- ## Building Linux Driver at Scale: Our Automated Multi-Distro, Multi-Arch Build Pipeline - URL: https://blog.riptides.io/building-linux-driver-at-scale-our-automated-multi-distro-multi-arch-build-pipeline - Published: 2025-07-28 - Author: Peter Balogh - Category: Kernel - Tags: kernel, linux, build, automation ## Anchoring Identity in the Kernel Requires Building at Scale Modern infrastructure requires strong, scalable, and transparent identity mechanisms — especially for non-human actors like services, workloads, jobs, or AI agents. At Riptides, we deliver exactly that by [anchoring non-human identity at the kernel level](/blog/rethinking-workload-identity-at-the-kernel-level). Using technologies like [SPIFFE](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust), [kTLS, and in-kernel mTLS handshakes](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe), our platform offers zero trust security: authenticated, encrypted communication for user-space applications, without requiring developers to modify their code. But to make this work, our system needs deep integration with the Linux kernel. We deploy custom [eBPF-based telemetry](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability) and kernel modules that issue identity, track posture, and enforce policy at runtime. That means supporting a growing matrix of kernel versions, distributions, and CPU architectures — and being able to build and deliver the right driver to the right workload, at the right time. This post walks through how we built a cloud-native, incremental kernel module build system using Docker, GitHub Actions, and EKS — one that produces hundreds of kernel-specific drivers daily, with minimal friction and full automation. ## Incremental, Cloud-Native Builds for Hundreds of Kernel Versions Powered by Docker, GitHub Actions, and EKS Building and maintaining kernel drivers for half a dozen Linux distributions and two CPU architectures is a deceptively complex task. Dependency hell, constantly changing kernel Application Binary Interfaces (ABIs), and the need for rapid vulnerability fixes turn a "simple" `make && make install` into a true at-scale engineering challenge. This article walks through the architecture, and inner workings of our build pipeline which is a Docker-based, GitHub Actions powered system that compiles and ships drivers to an S3 bucket across **Ubuntu, Amazon Linux 2, Amazon Linux 2023, Fedora, AlmaLinux, and Debian** on both *x86_64 and arm64*. ## Challenges Maintaining a modern kernel driver farm means satisfying three moving targets simultaneously: supported distributions, supported kernel releases within each distribution, and supported CPU architectures. Each axis amplifies matrix complexity exponentially. - **Distribution fragmentation:** Different package managers, divergent header package naming conventions, and incompatible toolchains force per-distro build containers. - **Kernel cadence:** Distributions such as Fedora and Ubuntu LTS publish new kernels weekly to address CVEs, so prebuilt drivers become obsolete almost overnight. - **Architecture diversity:** x86_64 still dominates, but aarch64 (ARM64) is rapidly gaining traction in Graviton-powered AWS fleets and edge devices. Cross-compilation often fails because tracing hooks and CONFIG_* flags differ by arch. **A quick calculation of driver build matrix size**: | Axis | Current coverage | | -------------------------------- | ---------------------------------------------------------------------------- | | Distributions | Ubuntu, Amazon Linux 2, Amazon Linux 2023, Fedora, AlmaLinux, Debian | | Kernel releases per distro (AVG) | 40 | | Architectures | x86_64, aarch64 | | Theoretical build variants | 540 (6 distros × 40 kernels × 2 arches) | Even a modest 480 variant grid exceeds what can be built manually, so the **automation is non-negotiable**. ## How the Falco Community Tackles the Problem The Falco security project faced a nearly identical challenge compiling its kernel module and eBPF probes. Their answer combines three open-source components: - [kernel-crawler](https://github.com/falcosecurity/kernel-crawler): Scrapes distro mirrors weekly, producing a JSON manifest of every kernel header package. - [dbg-go](https://github.com/falcosecurity/dbg-go): Consumes the JSON manifest and generates driver-specific build configurations. - [prow](https://github.com/kubernetes-sigs/prow): A Kubernetes-native CI system originally built for the Kubernetes project, orchestrating containerized build jobs at scale. Falco’s test-infra repository wires these pieces together: kernel-crawler opens a PR with new kernels; Prow detects the change, fans out parallel driverkit builds, and uploads finished artifacts. We decided against using Prow primarily due to its configuration complexity and steep learning curve. Additionally, Prow’s terminology, such as ProwJob, Deck, Tide, and Hook is tightly coupled to its internal architecture and not immediately intuitive. This makes onboarding and day-to-day maintenance more difficult compared to alternatives that prioritize simplicity. ## Our Opinionated Approach: Docker + GitHub Actions + EKS We distilled the Falco pattern into a leaner stack that leverages tools our engineers already use daily. | Layer | Our Implementation | Rationale | |-----------------------|-------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------| | CI Orchestrator | GitHub Actions | Native to repos, rich marketplace, familiar YAML workflow syntax | | Runner Fleet | Self-hosted runners on Amazon EKS via [Actions Runner Controller](https://github.com/actions/actions-runner-controller) | Autoscaling, spot-instance friendly, GitHub-supported | | Containerized Builder | Distro-specific Dockerfiles | Reproducible toolchains, no host pollution | | Matrix Definition | Custom Go tool parsing Falco's [kernel-crawler](https://github.com/falcosecurity/kernel-crawler) output | Reuses community data, but fine-tuned for our needs | | Artifact Store | Amazon S3 | Cheap, durable, global distribution | **High Level Architecture:** ![High Level Archutecture](../../assets/kmod-build-at-riptides/high-level-arch.png) ### Generating the Build Matrix At the heart of the pipeline is our matrix-gen CLI tool that consumes Falco’s kernel-crawler JSON dataset and emits a filtered JSON tailored to our needs. ```bash matrix-gen generate\ --distro=ubuntu,amazonlinux2,amazonlinux2023,fedora,almalinux,debian \ --arch=x86_64,aarch64 \ --driver-version=0.1.1 \ --kernel-release='.*6\.[0-9]+\..*' ``` **Sample Output Element**: ```json [ { "kversion": "6.1.12-17.42.amzn2023.x86_64", "distribution": "amazonlinux2023", "architecture": "x86_64", "kernelurls": [ "https://amazonlinux-2023-repos.s3.us-west-2.amazonaws.com/2023.0.20231016.0/kernel/6.1.12-17.42.amzn2023.x86_64/kernel-6.1.12-17.42.amzn2023.x86_64.rpm" ], "output": "0.1.0/amazonlinux2023/x86_64/6.1.12-17.42.amzn2023.x86_64", "driverversion": "0.1.1" }, ... ] ``` Key fields: - **kernelurls:** Direct links to header RPM/DEB packages, used when distro mirrors purge older kernels. - **output:** S3 key prefix (patch versions does not change the path in the output). - **driverversion:** Semantic version baked into the built .ko files. ### Matrix Diffs: Building Only What Changed Each nightly GitHub Actions workflow takes three steps: 1. Download previous matrix from S3. 2. Regenerate a fresh matrix with matrix-gen. 3. Smart diff the JSON files. The matrix-gen CLI includes a dedicated diff subcommand, so it computes the delta between the previously published matrix and the freshly generated one. Only new or modified entries feed the build matrix. This approach shrinks CI time drastically. If just two new Fedora kernels land overnight, the workflow launches only two jobs instead of rebuilding hundreds. When we release a driver patch, the diff naturally marks every entry as changed, ensuring all variants are rebuilt with fixes. ### Containerized Build Execution with Elastic Infrastructure After the build matrix is generated, we rely on three key pillars to make our driver builds fast, simple, and reproducible. **1. Docker-Based Build Jobs**: We maintain a directory of slim, distro-specific Dockerfiles that have common characteristics: - Mirror install first (apt-get install linux-headers-${kversion} style). If that fails, fall back to the explicit kernelurls defined by the matrix. - ENTRYPOINT is a shared shell script that runs the driver build, and copies bearssl.ko + riptides.ko into /output. **2. Architecture-Aware Scheduling**: Each matrix element carries an 'architecture' field. GitHub Actions assigns a runs-on: [self-hosted, build, x86_64] or [self-hosted, build, aarch64] label dynamically, ensuring ARC schedules the pod on a node with the matching CPU. **3. Runner & Node Autoscaling**: [Actions Runner Controller (ARC)](https://github.com/actions/actions-runner-controller)’s RunnerDeployment objects declare how many idle runners should persist. HorizontalRunnerAutoscaler (HRA) adjusts replicas using Live Job Queue metrics. Meanwhile, the Kubernetes Cluster Autoscaler on EKS grows or shrinks EC2 nodes based on pending pods, giving us a two-tier elasticity model: - **Inner loop:** HRA scales runners (pods) from 1 → N based on demand. - **Outer loop:** Cluster Autoscaler provisions new nodes if the pod-level scale-up cannot fit. The result: cost-effective builds that spin up on demand and disappear minutes after completion. **Sequence Diagram of Driver Build:** ![Build drivers](../..//assets/kmod-build-at-riptides/kmod-build-seq.png) ## Future Evolution: RunnerScaleSets A Runner Scale Set is a managed group of self-hosted GitHub Actions runners that can automatically scale up or down based on job demand. Key Advantages of RunnerScaleSet Over HorizontalRunnerAutoscaler: - **No Secrets in Runner Pods:** Authentication is handled at the controller, so sensitive tokens aren’t injected into each runner, improving security. - **Fewer GitHub API Calls:** Scale Sets use efficient long-polling connections, reducing API rate-limit issues common with HorizontalRunnerAutoscaler. - **More Reliable and Responsive Scaling:** Direct 'job available' signals from GitHub provide faster, more accurate scaling, including robust scale-to-zero. - **Advanced Features:** Improved runner pod customization and better support for large-scale or multi-repo environments - **Reduced Operational Overhead:** Fewer moving parts make management and upgrades easier. RunnerScaleSets are more secure, efficient, and easier to manage compared to HorizontalRunnerAutoscaler. ## Conclusion By integrating open-source tools like `kernel-crawler`, leveraging containerized builds, and deploying a cloud-native CI/CD pipeline with elastic scaling, we turned a notoriously brittle part of kernel engineering into a reliable, automated system. The result is a scalable driver build infrastructure that keeps pace with upstream kernel releases, supports a diverse range of environments, and requires minimal human intervention. This investment is foundational. The telemetry and enforcement capabilities that set Riptides apart begin with kernel-level visibility. And that visibility depends on having the right module, built for the right kernel, shipped to the right place — every time. With this system in place and upcoming migration to GitHub Runner Scale Sets, we're ready to scale even further, simplify operations, and continue delivering on our mission to secure workloads from the kernel up. **Key takeaways:** - Use the existing data to save time by leveraging collected kernel-crawler JSON instead of scraping distribution mirrors yourself. - Diff your matrix to keep nightly builds incremental. - Leverage two-layer autoscaling ARC for runners, Cluster Autoscaler for nodes (both speed and cost control). - Keep Dockerfiles declarative and immutable, fallback URLs shield you from mirror churn. This setup meets today’s scale comfortably, and with upcoming RunnerScaleSets we expect even smoother scaling and simpler secrets management. --- ## SharePoint Under Siege: Lateral Movement Is Still Security’s Blind Spot - URL: https://blog.riptides.io/sharepoint-under-siege-lateral-movement-is-still-securitys-blind-spot - Published: 2025-07-24 - Author: Janos Matyas - Category: Security - Tags: lateral-movement, cloud, zero-trust, kernel The recent SharePoint 0-day (CVE-2025-53770), now linked to the ToolShell campaign, has reawakened enterprise fears around lateral movement. What started as a remote code execution (RCE) vulnerability on exposed SharePoint servers quickly escalated to full internal compromise, with attackers using forged tokens, dropped webshells, and privilege escalation to move sideways into Teams, Exchange, and other Microsoft services. As [Eye Security's excellent triage highlights](https://research.eye.security/sharepoint-under-siege/), the exploit enables attackers to plant a web shell and extract ASP.NET machine keys. With those keys, they forge authentication tokens, impersonate users, and move laterally with near-total stealth. This isn't just about SharePoint. It's about a bigger question: **why is lateral movement still this easy in 2025?** And what will actually stop it? ## What Is Lateral Movement and Why Is It So Effective? Lateral movement refers to the set of techniques attackers use to pivot from an initial foothold to other systems within the network. It's the "quiet part" after the breach - the phase when the attacker maps out your environment, impersonates legitimate users, steals credentials, and quietly escalates their access. ToolShell, like many modern attacks, leverages stolen keys or tokens to create authenticated sessions that bypass standard defenses. Once inside, network segmentation offers little protection. Why? Because most enterprises **still rely on perimeters between machines - not between processes**. ## Why Traditional Defenses Fall Short Recommendations like network segmentation and least privilege access are standard. They're also mostly ineffective against modern attacks. Segmenting VMs or VLANs is not the same as segmenting the actual runtime identity and behavior of each process. Once a legitimate process is hijacked - as in the SharePoint attack - the lateral movement begins inside the same host. No traditional firewall, WAF, or microsegmentation solution can see that level of granularity. Worse, many tools assume trust based on IP addresses or service names - which are trivial for attackers to spoof once inside. ## What If Lateral Movement Was Impossible? At **Riptides**, we approach this differently. We believe every process - even those running on the same machine - must be explicitly authenticated, authorized, and continuously verified. That’s why we built a kernel level telemetry and enforcement layer that: - Assigns **[SPIFFE-based identities](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust)** to every process - Enforces **[mutual TLS (mTLS)](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe)** between all communications, even local ones - **[Monitors posture](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability)** continuously, not just at connection time - Controls flows with **[per-process, per-intent policies](/blog/rethinking-workload-identity-at-the-kernel-level)** at the kernel level This means if an attacker compromises a workload but tries to pivot to other applications, or an internal data pipeline, they’ll be blocked. Not because of a VLAN rule. But because the process trying to communicate doesn’t have the right SPIFFE identity or posture. It simply won’t pass the mTLS handshake, and the kernel will drop the packet. ## Why Zero Trust Must Include Internal Tools A dangerous myth persists in security: that internal services can be "trusted by default." But the ToolShell campaign is yet another reminder that internally hosted tools - even critical ones like SharePoint - can become entry points. Zero Trust isn’t just about users or external APIs. It’s about every binary, every tool, every CLI, every service. Riptides brings **Zero Trust down to the process level** - with SPIFFE based identity, posture enforcement, mTLS, and flow control from the kernel up. We don’t just log attacks. We stop them - before they move. It's time to move beyond surface level defenses - real Zero Trust starts with every process, every flow, and every decision enforced close to execution. That’s why **[we’re rethinking workload identity at the kernel level](/blog/rethinking-workload-identity-at-the-kernel-level)**. **Take control of lateral movement risks - [let's talk!](https://riptides.io/request-a-demo)** --- ## From Tracepoints to Prometheus: The journey of a kernel event to observability. - URL: https://blog.riptides.io/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability - Published: 2025-07-21 - Author: Zsolt Rappi, Balint Molnar - Category: Kernel - Tags: kernel, ebpf, tracing, event, metrics ## Summary and Hands-on with Riptides' Telemetry Infrastructure Telemetry isn’t just observability at Riptides — it’s foundational to everything we do. To enforce policy at the source, issue **[identity from the kernel](/blog/rethinking-workload-identity-at-the-kernel-level)**, and monitor service posture with precision, we’ve built a custom eBPF-based telemetry pipeline that gives us deep, real-time visibility into workload behavior and inter-service communication. Over the past few weeks, we’ve shared how this foundation was built. If you missed those posts, we recommend starting there, as this entry builds on the concepts introduced earlier. In our [first blog post](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing), we explored how to extract events/signals from kernel space, explaining why we chose the tracepoints and how it works. The [second blog post](/blog/from-tracepoints-to-metrics-a-journey-from-kernel-to-user-space) focused on streaming events into user space, our rationale for using eBPF, and the key eBPF features powering our approach. Finally, in the [third blog post](/blog/linux-kernel-module-telemetry-beyond-the-usual-suspects), we showed how to turn kernel tracepoint events into OTEL-compatible telemetry, enriching raw data with user space context to produce actionable insights. In today’s blog post, we’re introducing a simple open-source project that demonstrates how to transfer enriched kernel tracepoint data using eBPF, OTEL, and Prometheus. The project includes: - A kernel module that emits a tracepoint event when a file is created. - eBPF code that uses a ring buffer to transfer the event data to user space. - A Go application that manages eBPF requirements and initializes OTEL to export the metrics to a Prometheus exporter. **Disclaimer:** ‍We’re aware that more advanced eBPF features, such as CO-RE (Compile Once – Run Everywhere), could simplify development and improve portability across kernel versions. However, for the purposes of this demo and to ensure compatibility with a wide range of LTS (Long-Term Support) kernel versions, we intentionally chose a more manual and broadly compatible approach. ## Tracing File Creation with a Kernel Module > Example code can be found in this [repo](https://github.com/riptideslabs/ebpf-tracing-demo). ### Deep Dive into Kernel Module Details To detect file creation events in the kernel, we rely on the `do_filp_open` system call. Since we don’t want to recompile the kernel to modify this function directly, we can’t insert a tracepoint into its code. Fortunately, as we discussed in our blog posts, we can use **kprobes** and **kretprobes** to hook into it. We register a handler function that triggers our custom tracepoint event after each call to `do_filp_open`. ```c static struct kretprobe rp = { .kp.symbol_name = "do_filp_open", .entry_handler = do_filp_open_pre, .handler = do_filp_open_ret, .data_size = sizeof(struct probe_data), }; ``` As you can see, we're using a **kretprobe**, which is necessary because we need access to the return value of the `do_filp_open` call to determine whether a file was successfully opened. First, we define the symbol name where the probe attaches. Then—this is the tricky part—we attach an `entry_handler` function (which acts like a kprobe). We do this because the information needed to identify a file creation is only available in the function’s input parameters, which are accessible in the entry handler. The actual tracing happens in the `handler` function (the return handler), where we retrieve the file name and decide whether to emit the tracepoint event. Interestingly, both the entry and return handlers receive the same `struct pt_regs *regs` parameter, which is used to access function arguments like the file name or detect the nature of the operation. However, since this is such a low-level interface, the structure and calling convention can vary between system architectures. To make our implementation at least somewhat portable, we need to manually handle these differences based on the target architecture. ```c #if defined(CONFIG_ARM64) op = (const struct open_flags_partial *)regs->regs[2]; #elif defined(CONFIG_X86_64) op = (const struct open_flags_partial *)regs->dx; ``` These function parameters are determined by the Linux kernel configuration and architecture, so you can rely on compile-time evaluation to select the correct code path for the target system. However, that’s not the only concern **—the signature of** `do_filp_open` **may change between kernel versions.** That’s why we explicitly state that our implementation has been tested on **Linux kernel version 6.11**. It might work on other versions, but since we're accessing low-level memory and kernel internals, there’s a real risk of crashing your system if something changes unexpectedly. The final member of the `kretprobe` struct is `data_size`. This field defines the size of the per-instance data structure (in our case, `struct probe_data`) that’s used to pass information between the entry and return handlers. The entry handler checks if the file operation is a **file creation** and stores that information in the `probe_data` struct. The return handler then uses that data to decide whether or not to emit the tracepoint event. ### Build and Load the Module > If you haven’t already, please clone the [repository](https://github.com/riptideslabs/ebpf-tracing-demo). This blog assumes you're using a **Mac with Apple Silicon**. To run Linux, our tool of choice is [lima](https://lima-vm.io). Start by creating and logging into a virtual machine: ```shell limactl start --set '.mounts[0].writable=true' --name ebpf template://ubuntu limactl shell ebpf ``` > All commands from here onward should be run **inside the VM**. First, install the dependencies required to build the kernel module: ```shell # First, we need to install GNU Make, as we have prepared convenient Makefile targets for building and managing the module. sudo apt install make make setup ``` Once the setup is complete, try building and loading the module: ```shell # Build the module make # Load the module make insmod # Verify the module is loaded cat /proc/modules | grep filewatcher ``` ### Verify the Module is Working Once the module is loaded, it should register a custom tracepoint in the kernel. To verify this, run: ```shell # Switch to root user sudo su # List the registered tracepoint directory ll /sys/kernel/tracing/events/filewatcher/ ``` If everything is set up correctly, you should see output similar to: ```shell total 0 drwxr-xr-x 1 root root 0 Jul 18 11:26 ./ drwxr-xr-x 1 root root 0 Jan 1 1970 ../ -rw-r----- 1 root root 0 Jul 18 11:26 enable drwxr-xr-x 1 root root 0 Jul 18 11:26 file_created/ -rw-r----- 1 root root 0 Jul 18 11:26 filter ``` Before moving on to the eBPF part, let’s test the tracepoint using **ftrace**. First, enable the tracepoint by writing `1` into the `enable` file, then use `trace_pipe` to watch for events: ```shell sudo su cd /sys/kernel/tracing/ echo 1 > events/filewatcher/enable # Monitor tracepoint events as they occur cat trace_pipe ``` On a separate shell ```shell limactl shell ebpf touch riptides ``` If everything is working correctly, you should see output similar to this: ```bash root@lima-ebpf:/sys/kernel/tracing# cat trace_pipe touch-6821 [001] d..2. 8711.205770: file_created: /Users/baluchicken/prj/riptidesio/ebpf-tracing-demo/riptides ``` Here, we used the kernel's tracing infrastructure, **ftrace**, to validate that our module is functioning properly. In short, **ftrace** is a powerful framework built into the Linux kernel. It offers several built-in tools and high-throughput mechanisms—like **ring buffers**—to efficiently handle kernel event tracing. For a deeper dive into ftrace and its internals, check out our [first blog post](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing). ## Streaming Events from the Kernel to User Space Now that the kernel module is working, the next step is to **stream events to user space** using eBPF. ### Deep Dive into the eBPF Module To achieve high-throughput, low-latency communication between the kernel and user space, we use an **eBPF ring buffer**: ```c struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 65536); __type(value, struct file_created_event); } filewatcher_ringbuf SEC(".maps"); ``` For simplicity, we’ve explicitly defined the value type (struct file_created_event). While this isn't required, it restricts the buffer to a specific data type. In practice, the ring buffer can store **any data structure**—as long as it fits within the allocated size—making it flexible for multiple use cases. ```c SEC("tracepoint/filewatcher/file_created") int trace_file_created(struct file_created_ctx *ctx) ``` The familiar `SEC` macro is used to attach our eBPF handler to the custom tracepoint. In this case, we're listening to the `file_created` event under the `filewatcher` tracepoint. Inside the handler, we use multiple `bpf_printk` calls for debugging. These messages can be viewed in the kernel log using `dmesg`, which helps verify that our eBPF logic is being triggered correctly. Manually building and loading eBPF programs can be quite involved. To simplify the process, we use [Cilium’s eBPF library](https://ebpf-go.dev), which provides a powerful and developer-friendly interface for working with eBPF in Go. ### Deep Dive into the eBPF Module Loader/Reader To generate Go bindings from our eBPF C structs, we use **Cilium’s** `bpf2go` tool. It automates the process of compiling eBPF code and generating the corresponding Go types and accessors. To use it, simply add a `//go:generate` directive in your Go source file. When you run `go generate`, `bpf2go` takes care of compiling the eBPF code and creating Go wrappers. One of the first steps is to load the generated eBPF objects using the loader: ```go var objs tracerObjects err := loadTracerObjects(&objs, nil) if err != nil { log.Fatalf("Error loading ebpf objects: %v", err) } defer objs.Close() ``` Once the eBPF objects are loaded, we can attach the eBPF program to our custom tracepoint: ```go tp, err := link.Tracepoint("filewatcher", "file_created", objs.TraceFileCreated, nil) ``` Next, we open a reader for the ring buffer to receive data from kernel space: ```go rb, err := ringbuf.NewReader(objs.FilewatcherRingbuf) ... rec, err := rb.Read() ... err = binary.Read(bytes.NewBuffer(rec.RawSample), binary.LittleEndian, &event) ``` At this point, the data has made its full journey: from a **kernel tracepoint**, through an **eBPF ring buffer**, into **user space memory**. The final piece of the puzzle is **turning these raw events into metrics**. ## Enriching Raw Events into Metrics with OTEL To convert raw events into meaningful metrics, we use **OpenTelemetry (OTEL)** the industry standard for collecting, processing, and exporting telemetry data. OTEL allows you to transform raw events into whatever telemetry format you need. In our case, we’ll expose them as **Prometheus counters**. Thanks to OTEL’s rich ecosystem, initializing a Prometheus exporter is simple: ```go exporter, err := prometheus.New() mp := metricsdk.NewMeterProvider(metricsdk.WithReader(exporter)) otel.SetMeterProvider(mp) ``` Next, we define the metric itself. Here, we create a **counter** named `file_created_total` and attach custom labels such as `path` and `name`: ```go fileCreatedCounter, err = meter.Int64Counter("file_created_total", metric.WithDescription("Number of files created")) ... attribute.Key("path").String(path), attribute.Key("name").String(name), ``` At this point, everything is wired up: - Kernel tracepoint triggers on file creation - eBPF sends the event to user space via a ring buffer - Go code reads the event, processes it, and emits a labeled metric via OTEL Now you're ready to try out the full demo! To load and run the eBPF module along with the OTEL exporter, simply run: ```shell make run-ebpftracer ``` ### Check the Metrics If everything is working correctly, a metrics server should be running at `localhost:8080`. Before querying it, let’s generate some events by creating a file: ```shell touch riptides-demo ``` Then fetch the metrics: ```shell curl localhost:8080/metrics | grep file_created ``` You should see an output similar to: ``` file_created_total{name="riptides-demo",otel_scope_name="filewatcher",otel_scope_schema_url="",otel_scope_version="",path="/Users/baluchicken/prj/riptidesio/ebpf-tracing-demo"} 1 ``` This confirms that a tracepoint event was captured by the kernel, passed through eBPF to user space, and successfully transformed into a Prometheus metric via OTEL. ## Conclusion And that’s it—in a nutshell, this is how an event originating in the kernel can be transformed into meaningful, user-consumable telemetry. We started by writing a simple kernel module that emits a tracepoint event when a file is created. To validate it, we used the kernel’s built-in tracing infrastructure, just as we do when debugging telemetry at Riptides. Next, we built an eBPF program to stream those events into user space. Finally, we used OTEL to enrich the raw data with additional context and export it as Prometheus metrics. This wraps up our blog series on tracing and telemetry. Throughout the series, we’ve shared how Riptides built its observability stack, and in this final post, we walked through a working example that ties everything together. We encourage you to explore, tweak, or fork the demo to deepen your understanding of these powerful technologies. Thanks for following along—and see you next time with another exciting topic! --- ## The API Key Leaks Keep Coming - URL: https://blog.riptides.io/api-key-leaks-keep-coming - Published: 2025-07-17 - Author: Janos Matyas - Category: Security - Tags: ai, kernel, identity, x509 ## The API Key Leaks Keep Coming — Now It’s x.AI The latest high-profile API key leak has struck x.AI. A static API key granting access to internal AI models hosted on Groq’s infrastructure was accidentally committed to GitHub. As first reported by [Krebs on Security](https://krebsonsecurity.com/2025/07/doge-denizen-marko-elez-leaked-api-key-for-xai/), the key was live for several days, exposing valuable backend resources to anyone who stumbled upon it. This isn’t an isolated case. Just in the past few months, Hugging Face, Docker Hub, and even GitHub itself have suffered similar exposures. The trend is accelerating. The reason? Static keys are dangerously convenient. Developers, often under pressure and without deep security training, treat API keys as throwaway credentials, just another config value to copy-paste and commit. Once pushed to Git or dumped into a shell script, these secrets become ticking time bombs. ### Still the Norm: Static Keys and Shared Secrets Despite the growing list of breaches, most companies, even those with large engineering teams still rely on static API keys and shared secrets to authenticate workloads and services. These credentials are often embedded directly in source code, committed accidentally to Git, passed through environment variables, mounted via Kubernetes secrets, or stored in plaintext on local filesystems. In many cases, these secrets are shared across multiple services or teams, increasing the blast radius if even a single system is compromised. Even when secrets managers like AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager are used, the secret itself is still treated as a fixed asset, fetched at runtime and cached in memory or on disk. The assumption is that the environment is trusted enough to handle the secret securely. But in cloud-native systems where workloads are short-lived, containers are reused, and debugging access is widespread, this assumption often breaks down. Sidecar proxies and agents are sometimes used to pull credentials on behalf of the application, but these don’t eliminate the secret, they just move the trust boundary. Any process with access to the sidecar or the local network interface can potentially extract the token. The result is an illusion of security layered over a fundamentally fragile model: credentials that live too long, travel too far, and are visible to too many components. This model wasn’t built for today’s threat landscape. It assumes that secrets can be distributed and managed like configuration. But unlike config values, secrets have real security implications and handling them casually creates serious exposure. ### A Kernel-Native Alternative: Ephemeral Identity by Design At Riptides, we take a fundamentally different approach to workload identity, **[one that eliminates the need for static secrets altogether](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust)**. Our system dynamically generates ephemeral identities (X.509 certificates, JWT tokens, etc) at the point of need, scoped to a single process, and tied directly to kernel-level runtime context. Secrets are never written to disk, never passed via environment variables, and never exposed to userspace. Everything happens at the **[kernel level](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe)**, tightly bound to real-time metadata (like process ID, cgroup, network communication, location, binary hash, etc). The identity is injected only when the workload genuinely needs to call an external service, and only if the live behavior matches an authorized policy. This also allows us to enforce contextual access: a workload can’t just ask for a secret, it must be actively communicating with the intended destination, as verified by [kernel-level telemetry](/blog/linux-kernel-module-telemetry-beyond-the-usual-suspects). If a process tries to access a service it has no business talking to, the identity issuance is denied by design. No secret ever exists in a persistent or reusable form. We call this **[kernel-native identity](/blog/rethinking-workload-identity-at-the-kernel-level)**, and it’s built for the realities of modern infrastructure: distributed, dynamic, and under constant attack. No static credentials. No proxies. No sidecars. Just identity, embedded where it belongs — in the kernel. ![riptides-ui-identities](../../assets/api-key-leaks-keep-coming/illustration1.jpg) > Workload identities defined by Riptides ### Visibility: Monitoring Credentials in Motion Beyond identity issuance, Riptides actively monitors all inbound and outbound connections at the kernel level. This gives us real-time visibility into which workloads are communicating, where credentials are being transmitted (if at all), and whether any secrets are being exposed over the wire. By inspecting traffic metadata and correlating it with process-level identity, we build a live communication graph and enforce policies based on observed behavior. This approach allows us to detect when credentials are leaving expected boundaries, flag insecure usage patterns, and continuously assess the security posture of running systems — not just at deployment time, but throughout the workload’s lifecycle. ![riptides-ui-connections](../../assets/api-key-leaks-keep-coming/illustration2.jpg) > The Riptides UI in action, showing all monitored connections ### It’s Time to Kill the API Key The repeated leaks we’re seeing — from startups to hyperscalers — aren’t just accidents. They’re symptoms of a broken model: static secrets scattered across infrastructure, owned by no one, and visible to everything. The industry has outgrown this approach. At Riptides, we're building a future where identity is ephemeral, policy-aware, and deeply integrated into the kernel and where secrets don’t move because they don’t need to. Combined with live traffic monitoring and contextual enforcement, this gives teams a radically clearer view of what’s happening across their infrastructure and the power to stop misuse before it spreads. The sooner we **[stop treating credentials like config](/blog/the-riptides-vision-identity-first-infrastructure)**, the safer our systems will be. Let's kill the API key, and replace it with something that actually fits the world we run today. We've written extensively about our kernel-level architecture — from tracing and telemetry pipelines to eBPF-based observability and real-time identity enforcement. You can explore more in our #KERNEL blog series: - [Riptides: Kernel-Level Identity and Security Reinvented](/blog/riptides-kernel-level-identity-and-security-reinvented) - [From Breakpoints to Tracepoints: An Introduction to Linux Kernel Tracing](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing) - [From Tracepoints to Metrics: A journey from kernel to user-space](/blog/from-tracepoints-to-metrics-a-journey-from-kernel-to-user-space) - [Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) - [Linux kernel module telemetry: beyond the usual suspects](/blog/linux-kernel-module-telemetry-beyond-the-usual-suspects) --- ## Rethinking Workload Identity at the Kernel Level - URL: https://blog.riptides.io/rethinking-workload-identity-at-the-kernel-level - Published: 2025-07-14 - Author: Janos Matyas - Category: Kernel - Tags: vision, spiffe, identity, zero-trust, kernel, linux ## Beyond Sidecars and Proxies: Why the Linux Kernel Is the Future of Non-Human Identity As SPIFFE (Secure Production Identity Framework for Everyone) solidifies its role as the industry standard for workload identity, the question is no longer *whether* to adopt it — but *how*. SPIFFE provides a powerful abstraction for securely identifying workloads using cryptographic SVIDs (SPIFFE Verifiable Identity Documents), but the ways it’s currently deployed often fall short of the operational efficiency, portability, and security guarantees that modern enterprise infrastructure demands. In today’s distributed, cloud-native world, non-human identity is foundational to Zero Trust architectures. But the way these identities are issued and enforced — especially at scale — matters more than ever. And hiding in plain sight is a simpler, more universal and scalable foundation for workload identity: **the Linux kernel**. ## SPIFFE Today: It Works But at an Operational Cost Most SPIFFE deployments today fall into one of three patterns, all with real compromises. ### 1. Application-Level Integration This method embeds SPIFFE logic directly into workload code via libraries. In theory, it’s flexible. In practice: - It **burdens developers** with identity and security decisions that shouldn’t belong in application logic. - It relies on a **limited set of supported languages** and libraries. - It violates the principle of **separation of concerns**: developers should write business logic, not identity plumbing. Ultimately, pushing identity into application code turns a universal infrastructure concern into a fragmented implementation detail — fragile, hard to scale, and easy to get wrong. ### 2. Sidecars and Proxies (Including Service Meshes) In most modern SPIFFE deployments, identity is handled by an external agent or proxy either injected as a sidecar or embedded in a service mesh proxy like Envoy. These architectures have gained popularity for automating mTLS and simplifying SVID rotation, especially in Kubernetes environments. But they come with real drawbacks: - Kubernetes-only by default: These models rely **heavily on Kubernetes** primitives like sidecar injection, making them hard to adopt in VMs, bare metal, or edge deployments. - Heavyweight and resource-intensive: Every workload gets a proxy process, **adding compute, memory, and management overhead** across the fleet. - Indirection breaks identity fidelity: The **workload’s identity is no longer directly tied to the process initiating communication**, but to a neighboring proxy or agent. This weakens the trust boundary and opens the door to **identity confusion or lateral movement**. - Opaque security posture: Policies, credentials, and rotations happen in separate agents. Debugging or verifying behavior often requires understanding multiple moving parts across control and data planes. Sidecars and service meshes helped SPIFFE gain traction but they are scaffolding, not the foundation. As organizations scale into tens or hundreds of millions of workloads, these externalized identity layers introduce complexity, cost, and architectural drag. Kubernetes isn’t everywhere. Proxies aren’t everywhere. But the **Linux kernel is**. Whether your workloads run on VMs, containers, bare metal, or edge nodes, they all ultimately interface with the kernel opening sockets, issuing syscalls, launching processes. So instead of gluing SPIFFE on from the outside, **why not issue identities from the inside, at the source of execution itself?** ## The Riptides Approach: Identity Issued in the Kernel Riptides issues SPIFFE-compliant SVIDs as ephemeral **X.509 certificates directly inside the Linux kernel**, binding them to the actual **process** responsible for initiating communication. Using **kernel TLS (kTLS)**, we inject those identities directly into the TLS handshake at the record layer not through userspace, not via shared files, not through sidecar mediation. From the first packet, the workload’s identity is embedded, verified, and enforced. 🔍 Dive deeper: [Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) This model flips the script: - **No code changes**: Applications remain unaware of SPIFFE, and that's the point. - **No sidecars, no proxies**: You eliminate architectural bloat and performance tax. - **Works anywhere Linux runs**: Kubernetes, VMs, or bare metal, no platform lock-in. - **Strong process binding**: Identities are assigned per-process at the syscall level, not per-pod or per-node. - **Zero userspace dependency**: Secrets never touch disk or userspace memory. No gRPC APIs. No filesystem mounts. ![The Riptides Approach: Identity Issued in the Kernel](../../assets/linux-kernel-future-of-nhi/illustration1.png) ## How It Works: A High-Level Technical Overview Once Riptides is installed, the system operates transparently and automatically at the kernel level. 1. The Riptides user-space **daemon** receives SPIFFE identities, trust bundles, and connection policies from the control plane. 2. These are loaded into the kernel `riptides-driver` via **device driver communication**. 3. When a TCP connection is initiated, the driver: - Checks the destination against policy. - Initiates a full mTLS handshake (not just record protection) in the kernel using SPIFFE-based certificates. - Establishes a kTLS session for record encryption/decryption. This design has multiple benefits: - **Security:** Secrets never leave kernel memory, reducing exposure. - **Transparency:** Applications are unaware they’re speaking mTLS - zero code changes. - **Policy enforcement:** Fine-grained identity-based access control happens at connection time. There are **no persistent credentials**. Nothing to leak. No proxy to trust. It’s security rooted in the actual workload, at the lowest trustworthy layer. ![How It Works: A High-Level Technical Overview](../../assets/linux-kernel-future-of-nhi/illustration2.jpg) ## Kernel Development Is Not for the Faint of Heart Issuing and enforcing identity in the kernel isn’t just a bold technical decision, it’s what enables Riptides to deliver *per-process identity binding, zero credential leakage, and platform-agnostic enforcement* but it’s also a non-trivial engineering challenge. Unlike userspace software, kernel modules must operate across a wide landscape of **Linux distributions**, **kernel versions**, and **cloud provider variants**. Supporting even a small matrix of environments means contending with: - **Version fragmentation**: Different distros backport patches and modify kernel interfaces, making compatibility a moving target. - **Debugging complexity**: Traditional tooling breaks down in kernel space debugging often requires tracepoints, kprobes, custom logging infrastructure, or full crash analysis. - **Upgrade resilience**: Kernel module APIs and ABIs change over time, so staying ahead of updates requires continuous tracking and rigorous regression testing. - **Security scrutiny**: Operating inside the kernel means adhering to strict memory safety, syscall hygiene, and sandboxing rules — because any flaw has privileged blast radius. - **Distribution challenges**: To ensure wide adoption, kernel modules need to be shipped through the appropriate vendor ecosystems (e.g. Red Hat’s kmod programs, Amazon’s kernel module packaging workflows, or custom drivers for cloud-native runtimes). These aren’t theoretical hurdles, they’re real-world blockers that have historically limited the adoption of kernel-based solutions, especially for security-critical workloads. ## We've Done the Hard Work So You Don’t Have To At Riptides, we’ve built deep kernel expertise from the ground up, not just in developing high-performance, memory-safe modules, but in the **tooling, compatibility layers, and distribution channels** required to operate reliably across fleets. - We maintain a **continuous integration pipeline** that builds and tests our kernel module against dozens of kernel versions, including major distributions and LTS branches. - Our system integrates with **eBPF-based observability and tracepoints**, allowing us to diagnose issues and ensure correctness without disrupting workloads. - We participate in **kernel module packaging programs** across Red Hat, Amazon, and other cloud providers to ensure seamless, verified distribution. - And critically: **installation and upgrades are automatic and unobtrusive**. Riptides deploys as part of your existing provisioning process and silently adapts to the underlying system - no kernel patching, no user intervention. While kernel development is hard, **running Riptides is not**. You get all the benefits of syscall-level identity enforcement without having to touch the kernel yourself. We've written extensively about our kernel-level architecture — from tracing and telemetry pipelines to eBPF-based observability and real-time identity enforcement. You can explore more in our #KERNEL blog series: - [Riptides: Kernel-Level Identity and Security Reinvented](/blog/riptides-kernel-level-identity-and-security-reinvented) - [From Breakpoints to Tracepoints: An Introduction to Linux Kernel Tracing](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing) - [From Tracepoints to Metrics: A journey from kernel to user-space](/blog/from-tracepoints-to-metrics-a-journey-from-kernel-to-user-space) - [Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) - [Linux kernel module telemetry: beyond the usual suspects](/blog/linux-kernel-module-telemetry-beyond-the-usual-suspects) - [From Tracepoints to Prometheus: the journey of a kernel event to observability](/blog/from-tracepoints-to-prometheus-the-journey-of-a-kernel-event-to-observability) ## A Cleaner, Faster, Safer Future for Non-Human Identity Non-human identity is too foundational and too sensitive to be bolted on as an afterthought. It deserves a native, first-class treatment. By operating directly in the Linux kernel, Riptides removes layers of complexity and guesswork, grounding identity in the one place all workloads truly run. No proxies, no sidecars, no credential sprawl just **cryptographic trust, bound to the process itself**. The future of identity isn't just more secure. It's leaner, simpler, and built in from the start. And with Riptides, it’s already here --- ## Non-human identity federation with external IDPs: a guide for AWS, GCP, and Azure - URL: https://blog.riptides.io/federating-non-human-identities-with-external-idps-using-id-tokens-in-aws-gcp-and-azure - Published: 2025-07-07 - Author: Sebastian Toader - Category: Federation - Tags: federation, non-human identity, cloud In today’s cloud-native world, many applications and services no longer rely solely on human users to interact with infrastructure. Instead, **non-human identities** — such as automation agents, continuous integration (CI) pipelines, backend services, AI systems, and Internet of Things (IoT) devices perform many of these tasks autonomously. These non-human identities act similarly to users but represent software entities rather than people. Managing and securing these non-human identities is becoming increasingly complex, especially when your infrastructure spans multiple cloud providers like AWS, Google Cloud Platform (GCP), and Microsoft Azure. Traditional methods that rely on static credentials—like long-lived access keys or tokens—introduce significant security risks. These credentials are difficult to rotate regularly, can be accidentally leaked, and lack fine-grained control and visibility. This is where **ID tokens** come into play. An ID token is a cryptographically signed token issued by an identity provider that proves the identity of the requester. By leveraging **OpenID Connect (OIDC)**, you can use ID tokens to establish trust between your non-human identities and cloud providers securely. In this blog post, we will explore how to **federate non-human identities using an external OIDC-compliant Identity Provider (IdP)**. This federation approach allows your workloads to authenticate once with the external IdP and then securely access cloud resources across AWS, GCP, and Azure. You will learn how each cloud provider supports this federation, why establishing and maintaining trust is crucial, and how to design a system that provides temporary, auditable, and least-privilege access across clouds. Along the way, we will walk you through key components such as AWS Security Token Service (STS), GCP Workload Identity Federation, and Microsoft Entra ID (Azure AD), explaining how they fit into this architecture. ## What are non-human identities? When we talk about **non-human identities**, we refer to software-based entities that perform automated operations without direct human intervention. Examples include: - Continuous integration and deployment pipelines (CI/CD) - Backend microservices and APIs - Artificial intelligence and machine learning agents - IoT devices and edge compute nodes - Autonomous workflows and scheduled jobs As described by the [Riptides](https://riptides.io), these workloads: - Are not tied to any individual user or session. - Often operate across organizational or security boundaries. - Need strong cryptographic proof of their identity to be trusted. - Require fine-grained permissions and detailed auditability when accessing cloud APIs and resources. Historically, many organizations have relied on static credentials such as API keys or long-lived tokens for these workloads. However, these approaches suffer from serious drawbacks: - They are cumbersome to rotate and manage. - They increase the risk of credential leakage. - They provide little to no auditing or control over how credentials are used. **Federation** offers a better solution by allowing these workloads to delegate authentication to a trusted external identity provider. This shifts credential management to a centralized, secure system that issues short-lived, verifiable tokens, improving security and operational agility. ## Federation, trust, and external identity providers At the core of federated identity systems is the concept of **trust**. For cloud providers like AWS, GCP, and Azure to accept identity claims issued by an external IdP, there must be an explicit and secure trust relationship in place. Here’s how the pieces fit together: - The **external Identity Provider (IdP)** authenticates your non-human identities and issues them a signed **ID token** following the OIDC standard. This token contains verified claims about the identity, such as who they are and what they are allowed to do. - The **cloud provider** receives this ID token and validates its authenticity and integrity. It checks the token’s issuer, signature, audience, expiration, and relevant claims to confirm the identity. - The **trust relationship** between the cloud provider and the external IdP is explicitly configured. This includes specifying the trusted issuer URLs, accepted audiences, and claim mappings. Without this trust setup, the cloud provider will reject tokens from unknown or untrusted IdPs. ## Benefits of using federated identities for mon-human access Switching to federated identities for your workloads brings a wide range of security, operational, and compliance advantages compared to traditional static credentials. ### 1. Improved security posture - **Eliminates long-lived static credentials:** Static API keys or tokens are often stored in code repositories, configuration files, or environment variables — making them vulnerable to accidental exposure or theft. Federated identities rely on short-lived, cryptographically signed tokens that expire quickly, drastically reducing risk if a token is compromised. - **Cryptographic proof of identity:** ID tokens issued by trusted identity providers contain digitally signed claims that cloud providers can validate. This strong proof prevents impersonation and unauthorized access. - **Fine-grained access control:** Federation lets you map token claims to precise permissions in the cloud. This means wokrkloads get only the access they need — following the principle of least privilege — rather than broad or unlimited rights. ### 2. Simplified credential management - **Centralized identity provider:** Instead of managing separate credentials for every cloud environment or service, all authentication is delegated to a single external IdP. This simplifies onboarding, rotation, and revocation processes. - **Automatic credential rotation:** Because tokens are short-lived and dynamically issued on demand, you don’t need manual key rotations or complex secret distribution mechanisms. - **Unified audit trails:** All authentication activity funnels through the external IdP, making it easier to track who accessed what and when — vital for security audits and compliance. ### 3. Cross-cloud and cross-organization scalability - **Seamless multi-cloud access:** With federation, your non-human identities authenticate once against your external IdP and then can gain access to multiple cloud providers without juggling separate credentials for each. - **Supports complex trust boundaries:** If your infrastructure spans multiple teams, business units, or partner organizations, federated identities help you enforce trust and permissions consistently while respecting boundaries. ### 4. Better developer and operator experience - **Reduced operational overhead:** Developers and DevOps teams no longer have to handle cumbersome credential management or worry about secrets leaking in logs or repositories. - **Easier integration:** Many cloud-native tools and platforms now natively support OIDC federation, allowing smoother onboarding and less custom glue code. In summary, adopting federated identities is a foundational step toward a modern, secure, and scalable cloud infrastructure that respects best practices for identity and access management. The security, operational efficiency, and flexibility benefits compound quickly — especially as you expand your cloud footprint or embrace automation and AI-driven workflows. ## Setting up federation with AWS, GCP, and Azure Now that we’ve covered the core concepts and benefits of federating non-human identities with an external Identity Provider, let’s look at how to put this into practice. Each major cloud provider—**AWS**, **Google Cloud Platform (GCP)**, and **Microsoft Azure**—offers its own mechanisms and services to establish trust with external OIDC-compliant IdPs and to securely consume ID tokens for access control. In the following sections, we will explore the key components, configuration steps, and best practices for setting up federation on each platform: - How AWS uses Security Token Service (STS) to assume roles based on external ID tokens. - How GCP’s Workload Identity Federation enables token exchange without long-lived service account keys. - How Azure Entra ID (Azure AD) supports external identity providers for workload authentication. By understanding these workflows, you will be equipped to implement secure, scalable cross-cloud identity federation tailored to your infrastructure. ## 1. AWS: Web Identity Federation with OIDC AWS enables federated non-human identity by using **IAM roles for Web Identity** in combination with **OIDC identity providers**. This lets you delegate authentication to an external IdP and securely grant temporary permissions to workloads based on ID tokens. **Setup steps**: 1. **Create an OIDC identity provider in IAM** This step establishes a trusted relationship between AWS and your external Identity Provider. By registering the IdP’s issuer URL, client IDs, and certificate thumbprint, AWS can validate the ID tokens it receives during authentication. ```bash aws iam create-open-id-connect-provider \ --url "https://example-idp.com/oidc" \ --client-id-list "my-client-id" \ --thumbprint-list "9e99a48a9960b14926bb7f3b2e5e5b9e7e5e5e5e" ``` > **Note:** To get the certificate thumbprint required here, you can use OpenSSL as follows: > > ```bash > openssl s_client -connect example-idp.com:443 openssl x509 -fingerprint -noout | \ > sed 's/SHA1 Fingerprint=//' | tr -d ':' | tr 'A-Z' 'a-z' > ``` 2. **Create an IAM role with a Trust Policy for the external IdP** Next, create an IAM role that your federated identities will assume. The trust policy explicitly grants permission for AWS to accept ID tokens issued by your external IdP for this role. It defines which identities (based on claims like `sub` and `aud`) are allowed to assume the role. ```bash aws iam create-role \ --role-name my-oidc-role \ --assume-role-policy-document file://trust-policy.json ``` Example `trust-policy.json`: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/example-idp.com/oidc" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "example-idp.com/oidc:sub": "my-workload-identity", "example-idp.com/oidc:aud": "my-client-id" } } } ] } ``` 3. **Attach IAM policies to the role to define permissions** Attaching policies to the role defines what actions the federated identities can perform once authenticated. This step ensures that the identities have the necessary permissions to access AWS resources, such as read-only access to Amazon S3 in this example. ```bash aws iam attach-role-policy \ --role-name arn:aws:iam::943962173050:role/my-oidc-role \ --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess ``` 4. **Assume the role using the ID Token** Finally, your non-human identity uses the OIDC ID token obtained from the external IdP to assume the IAM role and receive temporary AWS credentials with the permissions defined above. This can be done via the AWS CLI: ```bash aws sts assume-role-with-web-identity \ --role-arn arn:aws:iam:::role/my-oidc-role\ --role-session-name my-session \ --web-identity-token "" ``` or programmatically via AWS SDKs. **Authentication sequence diagram:** ![Authentication sequence](../../assets/cloud-provider-nhi-federation/aws_fed.png) ## 2. GCP: Workload Identity Federation Google Cloud Platform provides a robust way to federate external identities through **Workload Identity Pools** and **Providers**. This enables non-human identities authenticated by an external OIDC IdP to access GCP resources without requiring long-lived service account keys. **Setup steps**: 1. **Create a Workload Identity Pool and an OIDC provider** The Workload Identity Pool acts as a container for external identities, while the Provider configures the trust relationship with your external IdP by specifying its issuer URL and allowed audiences. This setup allows GCP to accept and validate ID tokens issued by your IdP. ```bash gcloud iam workload-identity-pools create my-pool \ --project="my-project" \ --location="global" \ --display-name="My Pool" gcloud iam workload-identity-pools providers create-oidc my-provider \ --project="my-project" \ --location="global" \ --workload-identity-pool="my-pool" \ --display-name="My Provider" \ --issuer-uri="https://example-idp.com/oidc" \ --allowed-audiences="my-client-id" \ --attribute-mapping="google.subject=assertion.sub" ``` 2. **Grant the workload identity user role to allow identities from the pool to impersonate a GCP service account** This step links your external identities to a specific GCP Service Account. By granting `roles/iam.workloadIdentityUser` on the service account to the external identity (via the workload identity pool and claim), you enable the federated identity to act as the service account and inherit its permissions. ```bash gcloud iam service-accounts add-iam-policy-binding "my-service-account@my-project.iam.gserviceaccount.com" \ --project="my-project" \ --role="roles/iam.workloadIdentityUser" \ --member="principal://iam.googleapis.com/projects//locations/global/workloadIdentityPools/my-pool/subject/my-workload-id" ``` 3. **Authenticate using the external ID token to impersonate the service account** Instead of using a long-lived key file, you exchange your external OIDC ID token for short-lived GCP credentials tied to the service account. This can be done via the Google Cloud SDK’s gcloud CLI or programmatically using Google’s client libraries. Here is an example using the gcloud CLI to authenticate with the external token. ```bash gcloud auth login --cred-file cred-config.json ``` Example `cred-config.json` ```json { "universe_domain": "googleapis.com", "type": "external_account", "audience": "//iam.googleapis.com/projects//locations/global/workloadIdentityPools/my-pool/providers/my-provider", "subject_token_type": "urn:ietf:params:oauth:token-type:jwt", "token_url": "https://sts.googleapis.com/v1/token", "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/my-service-account@my-project.iam.gserviceaccount.com.com:generateAccessToken", "credential_source": { "file": "token.jwt", "headers": {}, "format": { "type": "text" } } } ``` > Save the actual OIDC token issued by your external identity provider into `token.jwt`. **Authentication sequence diagram:** ![Authentication sequence](../../assets/cloud-provider-nhi-federation/gcp_fed.png) ## 3. Azure: Federated Identity with User-Assigned Managed Identity (UAMI) Microsoft Entra Workload Identity Federation now supports **federating external non-human identities to a User-Assigned Managed Identity (UAMI)**. This allows workloads outside Azure — such as CI/CD pipelines or automation tools — to authenticate to Azure without using client secrets. Azure AD supports federated credentials via **App Registrations**. ### Setup steps 1. **Create a User-Assigned Managed Identity** ```bash az identity create \ --name my-uami \ --resource-group my-rg \ --location westeurope ``` Get the identity's client ID and resource ID: ```bash az identity show --name my-uami --resource-group my-rg \ --query "{clientId:clientId, id:id, principalId:principalId}" ``` 2. **Add a federated identity credential** This step connects the external IdP to the UAMI by configuring Entra to accept signed ID tokens from your external system. ```bash az identity federated-credential create \ --name my-federated-cred \ --identity-name my-uami \ --resource-group my-rg \ --issuer https://example-idp.com/oidc \ --subject "my-workload-identity" \ --audiences "api://AzureADTokenExchange" ``` 3. **Assign roles to the UAMI** Use Azure RBAC to grant the UAMI access to Azure resources. For example, to allow read access to your subscription: ```bash az role assignment create \ --assignee-object-id \ --assignee-principal-type ServicePrincipal \ --role Reader \ --scope /subscriptions/ ``` 4. **Use an external OIDC-issued ID token to get an Azure access token** Once your external identity has received an ID token from the external IdP, it can authenticate to Azure using az login and the `--federated-token` flag. This allows your workload to assume the federated **User-Assigned Managed Identity (UAMI)** without using a client secret. ```bash az login \ --service-principal \ --username \ --tenant \ --federated-token ``` > Replace `` with the UAMI's client ID, `` with your Azure AD tenant ID, and `` with the actual OIDC token issued by your external identity provider. **Authentication sequence diagram:** ![Authentication sequence](../../assets/cloud-provider-nhi-federation/azure_fed.png) ## Comparison of Federation across AWS, GCP, and Azure | Feature | **AWS** | **GCP** | **Azure** | | ----------------------------- | -------------------------------------- | --------------------------------------------- | --------------------------------------------- | | **Federation Mechanism** | IAM Role with Web Identity (STS) | Workload Identity Federation | Entra Workload Identity Federation (UAMI/App) | | **Claim Mapping Flexibility** | Basic – only `sub`, `aud` | Advanced – full attribute mapping supported | Basic – `subject`, `aud`, `issuer` | | **Session Duration Control** | Yes | Yes | Yes | | **OIDC Provider Reusability** | Yes – multiple roles per provider | Yes – pool-wide reuse | Yes – multiple federated credentials per UAMI | | **Auditing Support** | CloudTrail + IAM Access Analyzer | Cloud Audit Logs + IAM insights | Microsoft Entra Audit Logs | | **Identity Granularity** | One role per trust policy | Fine-grained via attribute selectors | Scoped via federated credentials per UAMI | ## Best practices To ensure secure and scalable federation for non-human identities, follow these guidelines: - **Use short-lived tokens**: Keep token lifetimes as short as practical to reduce the risk from token leakage or misuse. - **Restrict token claims**: Match on precise subjects and audiences to avoid accepting tokens from unintended sources. - **Minimize role scope**: Grant only the minimal permissions required using tightly scoped IAM roles or service account bindings. - **Monitor and audit usage**: Leverage cloud-native audit logs to track token issuance, role assumptions, and access behavior. - **Rotate identity provider metadata**: Plan for regular updates to OIDC metadata (keys, issuers, thumbprints) to handle IdP changes securely. ## Final thoughts Federating non-human identities is no longer optional — it's essential for operating securely in today’s multi-cloud environments. By adopting **OIDC-based federation with ID tokens from a trusted external Identity Provider**, you enable: - Ephemeral, automatically expiring credentials - Centralized identity lifecycle management - Fine-grained access control - Full auditability of automated access The true power of federation lies in how well you define **trust relationships and claim mappings**. A CI/CD job running in GitHub Actions should not inherit the same access as a production AI pipeline — even if both use tokens issued by the same IdP. As the number of machine actors grows, **federation becomes a pillar of secure identity architecture**. Build with it early, use it consistently — and revisit your trust boundaries often. ## Looking ahead At Riptides, we’re building a system that treats non-human identity as a first-class citizen in cloud-native environments. Riptides issues and manages SPIFFE-based workload identities, assigning them to services and agents at runtime. When needed, these identities can be represented as OIDC-compatible ID tokens, allowing them to act as federated identities with major cloud providers. Under the hood, these ID tokens are exchanged for short-lived, cloud-native credentials — such as AWS STS credentials, GCP access tokens, or Azure Entra tokens — and securely delivered to the workload, with no static credentials required. In a follow-up post, we’ll explore how Riptides enables this end-to-end federation flow, and how it helps teams eliminate manual credential handling while maintaining security, auditability, and least-privilege access across cloud providers. --- ## Securing MCP Communication with Riptides - URL: https://blog.riptides.io/securing-mcp-communication-with-riptides - Published: 2025-06-30 - Author: Zsolt Rappi - Category: MCP - Tags: ai, linux, identity, mcp In our [previous blog post about MCP](/blog/mcp-a-quickstart-guide), we introduced the core concepts behind this agentic technology. Today, we’ll explore how the **Riptides Non-Human Identity (NHI) platform** helps secure agentic applications that use the **Model Context Protocol (MCP)**. As a reminder, MCP servers can be either **local** or **remote**: - **Local servers** are spawned as subprocesses by the agentic app and typically require environment variables to configure access to restricted resources. - **Remote servers** require OAuth2 authentication so users can grant access to their resources. Riptides secures both scenarios using our **[kernel-based identity platform](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe)**. It provides an OS-level enforcement layer that transparently controls how agents, MCP servers, and backends securely communicate with zero changes to application code. ## Why MCP Communication Is Risky Agent-to-MCP communication introduces a new security boundary that is vulnerable to: - Prompt injection - Impersonation - Token leakage - Unauthorized access MCP is designed to decouple agents from the implementation details of tools and services. Agents discover and invoke tools from MCP servers at runtime, sometimes across the network. While this enables powerful agentic workflows, it also introduces several risks: - Agents may interact with malicious or misconfigured servers. - Servers might call untrusted or malicious endpoints. - Local servers need secrets like API keys or tokens via environment variables; using a malicious or compromised server could lead to credential leaks. ## How Riptides Secures MCP Traffic ### Kernel-Level TLS Riptides transparently upgrades outbound traffic to mTLS. The agent remains unaware of handshakes, certificates, or [trust anchors](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust), Riptides seamlessly handles everything at the [kernel level](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe). ### Policy Enforcement You can define policies that specify which MCP servers an agent is allowed to communicate with (based on SPIFFE IDs) and deny all others by default. This enables a **zero-trust, default-deny** network model. ### Secret Injection Based on the above mentioned policies, Riptides can inject the necessary tokens and API keys and modify requests at the kernel level, as communication pass by. This way users space applications never have access to such sensitive values, yet the connection still work. ### Auditability All identity assertions and network requests are traceable back to the specific agent and its runtime environment. This is essential for debugging, compliance, and forensic analysis. When both client and server use [SPIFFE](/blog/introduction-to-spiffe-secure-identity-for-workloads) aware environments, **mutual TLS provides a strong, verifiable handshake**, ensuring trust on both sides before any tool call is made. ## Example Scenarios ### Remote An agent running in a secure environment (e.g., a container in a Kubernetes cluster) needs to invoke a tool hosted on a third-party remote MCP server: `https://mcp.example.com/tools/send_email`. This MCP server might: - Forward requests to internal or external APIs (e.g., SMTP providers, notification systems) - Require OAuth2 tokens authorize outgoing traffic - Be managed by third party team or vendor If both client and server use Riptides to secure their environments, then: - Both the agent and the server get a **SPIFFE-compliant identity** bound to their respective workloads (e.g., Kubernetes namespace, process name, deployment name, labels, etc.). - All outgoing requests from the agent to the MCP server are **automatically upgraded to mutual TLS** using [kernel level](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) enforcement, no code changes required. - The MCP server is also required to present a valid [SPIFFE](/blog/introduction-to-spiffe-secure-identity-for-workloads) identity before any tool call is permitted. - A **policy engine** verifies that: - The destination MCP server is on an allowlist - The presented identity matches a trusted SPIFFE ID - The protocol is mTLS with verified [certificate chains](/blog/x-509-certificates-in-the-age-of-spiffe-and-zero-trust) - All requests are **auditable**, with logs mapping agent workload identity to each outbound call. This provides zero-trust enforcement even across organizational or network boundaries. ### Local MCP Server In the local scenario, the MCP server is spawned as a subprocess by the agent and communicates via stdio. While this communication is considered safe within a trusted runtime, the **security challenge lies in the MCP server's outbound traffic** to third-party services (e.g. APIs, cloud tools). Traditional approach: - Secrets (API keys, tokens) are injected into the MCP server as **environment variables**. - The server uses those secrets to make authenticated requests (e.g., `curl -H "Authorization: Bearer $API_KEY"`). This is risky because: - Any vulnerability in the MCP server could expose those environment variables. - Secrets may persist in memory, logs, or process introspection tools (`ps`, `/proc`). **With Riptides:** - **No secrets are passed via environment variables.** - When the MCP server initiates a request (e.g., via `libcurl`, `requests`, or `fetch`), Riptides intercepts the call at the socket level. - Based on the outbound domain/IP and request metadata, Riptides injects: - **Authorization headers** (e.g., Bearer tokens) - **mTLS client certificates** - **Custom headers or request modifications** - The secrets are never visible to the MCP server process. ## Best Practices for Securing Agent-to-MCP - **Use SPIFFE based identities for all agents and MCP servers** Avoid relying on static API keys or long-lived secrets. Use ephemeral, cryptographically backed identities issued at runtime. - **Enforce mTLS in remote scenarios** Prevent unauthorized access, impersonation, and data interception. - **Hide secrets from servers in local scenarios** Inject secrets at the kernel level, rather than exposing them through environment variables. - **Scope agent permissions with policy** Define exactly which MCP servers or tools an agent can communicate with. - **Trust no MCP server by default** Only allow connections to explicitly whitelisted servers. Avoid dynamically connecting to unknown endpoints. ## Conclusion: A New Trust Layer for Autonomous Agents MCP unlocks powerful agentic capabilities but only if agent-to-server communication is secure. By combining MCP’s standardized tool interface with SPIFFE based identity using Riptides, teams can: - Build composable, multi-agent systems - Leverage third-party tools with confidence - Maintain compliance and auditability - Enforce zero-trust principles at every boundary As AI agents grow more autonomous, **identity becomes the root of security**. Riptides delivers that identity without the complexity. --- ## X.509 Certificates in the Age of SPIFFE and Zero Trust - URL: https://blog.riptides.io/x-509-certificates-in-the-age-of-spiffe-and-zero-trust - Published: 2025-06-23 - Author: Zsolt Varga - Category: Security - Tags: spiffe, x509, workload-id In a world where secure communication between services is critical, identity is the new perimeter, and X.509 remains its cornerstone. At Riptides, we deal with this challenge daily as we build a [kernel-level](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) identity fabric for non-human actors in dynamic environments. Today’s reality is that machines vastly outnumber humans, and most authentications no longer happen at login screens, they occur silently between services, containers, jobs, and functions. This shift demands a secure, scalable way to establish non-human identity. This post explores why X.509 remains relevant, what its internal structure looks like, how its trust model operates, and where it struggles in modern environments. It also examines how Riptides, with the help of technologies like **SPIFFE** built on top of X.509 to support scalable, secure non-human identity. In this post, we’ll cover: - Why X.509 is still the foundation of trust - Its structure and how it maps to identity - The role of the certificate chain - Why it's complicated in modern systems - How projects like **SPIFFE** aim to fix it ## Trusting Machines, Not Just People Humans authenticate using passwords, tokens, or biometrics. Machines don’t have that luxury. They need **cryptographic credentials** that are: - Unique and verifiable - Automatically issued - Short-lived and rotated - Verified by other machines at connection time This is where **X.509** certificates come in. They provide a well-established format for **binding a public key to an identity**, signed by a trusted party. ## X.509 in a Nutshell An X.509 certificate is a digital document with a clear job: **prove who you are** by presenting a signed statement from someone trusted. At a minimum, a certificate includes: - A **public key** - A **subject name** (the identity) - A **validity period** (not before / not after) - A **signature** from a trusted issuer (a CA) - Optional **extensions**, like key usage or subject alternative names (SANs) Optional extensions let you specify: - Usage constraints - Additional identities (via **SAN** fields) - Whether the cert can issue others (**CA flag**) ## Anatomy of an X.509 Certificate Here’s an example of a **short-lived workload identity certificate** issued for a SPIFFE identity: ```shell $ openssl x509 -in workload.pem -text -noout Subject: URI:spiffe://acme.corp/production/service-a Issuer: CN = intermediate-ca.acme.corp Validity Not Before: Jun 17 08:00:00 2025 GMT Not After : Jun 17 09:00:00 2025 GMT Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) X509v3 extensions: X509v3 Subject Alternative Name: URI:spiffe://acme.corp/production/service-a X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication ``` ### Key Observations - The identity is encoded as a **URI SAN** (Subject Alternative Name) as per the [SPIFFE spec](https://github.com/spiffe/spiffe/blob/main/standards/X509-SVID.md) - The cert is valid for **1 hour** — much shorter than traditional certs - It’s signed by an **intermediate CA**, part of a trust domain - It’s used for **mutual TLS** authentication (server and client roles) This certificate would typically be presented by a workload in a **mTLS handshake**, where both sides validate each other. ## The Chain of Trust An X.509 certificate chain is a sequence of certificates: ``` Leaf Certificate (workload identity) signed by Intermediate CA (optional) signed by Root CA (trusted by the system) ``` Trust is established only if: 1. Every certificate is correctly signed by the next in the chain. 2. The root certificate is **explicitly trusted** by the verifier. 3. The leaf certificate meets all identity, usage, and time constraints. ## Structure of a SPIFFE-Aligned Chain ### Root CA - **Typically self-signed** - CA:TRUE in Basic Constraints - Trusted out-of-band via a **trust bundle** - Not used directly to sign workload certs ### Intermediate CA (Optional) - Signed by the root CA - CA:TRUE - Used to issue leaf certificates - Must include `keyCertSign` in `keyUsage` ### Leaf Certificate (X.509-SVID) - Signed by intermediate (or root, if no intermediate exists) - CA:FALSE - Must include the SPIFFE ID as a **URI SAN** - Must include proper key usage: - `digitalSignature`, `keyEncipherment` - Extended key usage: `clientAuth`, `serverAuth` - Subject DN is ignored (can be empty) - Short-lived (often ≤ 1 hour) ### How validation works in practice 1. **Extract the leaf and any intermediates** from the presented chain. 2. **Load trusted root CA(s)** from the verifier's trust bundle. 3. **Build a valid path** from leaf → intermediate(s) → root. 4. Validate: - Signature integrity - Validity periods - Basic and extended key usage - CA:TRUE/CA:FALSE constraints - SPIFFE URI in SAN - SPIFFE **trust domain** matches expectations ### Common Pitfalls | Problem | Cause | Solution | |----------------------|--------------------------------------------------|------------------------------------------| | "Unknown authority" | Intermediate CA missing from presented chain | Bundle intermediate with leaf | | No SPIFFE URI | SPIFFE ID missing from SAN | Ensure SPIFFE URI is encoded as URI SAN | | Expired certificate | Clock skew or cert not rotated | Use short-lived certs, rotate frequently | | Wrong key usage | Missing `clientAuth` or `serverAuth` | Fix EKU in certificate | | Invalid trust domain | SPIFFE URI does not match verifier’s expectation | Enforce domain-level validation | This model is at the heart of TLS, mTLS, Kubernetes, and modern zero trust infrastructure. ### Trust Domain Enforcement The SPIFFE URI must match the verifier's expected **trust domain**: ``` spiffe://acme.corp/production/service-a ``` Validation must ensure: - The **scheme** is `spiffe` - The **host** matches the trust domain (e.g., `acme.corp`) - The path is application-specific but stable and deterministic ### Root Certificate Handling Root CAs are not transmitted during TLS handshakes. They must be: - Distributed out-of-band - Loaded into verifier’s trust bundle - Anchors for the certificate path validation ## Why This Is Hard at Scale X.509 was designed for large, centralized ecosystems — not for ephemeral workloads in a Kubernetes cluster, or serverless jobs spun up for milliseconds. In modern environments: - Certificates need to be **issued automatically** - They must be **short-lived** (hours, not months) - Revocation is unreliable; rotation is preferred - Identity must map to **workload identity**, not just DNS names These aren’t edge cases. They’re the new normal. ## Enter: SPIFFE and the Modern Identity Stack That’s why efforts like the **SPIFFE (Secure Production Identity Framework For Everyone)** standard exist. SPIFFE defines a platform-agnostic identity model for workloads, built on proven primitives like X.509 but optimized for today's environments. Under SPIFFE: - Every workload gets a **SPIFFE ID** (a URI, e.g., `spiffe://acme.corp/production/service-a`) - The ID is encoded in a **short-lived X.509 certificate** - Trust domains are clearly defined - Rotation and issuance are handled automatically This is **non-human identity done right**: secure, dynamic, and abstracted from IPs, DNS, and human error. ## The Caveats (a.k.a. Why We're Building This) Even with standards like SPIFFE, integrating X.509 at scale is **non-trivial**: - Most libraries and tools weren’t designed for automatic cert handling - Certificate parsing and validation errors are often silent — or worse, misleading - Debugging mTLS failures can feel like black magic - Developers shouldn’t have to care about DER, SANs, or CRLs — but often do That’s where **Riptides** come in. The mission is to **make non-human identity effortless** — by embracing the solid foundation of X.509, while abstracting away its complexity. You get: - Transparent identity issuance and rotation - Secure propagation [below the application layer](/blog/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe) - PKI operations without needing to be a PKI expert Because the future of trust doesn’t just belong to people. It belongs to the services that talk to each other — a million times a second. ## Final Thoughts X.509 isn’t modern, but it’s battle-tested. It works across every major platform and protocol. When combined with modern identity models like SPIFFE and automatic issuance Riptides provides, it becomes a powerful foundation for service-to-service trust. Riptides is here to make that trust simple and reliable, especially for non-human actors. **Interested in how we are doing it?** Follow us for deep dives on mTLS, SPIFFE, and building zero trust identity infrastructure the modern way. The move to zero trust isn’t optional, and it doesn’t stop at user identity. With Riptides and SPIFFE, X.509 becomes a reliable engine for workload trust at scale. Embedded, automated, and built for the next generation of infrastructure. --- ## MCP: A Quickstart Guide - URL: https://blog.riptides.io/mcp-a-quickstart-guide - Published: 2025-06-14 - Author: Zsolt Rappi - Category: MCP - Tags: ai, linux, identity, mcp Today, we’ll explore **MCP (Model Context Protocol)**, a concept that’s gained serious traction in the agentic space over the past six months. This post will walk you through what MCP is, when to use it, and key security pitfalls to watch out for. ## What is MCP? LLM-based applications are growing increasingly sophisticated and complex. That’s largely because most top-tier LLMs now support larger contexts, allowing engineers to feed more data into prompts. This unlocks the ability to build agentic applications that go far beyond basic LLM wrappers: they can make decisions, interact with tools or other agents, and act on behalf of users. As these apps evolved, a clear need emerged: we needed a protocol to standardize how they interact with external resources. That’s where **MCP** comes in. It’s designed to be lightweight, straightforward, and easy to implement. MCP follows a client-server architecture: the app is the client, and the server exposes tools, resources, prompts, and more in a consistent format. Then, the LLM can decide which resource to use at runtime to complete its task. This setup allows developers to spin up new MCP servers and share them with the community or connect to existing ones. There’s no language dependency, servers and clients can be written in whatever language you prefer, as long as they adhere to the protocol’s standards. In this post, we’ll focus primarily on the architecture of the protocol and what you should know before implementing your first MCP-compliant agent. For deeper technical details, check out the [official docs](https://modelcontextprotocol.io/). ## Architecture MCP defines the endpoints a server must expose, the expected request/response message formats, and how data is transmitted between client and server. It’s based on **JSON-RPC**, and the protocol was intentionally kept minimal to encourage rapid adoption. We're not diving deep into the specifics of endpoints and messages here, but a few core examples are: `tools/list`, `resources/list`, and `tools/call`. They're straightforward and easy to reason about. At the heart of MCP's architecture is one of its most important components: **transports**. These define how messages are transmitted between client and server. Out of the box, MCP supports two transport types: **stdio** and **Streamable HTTP**, but you're free to implement others, just make sure both client and server support them. ### stdio This is the most widely supported transport among open-source MCP servers today. Here, the client spawns the server as a subprocess and communicates via standard input/output streams. No HTTP, no complex auth, just simple, effective messaging. API keys or other credentials are passed in as environment variables. ### Streamable HTTP This one adds a layer of complexity. Imagine the server and client aren’t on the same machine, and the server needs to support multiple concurrent clients. Now you have to consider authentication methods, API key handling, and whether to support human-in-the-loop flows for login. The first version of MCP didn’t address this. But as demand grew, the community asked for a standardized way to handle remote server authentication. The maintainers responded quickly, introducing authentication standards in the next release. MCP servers using Streamable HTTP are now expected to be fully **OAuth2** compliant. This was a smart choice, OAuth2 is mature, well-understood, and widely adopted. So now, MCP clients just need to implement the standard OAuth2 flow to talk to remote servers: - Register the client to get a Client ID from the authorization server. - Redirect the user to log in. - User authorizes the client. - Client receives access tokens and can start making authenticated calls. We’re glossing over a lot of OAuth2 details here, but the key point is this: by using OAuth2, authentication is offloaded to an existing, widely used standard. But there’s a catch. This only works if the access token granted to the MCP client is also accepted by the backend APIs that the MCP server will call. If not, the client can talk to the server, but the server can't reach the required backends. For this setup to work, the authorization server used by the MCP server must also be accepted by those backend services. Let’s walk through a real-world example. Imagine Google wants to expose a Gmail MCP server. They’d need to support OAuth2, no problem, Gmail already does. They could deploy the MCP server to use Gmail’s existing authorization server. So when a client interacts with the Gmail MCP server: - The user is redirected to Gmail’s login page. - They log in and authorize the client. - The client receives OAuth2 tokens. - The client can now use the Gmail MCP server to send emails, read inboxes, etc., because the Gmail backend will also accept the access tokens issued by the auth server. Again, this only works because the MCP server and Gmail’s backend share the same auth infrastructure. ## Clients Now that we've covered servers, let’s talk about MCP clients. As mentioned earlier, MCP is primarily useful for agentic applications. There’s no hard definition of what counts as "agentic," so we’ll use this one: if your app uses LLMs with tool-calling capabilities to solve complex tasks on demand, it’s agentic. These are the apps that benefit from MCP. They act as clients, connecting to servers that expose the necessary tools and resources. MCP client SDKs are already available in multiple languages and are easy to integrate. Just wire MCP server calls into your app using the SDKs, pass the available tools and resources to your LLM, and let it decide which one to call. That’s it. There's already a growing list of community-maintained servers at [mcpservers.org](https://mcpservers.org/), so integrating MCP into your app opens up a lot of possibilities. If you want to experiment with MCP without writing your own agentic app, you can use **Claude Desktop**, **VS Code with Copilot**, or any of the [following apps](https://modelcontextprotocol.io/clients) that support MCP. Just configure a few servers in a config file, and you’re good to go. Here’s an example `mcp.json` config you can drop into your `.vscode` folder. It defines both local and remote MCP servers your agent can connect to: ```json { "servers": { // Local servers // https://github.com/modelcontextprotocol/servers/tree/main/src/git "git": { "command": "uvx", "args": ["mcp-server-git", "--repository", "path/to/git/repo"] }, // https://github.com/modelcontextprotocol/servers/tree/main/src/fetch "fetch": { "command": "uvx", "args": ["mcp-server-fetch"] }, // Remote servers "neon": { // source: https://neon.tech/docs/ai/neon-mcp-server "command": "npx", "args": [ "-y", "mcp-remote", "https://mcp.neon.tech/sse" ] }, "paypal": { // source: https://developer.paypal.com/tools/mcp-server/ "command": "npx", "args": [ "-y", "mcp-remote", "https://mcp.paypal.com/sse" ] } } } ``` Since Copilot doesn't natively support remote servers via Streamable HTTP, we use [`mcp-remote`](https://github.com/geelen/mcp-remote), a lightweight proxy that bridges stdio-based clients with remote servers. It handles OAuth2 flows, manages client credentials and tokens, and translates communication across transport layers. Also worth noting: many remote MCP servers still use **SSE (Server-Sent Events)** rather than the newer **Streamable HTTP** format. That’s because the initial MCP auth spec used SSE; Streamable HTTP was introduced later as a more general replacement, and adoption is still catching up. ## Security Considerations As MCP adoption grows, it's important to think critically about the security risks of integrating third-party servers, especially in production environments. ### Code Security Keep in mind, many MCP servers require to pass in API keys or handle OAuth tokens and other secrets at runtime. This means you are trusting the server with priviliged access. This trust shouldn't be default, because: - This technology is still very new, a lot of the open source servers are not mature enough for production use. - Many of them are just side projects without security hardening. - There are already hundrends of servers out there, with more appearing daily. Treat them as untrusted by default and make sure you do your research before integrating them into your app. ### Prompt Injection Since LLMs are responsible for choosing which tool to call, they are prone to prompt injection attacks from two sides: - A malicious user prompt could manipulate the model into invoking a tool it shouldn’t. - A malicious tool response could bias the LLM's future actions. These risks could be mitigated using proper validation middlewares on both sides and by using trusted MCP servers only. ### Client-Server Trust Boundaries Whether you're using local or remote MCP servers, establishing trust between the agent and the server is critical: - In a local setup, you need to ensure that only trusted processes are spawned. This prevents rogue processes from impersonating MCP servers. - With remote servers, you need clear controls over which servers the agent can connect to. In both cases, trust boundaries should be configurable without hardcoding them into the application. This is where **non-human identities (NHIs)** become essential. Riptides’s kernel-level, [SPIFFE](https://spiffe.io)-based identity system offers a strong zero-trust foundation for agent/server authentication and policy enforcement. We cover this in more depth in our [previous post](/blog/non-human-identity-done-right-why-ai-agents-need-spiffe-and-riptides). ## Conclusion MCP is rapidly becoming the backbone of how LLM-based agents interact with structured, external tools. If you’re building apps where LLMs do more than just chat, apps that act, fetch, call, and decide, MCP is worth integrating now. As you adopt this protocol, don’t overlook the security implications of agentic autonomy. Riptides provides a zero-trust foundation for machine and non-human identity, and policy enforcement, ensuring that your agents only talk to what they’re supposed to, and nothing else. Stay tuned for deeper dives into how we’re securing this fast-moving landscape. --- ## Linux kernel module telemetry: beyond the usual suspects - URL: https://blog.riptides.io/linux-kernel-module-telemetry-beyond-the-usual-suspects - Published: 2025-06-10 - Author: Sebastian Toader - Category: Telemetry - Tags: kernel, telemetry, tracing At Riptides, we’re building an identity fabric that issues ephemeral, SPIFFE-based identities to workloads and AI agents, enforced entirely in the Linux kernel. This kernel level enforcement gives us deep control and visibility over service-to-service communications, but it also means traditional observability tools fall short. To operate confidently at this layer, telemetry isn't optional, it's foundational. It allows us to track how non-human identities behave at the kernel level, verify that policies are being enforced as expected, and correlate identity driven activity with system-wide behavior in real time. ## Telemetry: the backbone of modern software operations Telemetry—metrics, traces, and logs are fundamental to building and operating reliable software systems. For engineers, telemetry is not just a debugging tool; it’s how we observe, measure, and understand the real behavior of our code and workloads across any environment. During development, telemetry exposes performance bottlenecks and logic errors before they reach users. In production, it provides the signals needed for rapid detection, diagnosis, and resolution of incidents. Teams that treat telemetry as a first-class concern can iterate confidently, respond to issues proactively, and continuously improve both their systems and their workflows. ## OTEL: the open standard for telemetry OpenTelemetry (OTEL) has become the lingua franca of observability. It unifies metrics, traces, and logs under a single, vendor-neutral standard. This frees you from backend lock-in and enables flexible, composable telemetry pipelines—from your laptop to the cloud. OTEL supports most major programming languages and boasts a rapidly growing ecosystem. It’s the backbone of many modern observability stacks. Its unified data model allows you to correlate metrics, traces, and logs for deep, actionable insights. The active community ensures that integrations and features are always evolving. If you’re still relying on custom scripts and exporters, OTEL offers a more unified and future-proof approach to observability. However, it’s important to note that while OTEL is very mature for user space applications, its support for kernel-level telemetry is still evolving. Some features are not yet available or may require extra setup when working at the kernel level. ## OTEL and eBPF: powerful, but not yet complete The OpenTelemetry (OTEL) ecosystem is starting to use [eBPF (extended Berkeley Packet Filter)](https://ebpf.io) to get deeper insights into what’s happening inside Linux systems. This is exciting, because eBPF lets us see things at the kernel level—places that were previously hard to reach without writing custom code or patching the kernel. OTEL is already great at collecting telemetry from user space applications. Its APIs and SDKs make it easy to add metrics, traces, and logs to your code, and the OTEL Collector can send this data anywhere you want. But when it comes to the kernel, things are still a bit early. Most OTEL libraries are focused on user space, and kernel-level support is still catching up. Some projects are pushing the boundaries. For example, [opentelemetry-network](https://github.com/open-telemetry/opentelemetry-network) uses eBPF to collect detailed network metrics—like connection tracking, packet counts, and flow statistics—directly from the Linux kernel. This means you can get a lot of network visibility without changing your application code. But there’s a catch: opentelemetry-network only collects metrics, not traces or logs, and it’s focused on network data. If you want to collect generic, high-performance telemetry from deep inside your own kernel modules, you’ll probably need to build or extend something yourself. In short: OTEL and eBPF together are powerful, and the ecosystem is moving fast. But if you want the same level of observability in the kernel as you have in user space, there’s still some work to do. ## Why we built our own eBPF telemetry solution At Riptides, we quickly realized that existing solutions just didn’t cut it for our use case. Our requirements were: - **Traces to understand performance bottlenecks.** Metrics alone can tell you that something is slow, but only traces can show you *where* the time is being spent inside your kernel module. Without traces, you’re flying blind when it comes to optimizing performance or diagnosing subtle bugs. - **Logs for troubleshooting.** When something goes wrong deep in the kernel, logs are often the only way to figure out what happened. Kernel logs can help us catch rare edge cases, unexpected states, or errors that would otherwise be invisible. - **Lightning-fast operation.** Kernel modules run at the heart of the operating system. If telemetry collection is slow or inefficient, it can impact not just your module, but the entire OS and all user applications. That’s why we needed a solution that adds almost zero overhead. - **A generic solution.** Most existing libraries focus on specific events (like syscalls or network activity). We wanted to collect metrics, traces, and logs from *anywhere* inside our kernel modules, with fine-grained control over what gets instrumented. - **High-performance data transfer.** Many libraries use eBPF hashmaps to send data to user space, but we found that ringbuf (a lockless, high-speed buffer) is much better for our needs—offering lower latency, higher throughput, and simpler code. - **Flexibility to instrument multiple locations.** Off-the-shelf solutions often make it hard to add telemetry to many different places in your code. Since we control and implement the kernel modules, we wanted the flexibility to observe exactly what we care about, wherever it happens in our codebase. - **Control over what data is included for correlation.** In real-world systems, telemetry comes from many places—applications, infrastructure, and services, not just the kernel module. To correlate kernel telemetry with other data, we needed to control what fields and context were included, so we could tie everything together later. Off-the-shelf solutions rarely give you this level of flexibility. And while on a single machine you might get by with existing eBPF-based profiling tools or by looking at kernel logs with `dmesg`, this approach simply doesn’t scale in an enterprise environment. In production, telemetry needs to be collected from many machines and aggregated in a central place—otherwise, you lose the big picture and can’t respond quickly to issues across your fleet. q To address these needs, we built a telemetry pipeline that’s both flexible and fast. We hope sharing our approach will help others who are starting their own journey into kernel observability. ## Architecture: tracepoints, ringbuf, and a Go-based telemetry pipeline Our solution is built on a few key pillars: - **Tracepoints everywhere:** We instrument our kernel modules with tracepoints—special hooks that let us emit metrics, traces, or logs exactly where we need them. - **Stability and compatibility:** Tracepoints are stable and low-overhead, making them ideal for production use. They’re supported across all non-EOL kernel versions, unlike newer mechanisms such as kprobes or fentry, which may not be available everywhere. - **Fine-grained control:** Tracepoints give us precise control over where to emit telemetry, even from deep inside a function. In contrast, kprobes and fentry are best for hooking into syscall entry and exit points, but are more complicated to use for arbitrary locations. - **Efficient data transfer:** All telemetry data is sent to user space via a high-performance, lockless ring buffer (ringbuf). This avoids the contention and complexity of eBPF hashmaps, and lets us stream large volumes of events efficiently, without polling or complex synchronization. > If you want to learn more about tracepoints, check out our blog posts: [From Breakpoints to Tracepoints: An Introduction to Linux Kernel Tracing](kernel-tracepoints) and [From Tracepoints to Metrics: A Journey from Kernel to User Space](riptides-metrics). - **Minimal kernel data, enriched in user space:** We gather only the kernel-exclusive telemetry data, keeping things fast and efficient. Enrichment happens in user space, where we add additional context. - **Go and cilium/ebpf for user-space telemetry collection and eBPF deployment:** Our user-space collector is written in Go, leveraging the cilium/ebpf library to load eBPF programs, attach them to tracepoints, and read telemetry from the ringbuf. This combination lets us iterate quickly, maintain a modern codebase, and integrate seamlessly with the latest observability stacks and tools. - **Aggregation, enrichment, and OTEL export:** The collector aggregates, filters, and enriches telemetry data to add the necessary context for accurate and clear understanding. This enrichment and correlation step ensures that the telemetry is meaningful and actionable. After this, the data is converted to OTEL format. Metrics are sent to Prometheus, while logs and traces are sent to tracing platforms that support OTEL integration, such as Jaeger or Zipkin. ## Key takeaways Kernel telemetry isn’t just about collecting numbers—it’s about building feedback loops that make your systems smarter, safer, and more resilient. Whether you’re just starting out or you’re a seasoned expert, there’s always more to discover and improve. The journey to robust observability is ongoing, and every challenge is an opportunity to learn. At Riptides, this visibility is what powers our identity fabric, letting us enforce and validate non-human identity policies with confidence, precision, and speed. If you care about secure, kernel-native control, this level of observability is essential. --- ## Reflections from Identiverse: Why Security Needs Operational Efficiency - URL: https://blog.riptides.io/reflections-from-identiverse-why-security-needs-operational-efficiency - Published: 2025-06-09 - Author: Janos Matyas - Category: Conference - Tags: vision, spiffe, identity, zero-trust Security is not just about stopping threats, it’s about doing so in a way that doesn't disrupt or burden operations. As enterprises scale, the pressure to reduce costs while improving security grows. This makes operational efficiency not a luxury, but a mandate. At Identiverse 2025, one takeaway became crystal clear: both identity administrators and network administrators are hungry for a unified way to solve identity and access not just for humans, but for systems, services, and AI agents. That convergence is where true transformation lies. ## Aligning on Identity: Google and Riptides, Different Paths One of the standout talks for me was by Uttam Ramesh from Google: *Zero Trust Networking with Managed Workload Identities*. Conceptually, there are strong similarities between what Google is doing and our vision at Riptides. Both approaches align on using SPIFFE as the protocol for securely issuing and consuming workload identities. SPIFFE provides the foundation to federate trust across cloud providers, on-prem environments, and organizational boundaries. But our technical approaches differ: Google’s solution is centered around sidecars, proxies, and load balancers, while Riptides operates entirely from the Linux kernel. At Riptides, we’ve taken it one step further by embedding OPA (Open Policy Agent) directly into the kernel, letting us make inline decisions about what identity to assign to a connection based on metadata provided from user space. This gives us full visibility and policy control without needing to divert traffic to userland proxies. Historically, Google has been one of the few companies consistently driving innovation through thoughtful research and open publication of new technology paradigms. The Zero Trust model is no exception. Their continued work in this area sets a clear direction for the industry and aligns closely with our belief that identity driven access should be at the core of modern infrastructure. ## Industry Gaps and the Case for Active Security From the vantage point of Identiverse, it’s clear the industry still has work to do in how it talks about and handles non-human identities (NHIs). ### Mislabeling Credentials as Identities The term *non-human identity* is often misleading. What many refer to as NHIs are really just credentials, API keys, service accounts, IAM roles, that float untethered from the workloads they represent. At Riptides, we believe this model is fundamentally broken. Identities should be tightly bound to workloads. When credentials exist in isolation, they become vulnerable to misuse and provide no assurance about *who* or *what* is using them. We believe identity must originate from and be inseparable from the workload itself. > **At Riptides, our mantra is simple: kill credentials entirely, because true security starts when identity is no longer something you store, share, or manage.** ### Underutilization of SPIFFE Despite being an open standard purpose built for this challenge, SPIFFE remains underused across the industry. We see no better mechanism today to assign identities to workloads. SPIFFE provides the consistency, interoperability, and trust federation capabilities required to operate across cloud providers, on-prem environments, and organizational boundaries. Riptides is fully committed to SPIFFE, not just as an implementation detail, but as the foundation for identity in distributed systems. ### From Passive Posture to Active Enforcement Most solutions showcased at Identiverse focused on governance, compliance, and posture management. These are critical capabilities, but they are reactive by nature. Discovering a leaked credential is useful, but by then, the window for exploitation may have already passed. Worse, you often can’t even tell *when* or *where* it was used. Riptides takes an active approach: by operating in the kernel, we observe and enforce identity at the network layer, in real time. All workload-to-workload communication passes through the kernel, giving us complete visibility and control. This allows us to build a dynamic, realtime inventory of active identities, without requiring invasive access to your cloud accounts or SaaS providers. ### AI Agent Security is Still Workload Security AI agent security dominated much of the conversation at Identiverse, and while it’s an important topic, we feel it’s often overhyped. In particular, when it comes to connection security between MCP servers, orchestration tools, and agent-to-agent (A2A) systems, the foundational building blocks already exist. There is no need to reinvent the wheel. Well established mechanisms as mutual TLS, identity aware and conditional routing, workload bound credentials, and fine-grained authorization are more than capable of securing these connections when applied correctly. The challenge isn’t inventing new protocols, but using existing standards like SPIFFE, TLS, and OPA properly and consistently. We’ll be covering our specific approach to AI workload security in an upcoming post. ## Looking Ahead At Riptides, we’re committed to this vision. We believe that operational efficiency and security can go hand-in-hand. Our identity fabric delivers strong, workload-bound identities using a proven and mature foundation: x.509 certificates, SPIFFE, and in-kernel policy enforcement. We’re here to cut cost, reduce risk, and bring clarity to the complex landscape of non-human identities, at wire speed. --- ## Seamless Kernel-Based Non-Human Identity with kTLS and SPIFFE - URL: https://blog.riptides.io/seamless-kernel-based-non-human-identity-with-ktls-and-spiffe - Published: 2025-06-02 - Author: Nandor Kracser - Category: Kernel - Tags: spiffe, kernel, ktls, linux, identity Modern infrastructure requires strong, scalable, and transparent identity mechanisms - especially for non-human actors like services, workloads, jobs, or agents. **Riptides** is a novel approach that anchors non-human identity at the kernel level, leveraging **SPIFFE**, **kTLS**, and in-kernel **mTLS** handshakes to offer zero-intrusion security (identity and encrypted communication) for user-space applications. In this post, we’ll walk through how Riptides works, why kTLS is a game-changer for performance and security, and why this effort lives in the kernel rather than in eBPF. ## What is Riptides? **Riptides** is an identity platform purpose-built to deliver seamless, policy-based identity for non-human workloads. At the heart of the system is the `riptides-driver`, a kernel module that integrates directly into the Linux TCP stack. It intercepts TCP connection establishment and performs mutual TLS (mTLS) handshakes entirely within the kernel, together with **kTLS** record encryption. These handshakes are guided by identity and authorization policies defined in a central **Riptides control plane**. To bridge user space and kernel space, Riptides uses a dedicated **user-space daemon**. This communicates with the kernel module via **device driver communication**, synchronizing policies, trust bundles, and secrets down to the kernel. The result is a system where applications continue to open plain TCP connections as usual, while Riptides transparently upgrades those connections to authenticated and encrypted mTLS sessions. The application code remains unchanged, while security, identity enforcement, and observability are handled entirely beneath it, without even redeployment. ## Why kTLS? **kTLS** (Kernel TLS) is a [Linux kernel feature](https://docs.kernel.org/networking/tls-offload.html) that offloads TLS record encryption/decryption to the kernel. Handshakes still have to be done in user-space, so applications need to maintain the TLS sessions themselves. kTLS offers several advantages: - **Performance:** Avoids costly user/kernel context switches for each TLS record, improving throughput and reducing CPU usage. - **Zero-copy:** Enables efficient data paths by combining TLS with `sendfile()` and `splice()`. - **Offloading:** Makes it easier to offload TLS operations to NICs that support TLS hardware acceleration. - **Security surface:** Minimizes the attack surface by keeping TLS state within the kernel. While traditionally used purely for performance, Riptides leverages kTLS in a novel way — using it for record encryption while performing the entire TLS handshake separately in the kernel. This allows the full TLS connection to be established transparently, so applications remain completely unaware of the mTLS process. ### A Brief History of kTLS Historically, the concept and the initial practical exploration of kernel-level TLS offload were first associated with [FreeBSD, driven by Netflix](https://freebsdfoundation.org/end-user-stories/netflix-case-study/). Their research and prototyping in 2015 and onwards demonstrated the significant performance benefits of moving TLS processing into the kernel, especially for high-bandwidth applications. Inspired by this work on FreeBSD by Netflix, the Linux community then introduced the concept and began its own independent development of kTLS for the Linux kernel. [Dave Watson from Facebook](https://lwn.net/Articles/666509/) was instrumental in initiating this effort in late 2015, leading to its inclusion in Linux kernel 4.13. Since then: - **Receive (RX) support** was added. - **TLS 1.3** capabilities have been gradually integrated. - Major distros and cloud vendors began including kTLS in production-ready kernels by default. - Hardware vendors like Mellanox/NVIDIA and Intel provide NICs with kTLS offload support. ### OpenSSL ships kTLS for end-users While the major OS distributions eventually shipped kernels with kTLS support, it took significant effort for users to actually establish kTLS sessions in real-world applications. OpenSSL integrated support for kTLS in **version 1.1.1**, with enhanced functionality in **OpenSSL 3.x**. This allows applications using OpenSSL to leverage kTLS automatically, assuming proper kernel support and configuration. Several well-known projects and services have explored or integrated kTLS. For example, **nginx** supports kTLS when built with OpenSSL and properly configured, enabling zero-copy TLS for improved performance for the `sendfile` operation. **HAProxy** has seen experimentation and community proof-of-concepts around kTLS integration. The **Apache HTTP Server (httpd)** project has also explored support, and although not widely deployed yet, interest continues to grow. **curl** for example also supports kTLS but as you can see it takes effort to maintain and implement it in applications. This is where Riptides comes into picture. ### Why Riptides Embraces kTLS Riptides, by moving the **entire mTLS handshake into the kernel** transforms kTLS from a performance optimization into a **security boundary**, enabling: - Fully transparent, in-kernel mTLS handshakes. - SPIFFE-based identity negotiation with no user-space exposure. - Reduced secret sprawl and fewer context switches. - **Less kernel code to maintain**, since we rely on an **established, battle-tested kernel TLS implementation** rather than building cryptographic primitives or protocol logic ourselves. By employing the existing kTLS infrastructure, Riptides benefits from the continued evolution, optimization, and security hardening of the kernel’s TLS stack — reducing maintenance burden, increasing confidence in the correctness of the implementation, and enabling compatibility with modern NICs that support TLS hardware offload for even greater performance gains. ## Why Not eBPF? You might ask: “Why is this not an eBPF program?” (Or perhaps even multiple programs?) While **eBPF** is excellent for observability, packet filtering, traffic shaping, and lightweight decision-making close to the network layer, it is fundamentally **unsuited for managing TLS handshakes** or cryptographic state machines. First, eBPF programs are subject to strict verifier constraints — such as limitations on loops, recursion, and stack depth — which make it impractical to implement complex protocols like TLS. Additionally, TLS involves intricate memory management and timing-sensitive operations that are simply not feasible within the eBPF execution model. Even maintaining persistent cryptographic state across connection events is difficult to do safely and efficiently in eBPF. As a result, Riptides opts for a full **kernel module** implementation, where cryptographic and policy logic can operate with the necessary performance, safety, and architectural flexibility. ## Riptides Architecture: Full Kernel Handshake + SPIFFE Once Riptides is installed, the system operates transparently and automatically at the kernel level, the steps are the following: 1. The Riptides user-space **daemon** receives SPIFFE identities, trust bundles, and connection policies from the control plane. 2. These are loaded into the kernel `riptides-driver` via **device driver communication**. 3. When a TCP connection is initiated, the driver: - Checks the destination against policy. - Initiates a full mTLS handshake (not just record protection) in the kernel using SPIFFE-based certificates. - Establishes a kTLS session for record encryption/decryption. ![arch](../../assets/ktls-at-riptides/ktls-riptides.jpg) This design has multiple benefits: - **Security:** Secrets never leave kernel memory, reducing exposure. - **Transparency:** Applications are unaware they’re speaking mTLS - zero code changes. - **Policy enforcement:** Fine-grained identity-based access control happens at connection time. ## Conclusion Riptides effectively acts as a programmable, identity-aware TCP stack for your applications - with no impact to application logic, it brings application identity to the kernel. This setup ensures that all outbound connections are policy-compliant, cryptographically secure, and identity-aware, without additional application code or TLS logic in user space. The system runs silently and efficiently underneath, maintaining security guarantees and reducing complexity for developers and operators alike. If you're building distributed systems and are seeking robust security without added complexity, Riptides presents an elegant and forward-looking solution worth following closely. Some interesting reads in the topic: - - - --- ## Riptides is heading to Identiverse 2025! - URL: https://blog.riptides.io/riptides-is-heading-to-identiverse-2025 - Published: 2025-05-29 - Author: Janos Matyas - Category: Identiverse - Tags: kernel, non-human-identity, identiverse, conference We're excited to join the identity community next week to showcase how we’re rethinking non-human identity from the ground up. Managing secure service-to-service connections and non-human credentials is a growing security challenge. Secrets sprawl, credentials leak, and attackers exploit the gaps. At Riptides, we’re building an identity fabric for workloads and AI agents — issuing ephemeral, SPIFFE-based identities enforced entirely in the Linux kernel. No more long-lived secrets. No manual rotations. Just zero-touch, zero-trust authentication for internal and third-party connections — designed for security and infra teams, with no friction for developers. Whether you're solving for internal workload comms, external partner access, or agentic application security, we’d love to connect. --- ## From Tracepoints to Metrics: A journey from kernel to user-space - URL: https://blog.riptides.io/from-tracepoints-to-metrics-a-journey-from-kernel-to-user-space - Published: 2025-05-26 - Author: Balint Molnar - Category: Kernel - Tags: kernel, ebpf, tracing, event, metrics ## From Hooks to Userspace In our [last post](/blog/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing), we explored how to define custom tracepoints inside the kernel to expose meaningful events. In this follow-up, we shift our focus to the second half of the journey: how to move that data from the kernel to user-space efficiently. The question we set out to answer was: **what’s the best mechanism for streaming kernel events to user-space at scale?** Our journey led us through sockets, character devices, and virtual filesystems—before ultimately embracing eBPF. In this post, we’ll walk through how we built our tracing pipeline, the trade-offs we considered, and why eBPF turned out to be the right tool for the job. ## Streaming Events from the Kernel to User Space Riptides is designed to operate silently and unobtrusively, which is one of the reasons we chose to use tracepoints for generating events inside the kernel. However, this design choice is only effective if the event data can be streamed efficiently to user space for further processing. There are several methods to stream such events, each with trade-offs in performance, complexity, and suitability for different use cases. Below are the options we considered: ### Netlink Sockets Netlink is a special IPC mechanism between the Linux kernel and user space, built on top of sockets. It supports multicast, unicast, and asynchronous messaging, enabling structured communication with one or more user space processes. In essence, the kernel creates a special socket, and user space applications interact with it using traditional `recvmsg/sendmsg` calls. While Netlink supports structured messages, it is not ideal for high-rate, stream-based messaging. Each message involves context switches and kernel locking, and there is no support for zero-copy transmission. Netlink is best suited for status updates, control messages, and occasional events rather than continuous data streams. ### eBPF (Extended Berkeley Packet Filter) eBPF is the successor to the original BPF filtering mechanism in Linux. It allows programs to run within a privileged context, such as the Linux kernel, and is primarily used to extend kernel capabilities in a safe and efficient manner. While classic BPF was limited to packet filtering, eBPF enables a much broader range of functionality. Traditionally, extending kernel behavior required writing a kernel module is often overkill for many use cases. eBPF changes this by allowing developers to load small programs at runtime without modifying the kernel or rebooting. Safety and performance are enforced through a verification engine and a JIT compiler. Since eBPF programs can attach to kernel hooks like kprobes and tracepoints, it is an excellent choice for streaming events from kernel to user space. With features like ring buffer support and zero-copy data transfer, eBPF provides high performance with minimal overhead. Compared to character devices, it is also considered safer due to its built-in verification system. Note that user-space loaders are required to manage the lifecycle of eBPF programs and their associated resources (e.g., maps and ring buffers). ### Procfs/ Sysfs Procfs (`/proc`) and Sysfs (`/sys`) are virtual filesystems used to expose kernel data to user space. procfs is primarily used for exporting system and process information, while `sysfs` represents kernel object attributes. These interfaces are intentionally simple, using standard file read/write operations. However, this simplicity comes at a cost: there is no built-in push mechanism. User space must **poll** these files to detect updates, which can lead to increased CPU usage. Additionally, there is no buffering support, only the current snapshot of the data is available, making it unsuitable for scenarios where events occur frequently or in rapid succession. Overall, Procfs and Sysfs are best suited for **configuration** and **introspection**, not for real-time telemetry or high-frequency event streaming. ### Character Device A character device is a custom kernel device exposed under `/dev/mydevice`, allowing user space processes to interact with it using standard file operations like `open`, `read`, and `write`. To use a character device, a kernel module must register it and implement the necessary callbacks. Because the module owns the device, it is responsible for managing its entire lifecycle, including creation, deletion, data handling, and synchronization. This adds significant complexity. Moreover, there is no memory protection between your code and the kernel, so bugs in the module can lead to serious issues like kernel panics. Despite these risks, character devices offer **maximum flexibility** and **very fast I/O performance** for data transfer between kernel and user space. ### Comparing the Four | Feature | Netlink | eBPF | proc/sysfs | Character Device | |---------------------|----------------|------------------------|------------------|-------------------| | **Performance** | Medium | High | Low | High | | **Complexity** | Medium | Medium-High | Low | Medium-High | | **Stream Suitable?**| Limited | ✅ Yes (ring buffer) | ❌ No | ✅ Yes | | **Push-based?** | Partial | Yes | No (polling only)| Yes (poll/push) | | **Best For** | Events, config | Tracing, telemetry | Status/config | Custom data flows | At Riptides, we already use a **character device** to transfer control messages between our kernel module and user-space agent. We're fully aware of the complexity involved in managing a character device, so at first glance, it seemed like the perfect mechanism for streaming events as well. However, after careful consideration and internal discussions, we made the decision to **keep character devices solely for control messages** and adopt **eBPF** for all tracing and telemetry needs. Here’s why: - **Simplified Kernel Module** We aim to keep our kernel module as minimal and focused as possible. Offloading all streaming logic to eBPF makes the codebase cleaner and easier to maintain. - **Unidirectional Data Flow** For tracing and telemetry, we mostly need one-way data flow from kernel to user space. eBPF is perfectly suited for this use case with minimal overhead. - **Built-in Synchronization** eBPF provides standard mechanisms (e.g., ring buffers) with built-in synchronization and locking, eliminating the need to implement and maintain our own. - **High Performance with Zero-Copy** eBPF offers **first-class support** for ring buffers and **zero-copy data transfer**, making it extremely efficient for high-frequency event streaming. - **Easy Extensibility** With native support for hooking into tracepoints, kprobes, and other kernel hooks, eBPF allows us to expand our tracing capabilities quickly and safely. In summary, eBPF enables us to decouple tracing logic from our core kernel module, while delivering high performance, easier maintenance, and future flexibility. ## Deep Dive into Our Trace Highway In our tracing and telemetry architecture, we use **tracepoints** to statically define events within our kernel module. These tracepoints serve as well-defined hooks for significant system or application events. To transport these events to user space, we rely on **eBPF**, which provides an efficient and safe mechanism for capturing, buffering, and forwarding event data. This setup allows us to build telemetry and metrics with minimal overhead and high reliability. Let’s explore how this is achieved, step by step: ### Requirements to Use an eBPF Program If you’ve read our previous blog about tracepoints, you’ll recall that each tracepoint defines a **hookable location** in the kernel. These hooks can be used to attach eBPF programs, allowing us to run custom logic when the tracepoint is triggered. Here’s an example of one of our eBPF handlers: ```c SEC("tracepoint/riptides/riptides_accept_start") int riptides_accept_start(struct driver_socket_start_ctx *ctx) { return handle_socket_start_event(ctx, EVENT_DRIVER_SOCKET_ACCEPT_START); } ``` ### SEC() Macro The `SEC()` macro is a section annotation used in eBPF programs to specify the type of program and where it should be attached. In our example: ```c SEC("tracepoint/riptides/riptides_accept_start") ``` means this eBPF program attaches to the tracepoint `riptides_accept_start` inside the `riptides` subsystem. ### How to Find What to Write Inside `SEC()` You might wonder: *How do I know the exact tracepoint name to use here?* Linux provides a helpful virtual file called `/proc/kallsyms`, which lists all kernel symbols currently known to the system. Think of it as the symbol table for the running kernel. Since we want to find tracepoints related to our kernel module called `riptides`, we can filter symbols like this: ```bash cat /proc/kallsyms | grep 'riptides' ``` This produces many symbols because it lists all kernel symbols, including functions and data. To narrow it down to **tracepoints**, we add another filter: ```bash cat /proc/kallsyms | grep 'riptides' | grep 'tracepoints' ffff80007c2e0010 d __tracepoint_riptides_accept_start [riptides] ... ``` - The first column is the symbol’s memory address. - The `d` indicates it’s a local data symbol. - The symbol name tells us the tracepoint: `__tracepoint_riptides_accept_start`. - The `[riptides]` at the end shows it belongs to the riptides kernel module. ### A Simpler Way: Available Tracepoints List For convenience, you can also list all available tracepoints directly via: ```bash cat /sys/kernel/debug/tracing/available_events | grep riptides riptides:riptides_accept_start ... ``` Although the methods described can list tracepoints, they do not show kprobes or other hook types. That’s an important caveat to keep in mind when exploring available kernel hooks. Once the eBPF program is attached to a tracepoint, the kernel automatically invokes this function whenever the tracepoint fires. In our example, that’s the function: ```c int riptides_accept_start(struct driver_socket_start_ctx *ctx) ``` You may notice this handler has a single parameter, `ctx`. This parameter is a **context structure**, and its layout is strictly defined by the tracepoint itself. If the program’s context structure does not match the tracepoint’s expected layout, one of two things will happen: - You may get **garbage or invalid data** inside the handler. - Or, more commonly, the eBPF verifier will detect the mismatch and reject the program during loading. Therefore, it’s crucial to ensure your context struct aligns exactly with the tracepoint’s data format. In our case, the trace event was declared using the following macro in the kernel: ```c TP_STRUCT__entry( __array(u8, uuid, UUID_SIZE) __field(s64, trace_timestamp)), ``` To verify the actual layout used by the kernel, you can inspect the generated tracepoint format with: ```bash sudo cat /sys/kernel/debug/tracing/events/riptides/riptides_accept_start/format name: riptides_accept_start ID: 1704 format: field:unsigned short common_type; offset:0; size:2; signed:0; field:unsigned char common_flags; offset:2; size:1; signed:0; field:unsigned char common_preempt_count; offset:3; size:1; signed:0; field:int common_pid; offset:4; size:4; signed:1; field:u8 uuid[16]; offset:8; size:16; signed:0; field:s64 trace_timestamp; offset:24; size:8; signed:1; ``` You’ll notice the first four fields (`common_type`, `common_flags`, etc.) are **automatically inserted** by the tracepoint system. These are standard metadata fields shared across all tracepoints and are used internally for scheduling, filtering, or debugging. Despite not defining them ourselves, **you must account for them** in your context structure, as they are part of the layout passed to your eBPF program. Here’s the corresponding context structure in user code: ```c struct driver_socket_start_ctx { struct header h; // Matches the common_* fields __u8 uuid[16]; // Custom field __s64 timestamp; // Custom field }; ``` The `struct header` here represents the first four "common" fields and should be defined to match them precisely. ### Ring Buffer Before diving into what happens inside the `handle_socket_start_event` function, let’s take a moment to understand a key component of our setup: the **ring buffer**, which serves as the bridge between kernel space and user space. Here's how we declare it in eBPF: ```c struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 65536); } driver_socket_buf SEC(".maps"); ``` The ring buffer is a **lockless, high-performance FIFO** data structure provided by the eBPF subsystem. It is ideal for sending a large number of events efficiently from the kernel to user space, maintaining their order. Key points: - `max_entries`: This is the only required field. It defines the total buffer size in **bytes**, and must be a power of two (e.g., 65536). - **No struct is predefined**: Unlike hash or array maps, a ring buffer does not enforce a data schema. It is the responsibility of the producer and consumer to agree on the data format and parse it correctly. - **Memory-mapped in user space**: To consume data from the ring buffer, user-space code memory maps the buffer using `mmap()`, enabling zero-copy access. - **Epoll-friendly**: Ring buffers support `epoll`, allowing the user-space application to efficiently wait for new events without constant polling. This combination of **performance, simplicity, and ordering** makes ring buffers an excellent choice for high-throughput, one-way data transfer from the kernel. ### Propagate Data Inside the `handle_socket_start_event` function, we populate the ring buffer with telemetry data. This process relies on three key eBPF helper functions provided by the kernel: - `bpf_ringbuf_reserve(void *ringbuf, __u64 size, __u64 flags)` This function reserves a region of memory inside the ring buffer and returns a pointer to it. You can write data directly into this space from your eBPF program. - If the buffer is full, it returns `NULL`. - There is no need for additional memory copies—data is written directly into the buffer (zero-copy). - It’s especially useful for writing large samples since the data does not live on the stack. - The eBPF verifier ensures you don’t write beyond the reserved memory. - `bpf_ringbuf_submit(void *data, __u64 flags)` Once you’ve finished writing data into the reserved space, this call marks it as ready to be consumed by user space. - `bpf_ringbuf_discard(void *data, __u64 flags)` If you decide not to submit the data (e.g., based on a runtime condition), use this function to discard the reserved space. These three functions provide a reliable and efficient way to transfer structured data from kernel to user space with minimal overhead. Earlier in this post, we mentioned that eBPF code and associated maps (like the ring buffer) must be set up from a user-space application. So what does Riptides use to handle this initialization and data consumption? And how is the buffer read in user space? That’s what we’ll cover next. ### User-Space Helpers There are several libraries available to load, verify, and attach eBPF programs to various kernel hooks. The most mature and feature-rich option is [libbpf](https://libbpf.readthedocs.io/en/latest/libbpf_overview.html), written in C. However, since our agent component is implemented in Go, we opted to use Cilium’s eBPF library, which is a pure Go alternative. This library comes with a tool called `bpf2go`, which not only compiles your eBPF programs but also generates Go scaffolding code. This simplifies integration by eliminating the need to manually interface with the underlying C code. The Cilium eBPF library is optimized for performance and supports the latest eBPF features. It also abstracts away the complexity of managing the lifecycle of eBPF programs and related resources like ring buffers. With this setup, raw events from the kernel are efficiently streamed into user space, ready for processing by our telemetry and metrics pipeline. ## Conclusion With our eBPF-based tracing architecture in place, we now have a high-performance, low-overhead pipeline for streaming raw events from kernel space to user space using ring buffers. By leveraging tracepoints and the Cilium eBPF library in Go, we've built a clean and efficient mechanism for capturing real-time telemetry data without overcomplicating our kernel module. In the next post, we'll dive deeper into how these raw events are processed inside our user-space agent to generate meaningful metrics and telemetry data. --- ## Non-Human Identity Done Right: Why AI Agents Need SPIFFE and Riptides - URL: https://blog.riptides.io/non-human-identity-done-right-why-ai-agents-need-spiffe-and-riptides - Published: 2025-05-19 - Author: Janos Matyas - Category: Identity - Tags: AI, Identity, SPIFFE ## Why AI Agents Need SPIFFE, and Why Riptides Is the Seamless Way to Deliver It As AI agents grow in sophistication and autonomy, their need to securely communicate with third-party services and other agents becomes critical. These communications require strong authentication, fine-grained authorization, and end-to-end encryption—especially when AI agents are used in environments where trust boundaries must be enforced and regulatory controls apply. At the heart of this challenge is the question: **How do you securely authenticate and authorize software agents that have no human operator?** This post explores how the **SPIFFE** identity standard, when paired with a powerful identity management layer like **Riptides**, provides the missing security primitives for AI agents—without burdening developers with the complexity of identity management. ## The Problem – AI Agents Need Secure Identity Modern AI ecosystems are increasingly composed of multiple collaborating agents, often augmented with third-party plugins, tools, or services that provide computation, storage, or other critical functions. With the rise of **Model Control Plane (MCP)** systems, agents are beginning to delegate tasks to external software components in real-time. These interactions, while powerful, introduce a deep trust problem: - **How does a third-party system trust the agent requesting access?** - **How does one agent verify the identity of another when using inter-agent protocols like A2A?** - **Can trust be established dynamically, at runtime, without hardcoded secrets or static configuration?** Unfortunately, the status quo is not enterprise-ready. Both MCP and A2A protocols currently rely on weak or ad-hoc authentication methods: - **Shared secrets or API keys** often reside alongside agent code or in local filesystems, a high-risk pattern vulnerable to compromise. - **Static credentials** break core security principles such as rotation, least privilege, and auditability. - **Lack of mTLS enforcement** leaves communication channels open to interception or man-in-the-middle attacks. ## SPIFFE as a Foundation for Zero-Trust Identity **SPIFFE (Secure Production Identity Framework For Everyone)** solves these problems by offering a robust, cryptographically verifiable identity layer for workloads. It enables any agent, process, or service to get a short-lived X.509 certificate or JWT token that proves its identity. For AI agents, SPIFFE can: - Provide ephemeral credentials that bind agent identity to its runtime environment. - Support **mutual TLS (mTLS)** for secure communication with third-party systems or other agents. - Remove the need for manual secret distribution or key management. - Standardize identity claims across heterogeneous systems and platforms. But while SPIFFE offers a powerful specification, implementing it—especially at scale, with high assurance—is a non-trivial task. This is where **Riptides** enters the picture. ## Riptides – Seamless, Kernel-Level SPIFFE for AI Agents [Riptides.io](https://riptides.io) is purpose-built to secure **non-human identities (NHIs)** like AI agents, containerized workloads, or autonomous software actors. Unlike traditional SPIFFE implementations that require agents to fetch and manage their own identities, Riptides **automates everything**—securely, transparently, and at the kernel level. ### How It Works - **Ephemeral Identity on Demand**: When an agent reaches out to a third-party service, the Riptides kernel module automatically provisions a short-lived X.509 certificate or JWT token that complies with the SPIFFE standard. - **TLS by Default**: All outbound communication is automatically upgraded to TLS using these credentials, ensuring encryption without developer effort. - **Agent-to-Agent mTLS**: When agents use the A2A protocol to talk to each other, Riptides ensures both ends mutually authenticate via SPIFFE-compliant identities. - **No Credential Leakage**: Credentials never touch disk, memory, or user space. They exist only in the kernel and only for as long as they’re needed. This makes Riptides a powerful fit for the dynamic, distributed nature of AI agent ecosystems—especially when security must be seamless and automatic. ## Developers Shouldn’t Have to Be Security Experts Let’s face it: the average AI agent developer isn’t a security engineer. They want to build functionality, not manage identity lifecycles, TLS handshakes, or certificate rotation policies. When left to their own devices, we see dangerous patterns: - Credentials hardcoded into agent code. - API keys bundled into the same container as the agent. - Static tokens stored in Git repos, config files, or mounted volumes. These practices are not only unsafe—they scale poorly and introduce massive risk in enterprise deployments. Riptides eliminates these anti-patterns by **removing credentials from the developer's hands entirely**: - **Credentials are generated on the fly, only when needed.** - **They are injected by the kernel module, not managed by the developer.** - **They are ephemeral, scoped, and automatically expire.** This aligns with the core principles of secure NHI: - **No long-lived secrets** - **No human-in-the-loop** - **No credentials outside the kernel** As a result, developers can focus on what they do best—building intelligent, composable agents—while Riptides ensures they do so securely, by default. ## Conclusion: Riptides Is the Future of Secure Non-Human Identity In the emerging world of autonomous agents, secure identity is not a luxury—it's a **requirement**. AI systems need to communicate safely, prove their identity, and comply with enterprise-grade access controls. SPIFFE provides the right standard for this, but it's **Riptides** that brings it to life. With: - **Automatic, ephemeral credential issuance** - **Kernel-level enforcement and TLS upgrades** - **Zero developer overhead** - **Full compliance with SPIFFE** - **Seamless support for MCP, A2A, and third-party integrations** Riptides is not just a solution—it's a paradigm shift in how we think about identity for AI agents. If you're building AI-native systems and want security without friction, it's time to make **Riptides your foundation for NHI**. --- ## The Critical Role of Unique Workload Identity in Modern Infrastructure - URL: https://blog.riptides.io/the-critical-role-of-unique-workload-identity-in-modern-infrastructure - Published: 2025-05-07 - Author: Zsolt Varga - Category: Identity - Tags: spiffe, x509, identity As modern infrastructure grows more complex, securing service-to-service communication has become a major challenge. Workloads, whether they’re running in Kubernetes, virtual machines, or serverless environments, need a way to prove their identity when interacting with other services. Traditionally, this problem has been solved with shared secrets, API keys, or IP-based trust models. However, these approaches introduce security risks and operational overhead: - API keys and static credentials can be leaked or stolen, giving attackers unauthorized access. - IP-based trust models are unreliable in cloud environments, where workloads frequently move between different network locations. - Manually managing TLS certificates is complex, and expired certificates can lead to service outages. A strong workload identity system provides each service with a unique, verifiable identity that allows it to securely authenticate with others. This is where standards like [SPIFFE](https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE.md) (Secure Production Identity Framework for Everyone) come into play, along with x509 certificates, which offer cryptographic proof of identity and support for federated trust across environments. ## Why Workload Identity Matters When two humans communicate securely, they rely on established forms of identity, passports, driver's licenses, or employee badges. Workloads need something similar to prove who they are in an automated, scalable way. For years, many systems relied on network-based security. If a service was inside a trusted network, it was assumed to be legitimate. But as infrastructure has evolved, this model no longer works: - Cloud workloads frequently change IPs, making network-based access controls unreliable. - Microservices often need to communicate across different security boundaries, such as across cloud providers or between business partners. - Security breaches often involve credential leaks, meaning long-lived tokens and API keys are a liability. Instead of static credentials or network-based trust, a better approach is to give each workload a dynamic, cryptographically verifiable identity. ## Workload Identity with SPIFFE One way to standardize workload identity is through SPIFFE, an open framework that provides workloads with unique, structured identities. A SPIFFE ID follows a format like: ``` spiffe://example.com/staging/accounting/service/auth-backend ``` This gives workloads a consistent way to identify themselves, regardless of where they’re running. Instead of relying on long-lived secrets, services can dynamically get short-lived x509 certificates that embed their SPIFFE ID. These certificates should be automatically rotated and used to authenticate workloads through mutual TLS (mTLS). While SPIFFE provides a useful framework, the underlying mechanism that makes this work is a widely used standard for identity and encryption, x509 certificates. As we have previously discussed in one of our previous posts - [Introduction to SPIFFE: Secure Identity for Workloads](/blog/introduction-to-spiffe-secure-identity-for-workloads) - SPIFFE is one of the pillars of the Riptides’ identity platform. The Riptides Platform is a comprehensive solution for securing workload-to-workload communication, built on a foundation of identity. It provides a universal and transparent non-human identity solution that secures every connection between workloads and services. ## Why x509 Certificates Matter x509 certificates are a fundamental technology in securing the internet, used everywhere from HTTPS to VPNs. In the context of workload identity, they provide several key benefits: ### Strong Authentication Instead of trusting an API key stored in a config file, services authenticate each other using certificates, which are backed by cryptographic proof. ### Automatic Certificate Rotation Certificates are short-lived and automatically renewed, reducing the risk of credential leaks. ### Mutual Authentication with mTLS Both the client and server present certificates to verify each other’s identity, ensuring that only authorized workloads can communicate. ### Interoperability Across Environments x509 certificates provide a standardized format that works across different cloud providers, Kubernetes clusters, and legacy systems. ### Federated Trust Between Organizations Certificate authorities can be linked, allowing workloads in separate trust domains (such as two different companies) to securely authenticate with each other. ## Trust Federation: Extending Identity Across Boundaries One of the most powerful aspects of x509-based identity is its support for trust federation. This allows organizations to establish authentication relationships between different security domains, without requiring a central authority to issue credentials for all workloads. For example, imagine: - A multi-cloud deployment, where workloads in AWS and Azure need to securely communicate. - A supply chain scenario, where a logistics provider needs to integrate with a manufacturer’s systems. - An enterprise with multiple business units, where services in separate environments need to trust each other. With certificate chaining, an organization can define which external trust domains are valid, allowing workloads from different environments to authenticate securely. ## Moving Toward a Secure Workload Identity Model As infrastructure becomes more dynamic and distributed, organizations need to move away from static credentials and network-based trust models. A strong workload identity system should: - Be cryptographically verifiable (not just an API key in a config file). - Use short-lived, automatically rotated credentials (instead of long-lived secrets). - Support interoperability across environments (not tied to a single cloud provider). Using x509 certificates and trust federation, organizations can build a scalable, secure authentication framework for service-to-service communication. **Stay tuned as we explore how Riptides makes workload identity simpler, more secure, and more scalable.** If you are interested in replacing secrets with trusted identities and using the Riptides platform, please [talk to us](https://riptides.io/request-a-demo). --- ## From Breakpoints to Tracepoints: An Introduction to Linux Kernel Tracing - URL: https://blog.riptides.io/from-breakpoints-to-tracepoints-an-introduction-to-linux-kernel-tracing - Published: 2025-05-03 - Author: Balint Molnar - Category: Kernel - Tags: kernel, tracepoint, linux, tracing, event ## Providing NHI to Services: Piece of Cake or Not? We can all agree that in today’s modern infrastructure, the significance of non-human identities has grown dramatically. Automatically provisioning identities for services has become a critical responsibility for security and infrastructure teams. At Riptides, we aim to be the catalyst for this transformation by offering a simple, automated path to streamline your non-human identity journey. The Riptides platform provides a comprehensive view of your entire infrastructure, enabling you to monitor whether services are communicating securely, whether they have identities, and—most importantly—who is talking to whom. Armed with this information, administrators can automatically provision identities for every service. Of course, delivering this level of visibility requires tracing connections—but the question remains: how? ## Getting Events Out of the Kernel—or Lost in the Woods? At Riptides, we’ve committed to three guiding principles for the tracing solutions we adopt: - Never compromise the performance of live workloads - Design for maximum portability (it should run anywhere) - Leverage existing kernel tracing tools for easy debugging and integration With these principles in mind, our goal is to capture kernel-level events and route them efficiently to user space for analysis. While there are many existing tools available, we don’t aim to reinvent the wheel. Instead, we focus on sticking to our principles while maximizing the benefits of proven solutions. ### Dynamic Probes: kprobe & kretprobe If you're researching kernel tracing or event collection, sooner or later you'll come across Kernel Probes (kprobes). Kprobes allow you to dynamically hook into almost any point in the kernel code to capture valuable events. There are two main types of probes: kprobes and kretprobes. A kprobe triggers when a specific function is called, while a kretprobe fires when that function returns. Traditionally, you had to write a kernel module that subscribed to a specific address, triggering when execution reached that point. This module also registered a handler containing the business logic for what to do when the probe fired. With the rise of eBPF, this process has become much less error-prone and far more accessible and widely adopted. Let’s dig deeper and explore how kprobes actually work. **How Do They Really Work?** When a **kprobe** is registered, the probed address in the kernel is replaced with a **breakpoint**—much like how a debugger works. When the CPU hits that instruction, a trap occurs: CPU registers are saved, and control is transferred to the kprobe handler. This handler receives both the kprobe struct and the saved register state, executes the defined logic, and then resumes normal execution. A **kretprobe** operates similarly, but instead of breakpoints, it uses **trampolines**. These trampolines are small snippets of code injected by modifying the return address on the stack. They execute the handler logic and then return execution to the original function return address. To handle recursive or concurrent calls of the same function, the `kretprobe` struct includes a `maxactive` field, which defines how many simultaneous instances can be probed. Additionally, you can specify an `entry_handler` to pre-filter calls, deciding whether the main kretprobe handler should be executed or skipped. > **Breakpoint:** A special CPU instruction injected into executable code to trigger a trap. > **Trampoline:** A small, custom code snippet inserted into the control flow by overwriting the return address. It executes handler logic and returns to the original flow. In certain scenarios, `kprobes can be optimized` to use trampolines instead of breakpoints for improved performance. ### Function Tracing: fprobe When narrowing the scope to function entry and exit points, we encounter another built-in tracer: **fprobe**. Fprobe leverages the **ftrace** infrastructure—a powerful tracing framework integrated directly into the Linux kernel. Like `kprobe` and `kretprobe`, `fprobe` allows you to attach handlers to the entry and return points of functions. However, instead of relying on breakpoints or trampolines, fprobe uses **ftrace hooks**. When the kernel (or a kernel module) is compiled with specific configuration options, lightweight entry hooks are embedded into many functions. These hooks are initially implemented as **no-ops** or lightweight jumps, which are later patched at runtime to redirect execution to the ftrace handler—this includes your custom handler logic. In short, while fprobe is limited to specific, pre-instrumented points, it is **significantly faster** and **better suited to high-frequency function calls**, making it ideal for performance-sensitive tracing. ### Static Instrumentation: Tracepoints The Linux kernel provides a way to add **statically defined instrumentation hooks** called **tracepoints**. These can be embedded directly into the code at meaningful locations using the `TRACE_EVENT()` macro. Although tracepoints use the **ftrace** infrastructure under the hood, this is completely transparent to developers. By following a few guidelines, developers can gain full access to the ftrace parser. Tracepoints are inserted at compile time, resulting in call sites in the kernel that check whether the tracepoint is enabled. If it’s disabled, the overhead is minimal—just a tiny time and space penalty. If enabled, it calls the registered function, executes the handler logic, and returns to the original caller. Tracepoints are **safe and easy to use**: - They’re part of the official kernel API. - The data structures they expose are stable, so you don’t need to worry about kernel version changes breaking your code. - And they’re **lightning-fast**, thanks to their use of ftrace hooks, as described earlier. ### Comparing the Three | Feature | Tracepoints | fprobe | kprobe | |---------------------|-------------------------------------|------------------------------------|---------------------------------| | Inserted | Statically | Dynamically | Dynamically | | Insertion time | At **compile-time** | At **runtime** | At **runtime** | | Hooks into | Specific kernel events | Function entry/exit | Arbitrary instruction | | Cost when inactive | Zero | Low | Medium–high | | Stability | ✅ Stable ABI | ❌ May break across versions | ❌ May break across versions | | Flexibility | Medium (predefined points only) | High (any function) | Very high (any instruction) | As a Rule of Thumb: - **Use tracepoints** when: - You need **stable, structured data** - You’re tracing **commonly monitored** behavior - You control or maintain the code being instrumented - **Use fprobe** for: - High performance **generic function tracing** - **Use kprobe** for: - **Everything else**, especially when other options aren’t applicable At Riptides, we chose to rely on **tracepoints**—primarily because **we're the developers of the module** that needs tracing. By using tracepoints, we ensure that our tracing remains **stable across kernel versions**, since we control the data structures. Additionally, the meaningful events we want to observe happen **within the functions themselves**. Without tracepoints, we’d be forced to use kprobe—a powerful but more fragile and error-prone method, especially for internal logic. ## Deep Dive: Kernel Tracepoint API Over time, several iterations of the kernel tracing system have led to the development of the **TRACE_EVENT** macro, which is now the **standard and stable** way to create tracepoints. If you're interested in the historical evolution, [LWN.net](https://lwn.net/Articles/379903/) has several excellent articles on the topic. However, this blog will focus solely on the modern TRACE_EVENT interface. The TRACE_EVENT macro allows you to insert a tracepoint at a specific location in your code and automatically integrates it with ftrace. This macro is defined in linux/tracepoint.h. To define a fully functional tracepoint, you have to meet couple of requirements: - A tracepoint definition which can be placed within kernel or kernel module code - A callback function to handle the event - The callback must record incoming data into the tracer’s ring buffer - A "stringer" function that formats the data into a human-readable string for output Let’s take a look at how the `TRACE_EVENT` macro manages all of this under the hood.: **TRACE_EVENT(name, proto, args, struct, assign, print)**: Each TRACE_EVENT definition includes six components: - **name:** The unique name of the tracepoint. Tracepoint names must be globally unique across the kernel, so choose carefully. - **proto:** The prototype of the tracepoint callback function. - **args:** The arguments passed to the tracepoint, matching the prototype. - **struct:** The data structure used to store the tracepoint's data. - **assign:** Code that assigns values to the fields of the structure. - **print:** A "stringer" function that formats the stored data for display in human-readable form (ASCII). With the exception of name, all other fields are written using specific helper macros for readability and correctness like TP_PROTO, TP_ARGS, TP_STRUCT, TP_fast_assign, TP_printk. Let’s walk through an example TRACE_EVENT that logs a UUID and a timestamp. **TRACE_EVENT(stamp_with_ts_and_uuid**: This is the name of the tracepoint, and it will be used when invoking the tracepoint from your kernel module. Internally, the macro adds a trace_prefix, so the actual function becomes: trace_stamp_with_ts_and_uuid() This name must be **globally unique** across the entire kernel. **TP_PROTO(const uuid_t \*trace_id)**: This defines the prototype of the tracepoint callback. It sets the function signature that both the tracepoint and its handler will use. Keep in mind: the callback executes in the same context as where the tracepoint is called, so it must be fast and safe. **TP_ARGS(trace_id)**: This macro lists the actual arguments to be passed when calling the tracepoint. It's needed because the macro expands into a function call, and the compiler must know what values to pass. Think of like this: ```c // Function prototype void my_tracepoint(int a, int b); // <--TP_PROTO // Function invocation my_tracepoint(a,b); // <--TP_ARGS ``` **TP_STRUCT__entry()**: ```c TP_STRUCT__entry( __array(u8, uuid, UUID_SIZE) __field(s64, trace_timestamp)) ``` This macro defines the structure used to store trace event data in the tracer’s ring buffer. If you weren’t already fond of macros, get ready—they’re everywhere here. Within `TP_STRUCT__entry`, you'll find other specialized macros for defining fields: - `__field(type, name)`: Declares a standard scalar field (e.g., an int, s64, etc.) - `__array(type, name, len)`: Declares a fixed-length array - `__string(name, src)`: Declares a null-terminated string - `__dynamic_array(type, name, len)` : Declares a variable-length array with the size provided by the third parameter Special variable `__entry` , is introduced here. It represents a pointer to this structure and points directly into the ring buffer during assignment. **TP_fast_assign()**: ```c TP_fast_assign( memcpy(__entry->uuid, trace_id->b, UUID_SIZE); __entry->trace_timestamp = ktime_get_real_ns();) ``` This macro defines how data is assigned to the fields declared in TP_STRUCT__entry. No nested macros here—just plain C code using the__entry pointer to store data directly into the ring buffer. You can use any arguments listed in TP_ARGS() here. For example: - Simple scalar fields (like `__entry->trace_timestamp`) can be set with direct assignment. - For arrays (like `__entry->uuid`), use memcpy() to populate the buffer. There are two more helper macros available in this context: - `__assign_str(name, src)` – Used to assign values to fields declared with `__string` - `__get_dynamic_array(name)` – Returns a pointer to a dynamic array declared via `__dynamic_array`, which you can then populate with a memcpy **TP_printk()**: ```c TP_printk("uuid:%d, timestamp:%lld", __entry->uuid, __entry->trace_timestamp)); ``` The `TP_printk()` macro defines a format string for rendering the tracepoint data, similar to printf(). It controls how the contents of the __entry struct are displayed in user space (e.g., via `trace-cmd` or `perf`). The format string follows standard printf syntax, and __entry is again used to reference the structure fields defined earlier in TP_STRUCT__entry. Several helper macros are also available: - `__get_str(name)`: Retrieves a pointer to a `__string` field - `__get_dynamic_array(name)`: Retrieves a pointer to a `__dynamic_array` field (also used internally by __get_str) Now that we’ve covered the key components of the **tracepoint API**, there’s one piece that keeps coming up but hasn’t been discussed in detail yet: the ring buffer where trace events are stored. In the next section, we’ll briefly explore how this buffer works and how events move through it. ### Tracer's Ring Buffer The ring buffer, also known as a circular buffer, is a core component of the Linux kernel tracing subsystem. It uses a fixed-size buffer where the end wraps around to the beginning—allowing for efficient, continuous data storage without dynamic memory allocations. **Key advantages:** - Constant-time reads and writes - No memory allocation during writes - Lockless operation for performance The kernel tracing ring buffer can operate in two modes: - `Overwrite mode`(default): If the reader cannot keep up, the oldest data is overwritten to make room for new events. - `Producer/consumer mode`: If the buffer is full, new data is dropped, preserving older entries. To avoid contention and ensure scalability, the kernel allocates a separate ring buffer per CPU. This per-CPU design reduces locking and improves performance on multicore systems. When a trace event occurs, the TP_fast_assign() macro populates the relevant data fields directly into the ring buffer. These stored events can later be retrieved using tracing tools such as trace-pipe, perf, or strace. ## Conclusion In this first blog post, we explored the fundamentals of Linux kernel tracing: - Dynamic probes (kprobe/kretprobe) for general-purpose, on-the-fly instrumentation - Narrower, high-performance hooks via fprobe and the ftrace infrastructure - Static, rock-solid tracepoints powered by the TRACE_EVENT API - The role of the per-CPU ring buffer in efficiently capturing and storing event data By choosing tracepoints for our Riptides module, we gain stability across kernel versions, minimal overhead when disabled, and structured, human-readable data when enabled. Stay tuned for **Part 2**, where we’ll dive into how Riptides retrieves and processes trace events in user space—and demonstrate how to leverage familiar tools like strace and perf trace to debug and validate your tracepoints. --- ## The Riptides Vision: Identity-First Infrastructure - URL: https://blog.riptides.io/the-riptides-vision-identity-first-infrastructure - Published: 2025-04-23 - Author: Marton Sereg - Category: Identity - Tags: identity, spiffe, AI In modern digital infrastructure, non-human identities have exploded in both number and significance. But what is even a non-human identity? The terminology itself has sparked debates among people in the space, and for good reason. Even though we like to throw the term identity around, a non-human identity is usually just a credential for an application, workload, or device, allowing it to prove who it is when connecting to other systems. Today, these machine credentials far outnumber human users in most organizations. **Organizations now report machine-to-machine credentials outnumber human credentials by double-digit factors.** **Cloud-native computing has multiplied non-human identities** in general. Applications are broken into microservices, deployed across hybrid and multi-cloud environments, and integrated with third-party APIs. The rise of autonomous AI agents is accelerating this shift even further. And the trend is clear: AI agents will operate with minimal oversight, accessing sensitive data and interact with a range of services across networks. Securing these credentials will no longer be optional - each service account, token, or certificate is a potential entry point for attackers if compromised. The stakes are high: when non-human identities fail, we see everything from sensitive data exposure to chaotic outages. Clearly, the **importance of non-human identity** is no longer academic – it is a **central security concern** in the AI and cloud-native era. This secret sprawl, coupled with the rise of autonomous software, demands a holistic solution: a unified identity fabric that can serve as the foundation for secure, scalable, machine-to-machine trust. ## The Fragmentation Problem: Keys, Tokens, and Secrets Everywhere Today, machine identity is fragmented and inconsistent. Most organizations lack a unified approach, instead relying on a patchwork of secrets and credentials spread across different systems. Consider a typical scenario: an application uses a cloud API key for one service, a database password for another, a TLS certificate for in-datacenter calls, and perhaps an OAuth token for a SaaS integration. Each credential is stored and issued in a different place: some in code or YAML configs, others in a secrets manager, others manually configured. Each system has its own method for proving identity, with little cohesion or governance. This **scattered approach to key management** creates silos and complexity. Security teams struggle to maintain visibility and consistent policies when **credentials are managed by different owners and tools in isolation.** They have to navigate: - **Secret Sprawl and Exposure:** With countless API keys, tokens, and certificates floating around, the attack surface expands, increasing the chance that one leaks or is stolen. - **Operational Burden:** Managing and rotating all these disparate keys is burdensome and error-prone. Most organizations face frequent surprises like expired certs causing outages or forgotten tokens with excessive access. - **Inconsistent Trust:** Each system might use a different trust model. This inconsistency makes it hard to **enforce any unified security policy** or to enable end-to-end encryption universally. - **Over-Privileged Access:** Ad-hoc machine credentials often violate least privilege. For example, static tokens are commonly set with broad permissions and never expire. Our current approach leaves us juggling a jungle of secrets without a central source of truth. **Machine-to-machine communications remain secured in a piecemeal fashion**, if at all. As workloads become more dynamic and AI agents more capable, this model doesn’t scale. We need an infrastructure-native way to establish trust that doesn’t rely on manually managing secrets. ## A Unified Identity Solution The Riptides approach is simple but powerful: assign each non-human actor a cryptographically verifiable identity, issued and rotated automatically, recognized across your environment. In practice, this means assigning each workload a X.509 certificate, or signed token that serves as its **identity “badge.”** This digital identity can then be trusted by any other workload or agent to verify who it’s talking to. Crucially, the issuance and rotation of these identities are handled automatically by Riptides, not by developers baking keys into code. Think of it as an **“identity fabric”** woven through the infrastructure: rather than manually stitching together credentials between every pair of services, each service is born with an identity that can be universally verified. This unified approach yields several technical benefits: - Transparent Mutual Authentication — When every workload presents a verifiable identity, we can enforce mutual TLS authentication behind the scenes. - Secrets-Free Workflows — A unified identity system avoids the need for hard-coded secrets. Long-lived API keys and passwords can be phased out in favor of **short-lived cryptographic credentials** issued on the fly. We can issue certificates or tokens that are **ephemeral and automatically rotated.** - Unified Trust Anchors — Riptides leverages trusted Certificate Authorities (CAs) to issue identities for all services. This provides a single chain of trust. Instead of each team rolling their own keys, they all rely on the common identity service. It also enables federation: your domain’s identities can be recognized by a partner or cloud provider, extending trust without sharing raw secrets . - Improved Auditability and Governance — When every non-human actor authenticates with a trusted identity, it becomes easier to audit who did what. Policies can be written in terms of identities (“Workload A may talk to Service B”) rather than low-level constructs like IP addresses or ad hoc API keys. In essence, a unified identity solution **turns network security inside-out**: trust is attached to identities, not network locations, and security follows the workload wherever it runs. This is especially powerful for highly dynamic and scalable environments, - a container spinning up in Kubernetes, an agent calling an external MCP server, or a microservice handling payments can all use the same trusted foundation to authenticate and communicate. ## From Theory to Practice: SPIFFE, Kernel-Level Integration, and Beyond To bring this vision to life, we lean on standards like SPIFFE, which defines a uniform way to issue and verify workload identities across diverse environments. At its core, SPIFFE defines a format for *workload identities* (called SPIFFE IDs) and a method for issuing cryptographic identity credentials (like X.509 certificates or JWTs) to any workload that needs one. But we go further. At Riptides, we envision the identity fabric extending into the Linux kernel itself, enabling identity enforcement and propagation at the OS level. Through mechanisms like our Kernel driver, or eBPF, we can: - Attach verified identities directly to processes. It allows for fine-grained access control, where malicious processes can't hijack the identity of another workload, even if it runs on the same node, or in the same pod. - Enforce policies at the syscall level, for example, allowing only processes with verified identities to open outbound sockets, or initiate inter-process communication. - Automatically handle mutual TLS using the certificates proving machine identity at the socket layer, without developers needing to manage keys in user space. This makes identity enforcement fully transparent for both application developers and the network stack. No more TLS boilerplate. No more key management in user space. Just cryptographically backed trust, enforced in Kernel-space. By treating identity as a first-class primitive of the operating system, we make it universal, scalable, and invisible. ## Summary A strong identity fabric unlocks secure, scalable, and simple infrastructure. When every workload, microservice, or AI agent is born with a verifiable identity, we can **dramatically simplify network security**: every connection is mutually authenticated and authorized based on identity, not on brittle network trusts or shared secrets. By adopting unified, cryptographic identities for workloads and agents, we can **secure communications across AI agents, data center services, and third-party integrations in a consistent, and transparent way.** This approach paves the way for true zero-trust connectivity at scale. Riptides is building the infrastructure that makes this possible. Because in a world where machines talk to machines, identity is everything. --- ## Introduction to SPIFFE: Secure Identity for Workloads - URL: https://blog.riptides.io/introduction-to-spiffe-secure-identity-for-workloads - Published: 2025-04-23 - Author: Janos Matyas - Category: Identity - Tags: spiffe, identity, zero-trust The Riptides Platform is a comprehensive solution for securing workload-to-workload communication, built on a foundation of identity. It provides a universal and transparent non-human identity solution that secures every connection between workloads and services. One of the pillars of the Riptides’ identity platform is **SPIFFE**, the Secure Production Identity Framework for Everyone. With this post, we aim to introduce SPIFFE and highlight some of the approaches we use. While this is not a deep technical post, we encourage you to subscribe to our newsletter if you are interested in SPIFFE, as we will be following up with in-depth technical details of our usage and experiences with SPIFFE. As modern infrastructure becomes increasingly dynamic, traditional authentication and authorization mechanisms struggle to keep up. The growing adoption of cloud-native architectures, ephemeral workloads, and zero-trust principles has led to a demand for robust workload identity solutions that do not rely on static credentials. SPIFFE provides a standardized way to authenticate and authorize workloads across heterogeneous environments without relying on hardcoded credentials or secrets management systems. By leveraging cryptographic workload identities, SPIFFE enables a scalable and secure approach to workload-to-workload authentication, independent of network boundaries. In this post, we will introduce SPIFFE, explain its key components, and then dive deeper into its core identity format, including how SPIFFE IDs are structured, followed by SPIFFE Verifiable Identity Documents (SVIDs) and Trust Domains. ## The Need for SPIFFE Before diving into the specifics of SPIFFE, it is essential to understand the problems it aims to solve: 1. **Static Credentials Are a Security Risk**: Traditional authentication mechanisms often rely on API keys, passwords, or certificates stored in environment variables or configuration files. These credentials can be compromised if not properly managed. 2. **Infrastructure Is Dynamic**: Cloud environments and microservices architectures mean workloads are constantly spinning up and down, making it difficult to manage identity using static identifiers like IP addresses. 3. **Cross-Cloud and Hybrid Deployments**: Modern applications run across multiple clouds and on-premises data centers, creating challenges in establishing a unified trust model. 4. **Zero Trust Requirements**: Organizations are increasingly adopting zero-trust principles where workload identity and authentication must be independent of the network location. SPIFFE addresses these issues by providing a universal identity standard for workloads that is cryptographically verifiable and independent of infrastructure details. This aligns seamlessly with the core objectives of the Riptides platform. We have found SPIFFE to be a robust and flexible standard, making it well-suited for unifying all non-human identities we managed with the Riptides platform under a single framework, regardless of their varying authentication mechanisms. ## What Is SPIFFE? SPIFFE defines a set of standards for securely identifying and authenticating workloads. It consists of: - **SPIFFE ID**: A unique identifier assigned to a workload within a trust domain. - **SVID (SPIFFE Verifiable Identity Document)**: A cryptographically verifiable document that asserts a workload's SPIFFE ID. - **SPIFFE Workload API**: A standardized interface that workloads use to retrieve their identity. - **Trust Domain**: A logical boundary that defines a security context for workloads under a single administrative control. ## The SPIFFE Identity Format At the heart of SPIFFE is its identity format, which provides a standardized way to refer to workloads. The format is based on URIs and follows this structure: ``` spiffe:///path ``` ### Approaches to Structuring the Workload Identity Path There are multiple ways to structure the workload identity path component of a SPIFFE ID. While SPIFFE does not prescribe a specific approach, each format has its own advantages and trade-offs: #### **Hierarchical Structure** ``` spiffe://example.org/service/db spiffe://example.org/service/web spiffe://example.org/service/cache ``` **Pros:** Simple organization, easy policy enforcement. **Cons:** Lacks environment or role context. #### **Descriptive Names** ``` spiffe://example.org/frontend/web spiffe://example.org/backend/auth ``` **Pros:** Expressive, differentiates services. **Cons:** Complexity grows with descriptors. #### **Environment Segmentation** ``` spiffe://example.org/prod/frontend/web spiffe://example.org/staging/frontend/web ``` **Pros:** Clear separation of environments. **Cons:** More identity variations to manage. #### **Role-based Identification** ``` spiffe://example.org/service/auth/client ``` **Pros:** Enables least-privilege access. **Cons:** May be ambiguous if services have multiple roles. #### **Versioning** ``` spiffe://example.org/v1/service/web ``` **Pros:** Supports rolling deployments. **Cons:** Complexity in managing versions. ## SPIFFE Verifiable Identity Document (SVID) An **SVID** is the mechanism by which a workload proves its SPIFFE identity to another workload. It is a cryptographically signed document, typically in the form of an **X.509 certificate** or a **JWT**, that contains the SPIFFE ID of the workload and is issued by an entity trusted within the **trust domain**. SVIDs enable secure mutual authentication between workloads without requiring pre-configured secrets or static credentials. ## Trust Domain A **trust domain** in SPIFFE represents an administrative boundary within which workloads are identified and authenticated. It is specified within a SPIFFE ID as the domain name (e.g., `spiffe://example.org`). Trust domains allow organizations to separate security contexts and define how workloads within a domain trust each other, as well as how they federate trust with external domains. ## SPIRE and Custom Implementations While SPIFFE is a specification, its most widely used implementation is **SPIRE (the SPIFFE Runtime Environment)**, which helps manage SPIFFE identities in practice. SPIRE is responsible for issuing and rotating SVIDs, attesting workload identities, and enforcing identity policies. However, SPIRE primarily focuses on identity generation and certificate rotation, with limitations in policy enforcement and encrypted communication between services. At **Riptides**, we have developed our own implementation of SPIFFE, addressing these gaps by: - Defining and enforcing **fine-grained access policies** beyond identity issuance. - Securing **end-to-end encrypted communications** between services based on their workload identity. - Integrating **custom attestation mechanisms** for more flexible and secure workload authentication. While SPIRE is a great reference implementation, organizations with more advanced security and policy requirements may benefit from a custom SPIFFE-based identity management system tailored to their specific needs. However, if your organization is already using SPIRE, the Riptides platform can seamlessly integrate with it. Our goal in developing our custom solution went far beyond SPIRE’s capabilities. The Riptides platform is designed to provide a **comprehensive security management layer**, where security teams define access policies while Riptides handles the rest—including **certificate issuance, revocation, and rotation; automatic encryption of communications; secure identity exchanges for third-party credentials; and much more**. ## Conclusion SPIFFE provides a standardized and secure way to assign cryptographic identities to workloads, enabling workload-to-workload authentication in modern cloud-native environments. However, implementations like SPIRE focus primarily on identity issuance and certificate management, leaving gaps in policy enforcement and encrypted communication. At **Riptides**, we have addressed these challenges with a custom SPIFFE-based solution that ensures robust workload authentication, policy-driven access control, and encrypted service-to-service communication. As zero-trust adoption grows, SPIFFE will continue to be a critical component in securing cloud-native workloads. --- ## Riptides: Kernel-Level Identity and Security Reinvented - URL: https://blog.riptides.io/riptides-kernel-level-identity-and-security-reinvented - Published: 2025-04-23 - Author: Nandor Kracser - Category: Kernel - Tags: spiffe, kernel, mtls, linux, identity ## Reinventing Non-Human Identities with Kernel TLS At Riptides, we’re reshaping how non-human identities (NHIs) are managed and secured. In a world where AI agents, microservices, and containers constantly evolve and communicate, traditional identity systems—based on static or half-dynamic credentials and manual configurations—are no longer enough. Our solution introduces a custom Linux kernel module, seamlessly integrated with a user space agent and a centralized control plane that serves as the root Certificate Authority (CA). Together, they establish SPIFFE IDs and enforce mutual TLS (mTLS) via kernel TLS (kTLS) possible between all components of your system, ensuring each process is uniquely identified and securely connected. ## The Challenge: Identity in Motion Modern applications, especially AI agents, are inherently dynamic. They scale on demand, shift workloads, and communicate across distributed systems. Managing identity in this fast-moving environment presents several key challenges: - **Scalability:** Supporting large volumes of transient processes requires an identity system that scales effortlessly. - **Security:** Manual credential handling introduces risk through human error. - **Automation:** Rapid provisioning and deprovisioning of services demand a hands-off approach to identity management. ## The Riptides Solution: Dual-Layer Identity Architecture We tackle these challenges with a tightly integrated system that combines kernel-level security with user space flexibility. ### Kernel Module with kTLS: Secure by Design Our Linux kernel module anchors identity at the system level, offering deep integration that makes security unavoidable and invisible to the application layer. - **Socket Lifecycle Interception**: The module hooks directly into the socket creation process, intercepting every connection and ensuring it carries secure identity metadata from the outset. - **SPIFFE ID Assignment**: In tandem with the user space agent, the module assigns unique, standardized SPIFFE IDs to each process, providing consistent identity across distributed systems. - **mTLS with Hardware Acceleration**: Leveraging kTLS, mutual TLS handshakes and encryption operations are offloaded to the kernel and accelerated using hardware capabilities. This reduces latency while preserving high security standards. - **Dynamic Identity Updates**: As policies evolve, the kernel module dynamically propagates changes to relevant processes, minimizing risk exposure and ensuring continuous compliance. ### User Space Agent and Control Plane: Policy and Authority While the kernel handles the heavy lifting, our user space agent and control plane provide centralized coordination and policy enforcement. - **User Space Agent Component:** Acts as a bridge between the kernel and control plane, synchronizing identity data and managing logs, policy updates, and lifecycle events. - **Policy Framework:** Defines how identities are assigned and who can talk to whom. Rules are authored centrally and distributed securely, enabling fine-grained control over inter-process communication. - **Control Plane (Root CA):** Issues and manages SPIFFE-linked certificates. It oversees provisioning, renewal, and revocation through continuous interaction with the user space agent, ensuring credentials remain current and trustworthy. ## Open, Efficient, Trusted Kernel Component Because the kernel module plays a critical role, trust and transparency are essential. We plan to open-source the module, allowing users to build it using DKMS or download attested binaries from our repository. - **Performance Optimization**: Operating in kernel space reduces overhead and latency. Offloading cryptographic operations to kTLS improves performance, freeing applications to focus on their core logic. - **Real-Time Enforcement**: Security policies are updated and enforced in real time, keeping all system processes aligned with the latest configurations. - **Seamless Integration**: Applications benefit from secure communication channels without needing to implement mTLS logic themselves. Everything happens under the hood. ## Perfectly Suited for AI Agents AI agents are an ideal use case for our architecture, given their need for speed, adaptability, and security. - **Instant Provisioning**: Automated identity issuance allows agents to come online securely in seconds. - **Adaptive Trust**: Real-time updates ensure that even as agents change behavior or context, they remain properly authenticated. - **Minimal Overhead**: Hardware-accelerated mTLS keeps communication fast and secure. - **Controlled Communication**: Our policy framework enforces strict boundaries, allowing only authorized interactions among agents. ## Conclusion: Identity at the OS Layer Riptides is redefining non-human identity by embedding trust deep into the operating system. Through the use of SPIFFE IDs, kernel-based mTLS, and a dynamic policy framework governed from a centralized control plane, we deliver a solution that is secure, scalable, and built for the fluid demands of modern software — especially the fast-growing world of AI. By anchoring identity in the kernel and orchestrating it via a flexible user space layer, we’re creating a foundation where secure communication is the default, not an afterthought.