Machines Have Identities Too

47.0 What this chapter gives you#

1. You will be able to explain why a running program needs an identity of its own, and name the four things that identity can be bound to.
2. You will be able to list the six routes by which a long-lived API key escapes into the world, and name the detection that catches each one.
3. You will be able to quote current figures for how many secrets leak, how many are still valid years later, and use them to argue for a rotation budget.
4. You will be able to describe mutual TLS between two services precisely, and say what a certificate proves and what it does not.
5. You will be able to read a SPIFFE ID aloud, say what a trust domain is, and explain the difference between an X509-SVID and a JWT-SVID.
6. You will be able to describe node attestation and workload attestation as two separate steps, and say what breaks if either is weak.
7. You will be able to write the two HTTP requests that fetch a cloud instance identity document under IMDSv2, and explain exactly which attack the first request stops.
8. You will be able to trace the 2019 Capital One intrusion link by link, and say which single configuration change would have broken the chain.
9. You will be able to configure a Kubernetes service account so that a pod can call a cloud API with no static key anywhere in the system, and read the resulting token’s claims.
10. You will be able to explain what a TPM quote contains, what it proves about a machine, and the precise limit of that proof.

At three in the morning nobody is logging in. The office is empty, the phones are silent, and the number of humans typing a password anywhere in your company is zero. The number of authentications happening is not zero. It is enormous. A batch job wakes up and asks a database to let it in. A container starts and asks a cloud provider for permission to write a file. A message queue connects to a broker. A monitoring agent posts metrics. A payment processor’s webhook arrives and must prove it is the payment processor. Each of those is an identity being presented and checked, and there is nobody in the building.

That is the thesis of this chapter. Most authentications on earth are one program proving itself to another, and they outnumber humans by orders of magnitude. Palo Alto Networks, in its 2026 Identity Security Landscape report published in May 2026 and based on a survey of more than 2,900 security decision-makers, put the ratio at 109 machine identities for every human identity, up from 82 to one the year before. That is a vendor survey and should be read as one, not as a census. But every practitioner who has ever counted the credentials in their own estate recognizes the shape of it. Whatever the exact number, the machines win by a wide margin, and they are the population that identity practice has historically governed worst.

This chapter is about how a program proves who it is. We will start with the oldest and worst answer, which is to give the program a long password and hope, and we will walk forwards through short-lived certificates, workload attestation, cloud metadata services, Kubernetes tokens, federation between clouds, and finally hardware that can vouch for the machine underneath. Along the way we will follow one concrete piece of work all the way through: an invoice that must be written to storage at 03:07 on Wednesday 19 August 2026 by a program with no human anywhere near it.

Two neighbouring subjects belong elsewhere. What the program is permitted to do once it has proved who it is, which is the harder question, belongs to chapter 46 on authorization. How confident you are allowed to claim to be about any of this, expressed as a graded and auditable level, belongs to chapter 48 on assurance. This chapter is only about the proof of who.

The plain version#

The depot that never closes#

Picture a very large distribution depot on the edge of a city. Lorries arrive all night. They come from the port, from three warehouses, from a bonded store, and from a partner company. Each one drives up to a gate, and something has to decide whether to raise the barrier.

In the daytime this is easy, because there is a gatekeeper called Ravi who has worked there eleven years and knows every regular driver by face and by the sound of their engine. At three in the morning there is no Ravi. There is a barrier, a machine, and about four hundred lorries a night. The depot cannot employ a Ravi for every gate, and even if it could, the lorries are arriving faster than a human could look at them.

So the depot has to work out a way for a lorry to prove it belongs, without any person being involved. That is the whole problem. Everything in this chapter is a different answer to it.

Notice first what the depot is actually trying to establish. It is not trying to work out who is driving, because at three in the morning nobody is driving; these are automated vehicles. It is trying to establish which lorry this is, whose lorry it is, what it is here to do, and whether the thing at the gate is really that lorry rather than something painted to look like it. Those are four different questions and they get tangled together constantly.

There is one more feature of the depot that matters more than anything else. Lorries are not permanent. This depot does not own four hundred lorries. It hires them by the hour. A lorry appears, does one job, and is scrapped. Tomorrow night there will be four hundred different ones. Any scheme that requires somebody to write a name on a list before the lorry exists is going to fail, because the lorry did not exist five minutes ago and will not exist in an hour.

The pass in the glovebox#

The oldest answer, and still by a wide margin the most common one, is a printed pass.

When a lorry is first put into service, somebody at the depot office prints a pass, drives out to the vehicle, and puts it in the glovebox. The pass is a long string of letters and numbers. At the gate, the lorry holds the pass up to a reader, the reader compares it to a list, and the barrier goes up.

This works. It also has a set of problems that are worth being very precise about, because every one of them has a direct counterpart in software.

The pass never expires. It was printed in 2019 and it still opens the gate in 2026. Nobody remembers who printed it or why. The person who printed it left the company.

The pass is a copy of itself. Anybody who sees it has it. A photograph of the pass is as good as the pass. There is no way for the gate to tell the difference between the lorry that was issued the pass and a van that has a photograph of it.

The pass gets written down in places. Because drivers kept losing them, somebody started painting the pass number on the inside of the cab door. Because the maintenance depot needed it, somebody wrote it in the service manual. Because a contractor needed to test the gate, somebody sent it in a message. Now the pass exists in six places, and only one of them is the glovebox.

Nobody knows how many passes exist. The office has a list, but the list is of passes issued, not passes in use. Some passes on the list belong to lorries that were scrapped four years ago. Some lorries at the gate are using passes that are not on the list at all, because they were issued by a different office during a busy period.

Replacing a pass is frightening. The office knows perfectly well that the 2019 pass should be replaced. But nobody is sure which lorries are using it, and if they cancel it and one of those lorries is carrying refrigerated goods, the goods spoil. So the pass is not cancelled. It is never cancelled. It just accumulates.

That last point is the one that matters most, and it is worth saying plainly: the reason old credentials survive is not that people do not know they are dangerous. It is that cancelling them is an action with a known cost and an unknown blast radius, and nobody will take that trade at three in the morning.

The pass that expires at dawn#

A better depot does something different. It stops printing passes that last forever, and starts printing passes that last one night.

When a lorry is put into service, it is not given a pass. It is given a way to ask for one. Every night, before it sets off, it asks the depot office for tonight’s pass, and the office prints one that stops working at six in the morning.

Look at what this fixes. The photograph of the pass on somebody’s phone is worthless tomorrow. The number painted inside the cab door is worthless tomorrow. The pass in the old service manual is worthless. The list of passes issued four years ago describes nothing that still works. If a pass leaks, the leak has a deadline.

Look also at what it does not fix, because this is where people get careless. The lorry still has to prove something to the office in order to be given tonight’s pass. Whatever it uses to do that is now the thing that matters, and if that thing is a printed pass in the glovebox that never expires, then absolutely nothing has been achieved except adding a step. The office has become the new gate, and the old problem has moved rather than gone.

This is the single most important idea in machine identity, and the plain version of it is this: short-lived credentials are only as good as the thing that gets you the first one.

Two lorries that have to check each other#

So far only the gate is checking the lorry. In a real depot the checking goes both ways, and it has to.

Suppose a lorry pulls up to what it believes is the fuel bay and is told to hand over its cargo manifest. Should it? Only if it is really the fuel bay. Somebody who parks a shed in the right place, paints it the right colour and puts up a sign has just collected a manifest.

So both sides present something. The gate proves to the lorry that it is the depot’s real gate. The lorry proves to the gate that it is a real lorry of the depot. Neither one moves until both are satisfied. This is a mundane idea and it is astonishing how often it is left out, because building the one-directional version is easier and it appears to work.

Once both sides check, a second question appears immediately. The gate now knows this is lorry number 4471 of the partner company. It does not yet know whether lorry 4471 is allowed into the bonded store. Proving who you are and being allowed in are two separate steps, and chapter 46 handles the second one properly. Here we only care that the first step is honest.

The intercom in the yard, and the clerk who takes messages#

Now for the arrangement that caused the most famous accident in this field.

Inside the depot yard there is an intercom on a pole. Any vehicle already inside the yard can pull up to it and press the button, and a voice reads out today’s passes and the codes for the inner stores. This is convenient. It means a lorry does not have to carry anything at all; it just asks the intercom when it needs something. The security of the whole arrangement rests on one assumption: only vehicles that are legitimately inside the yard can reach the intercom.

Now add a second, entirely separate convenience. At the front office there is a clerk whose job is to take requests from the public. Members of the public write in and say “please check X for me and tell me what it says”, and the clerk, being helpful, goes and checks and writes back.

Neither of these two things is obviously wrong. Put them together and the depot is finished. A member of the public writes to the clerk and says: “Please go to the intercom on the pole in the yard and tell me what it says.” The clerk is inside the yard. The clerk is allowed to use the intercom. The clerk does not think of himself as a lorry, he thinks of himself as a helpful person answering a letter. He walks to the pole, presses the button, writes down today’s codes for the inner stores, and posts them to a stranger.

That is server-side request forgery, and that is, in outline, what happened to a large American bank in 2019. We will do it exactly, with the real components and the real dates, in the technical half. For now, hold on to the shape of it: a trusted insider that will fetch things on behalf of an outsider, plus a source of credentials that trusts anything inside.

The fix in the depot is instructive because it is so unglamorous. You do not remove the intercom. You change it so that using it requires two actions instead of one: first you must turn a physical key on the pole and be handed a numbered token, and only then can you press the button and be read the codes. The clerk can be tricked into pressing a button. He cannot easily be tricked into turning a key first, because turning a key is not a thing anybody would ask him to do in a letter. You also make the token physically unable to leave the yard: it is printed on paper that dissolves at the gate. That is, almost exactly, the change that Amazon Web Services made in November 2019.

The plate stamped into the chassis#

There is one more level, and it is the level at which the lies stop.

Everything so far is a piece of paper. Paper can be copied, photographed, forged and moved. Suppose instead that when the lorry was built, the factory pressed a metal plate into the chassis, and that plate contains a number that cannot be read out, cannot be copied, and cannot be moved to another chassis. The plate cannot tell you the number. What it can do is take a question you give it and stamp an answer that only a plate with that number could produce.

Now the depot can ask a question that no piece of paper can answer: not “do you have the right pass”, but “are you the specific physical vehicle that the factory built on the twelfth of March, and were your engine, brakes and load computer the ones the factory fitted”.

That is a hardware root of trust, and in a computer it is a small chip called a TPM, or a protected region inside the processor called an enclave. It is the only thing in this chapter that a piece of software cannot fake by copying a file, and it is the reason the whole tower of short-lived credentials can be stood up without a printed pass at the bottom of it.

It has an exact and important limit, which we will come to in the next section: the plate can prove that the right engine was fitted at the factory. It cannot prove that the engine has not been driven badly since.

One night in the depot, with real values#

Now the concrete example we will carry through the whole chapter.

Harbourline is a freight company. Its billing software runs in a cluster of machines rented from a cloud provider. At 03:07 on Wednesday 19 August 2026, a small program called invoice-api finishes calculating an invoice, INV-2026-08-14771, for 4,182.60 euros of shipping from Rotterdam to Ludhiana.

To finish its job, that program must do four things, and every one of them requires it to prove who it is:

It must read a customer record from a second program called ledger, which runs on a different machine in the same cluster.

It must write a PDF into a storage bucket at Amazon Web Services called harbourline-invoices-eu.

It must append a row to an analytics table at Google Cloud, because Harbourline’s reporting lives there and its billing does not.

It must call the payment provider’s public API to raise a collection request.

There is no human involved in any of the four. The engineer who wrote invoice-api is asleep in Pune. If the program has a password, that password was put there by somebody who is now asleep, and it will still be there in 2030. Every technique in the technical half of this chapter is an attempt to get those four steps done without any such password existing anywhere.

Where the plain version stops being true#

A lorry is a thing; a workload is not#

The depot analogy quietly assumes that a lorry is an object with edges. You can walk round it. It has a chassis number. It exists continuously from the factory to the scrapyard.

A workload has none of those properties, and this is the first place the analogy misleads.

Ask what exactly you are giving an identity to, and you will find at least six candidate answers, all of them used in practice and all of them meaning something different. Is it the process, meaning this specific running copy with this process number, which will not exist in four minutes? Is it the container, meaning this instance of an image? Is it the image, meaning the exact bytes with this content hash, of which there may be nine hundred copies running? Is it the deployment, meaning the abstract idea of “the invoice service”, which outlives every copy? Is it the account, meaning a name in a directory that a hundred unrelated things might use? Is it the machine, meaning the physical or virtual host underneath, which will be running something else tomorrow?

The honest version: there is no single correct answer, and the choice determines everything about how your system fails. Bind identity to the process and you get precision and a rotation storm. Bind it to the image and two teams sharing a base image share an identity. Bind it to the deployment, which is what most modern systems do, and you get something stable enough to write policy about, at the cost that every copy of that deployment is interchangeable to an attacker who gets inside one of them.

Rotating a key does not un-leak it#

The plain version says that a pass which expires at dawn makes a leaked photograph worthless. That is true only in the narrow window sense, and people routinely draw a much stronger conclusion than the facts support.

Rotation replaces a credential going forward. It does nothing at all about what was done with the old one while it was valid. If a key leaked on Monday and you rotated on Friday, everything the holder did between Monday and Friday still happened, and if what they did was create a second credential for themselves, or add their own public key to a trusted list, or copy the data, then rotating the first credential achieves nothing whatever.

The honest version: rotation limits future use. Revocation stops present use. Investigation is the only thing that tells you what past use occurred. Teams routinely do the first, claim the second, and skip the third.

Short-lived credentials buy you an availability problem#

A pass that lasts one night has to be reissued every night, which means the office that issues it must be reachable every night. In software this is a real and frequently underestimated cost.

If your service certificates last one hour, then your certificate authority is now a hard dependency of every service in your estate, on an hourly basis. If it is unreachable for ninety minutes, you do not have a degraded system, you have a total outage, and it arrives everywhere at once with the clean simultaneity that only a shared expiry can produce. Long-lived secrets are terrible for security and superb for availability, which is precisely why they persist.

The honest version: every reduction in credential lifetime is a transfer of risk from confidentiality to availability. The correct engineering answer is not “make them as short as possible” but “make them short enough that the leak window is acceptable, and make the issuing path more available than the things that depend on it”.

Attestation proves the start, not the present#

The metal plate in the chassis is the strongest thing in the plain version, and it is also where readers form the most confident wrong belief.

A hardware root of trust can measure what was loaded. It records, in a way that cannot be rewound, the exact bytes of the firmware, the boot loader, the kernel and, if you configure it that way, the code that was launched. A remote party can then be shown those measurements and check them against a list of what it considers acceptable.

What it cannot do is tell you what the software has done since. A kernel that was correct at boot and was compromised twenty minutes later through a memory bug produces exactly the same measurements as a kernel that is still fine. Measurement is a photograph of the starting line, not a video of the race.

There is a second, sharper limit. The gap between measuring something and using it is a gap an attacker can live in. This is a time-of-check to time-of-use problem, and in attestation it has a specific name: the measurement says a file with a certain hash was read into memory, but memory can be changed after the reading.

The honest version: attestation gives you a strong, hardware-anchored statement about the launch state of a platform, and a much weaker implication about its current state. Systems that need continuous assurance re-attest on a schedule and combine it with runtime monitoring; they do not treat one quote at boot as a permanent certificate of health.

Mutual checking proves the peer, not the request#

Two lorries checking each other establishes that each is who it claims to be. It establishes nothing at all about whether this particular request is a reasonable one.

In software this shows up constantly. A service mesh gives you mutual TLS between every pair of services, and a diagram in which every arrow is green. What that means is that the invoice-api process is genuinely invoice-api. It does not mean the request it just sent to ledger was one it should have sent. If an attacker achieves code execution inside invoice-api, they inherit its identity completely and legitimately, and every arrow stays green.

The honest version: authentication of the peer is a floor, not a ceiling. What that peer may then ask for is authorization, which chapter 46 handles, and it is the part that actually contains the interesting decisions.

There is always a secret zero#

The most seductive claim in this field is that a given system eliminates secrets. No system does. What good systems do is move the secret to a place where it is harder to steal and easier to watch, and reduce the number of copies from thousands to one.

Trace any chain back far enough and it terminates somewhere. In a cloud, it terminates in the provider’s own control of the hypervisor: the reason an instance can prove it is an instance is that the platform that created it says so, and you trust the platform because you are paying it and because your data is on it anyway. In a data centre with a TPM, it terminates in a key the chip manufacturer put in the chip and an endorsement certificate the manufacturer signed. In a bare-bones setup, it terminates in a join token a human typed once.

The honest version: the goal is not zero secrets. It is one secret, in hardware, that never leaves the machine, is used only to sign challenges, and cannot be copied out even by the operating system. Anyone selling you the absence of a root is selling you a root you have not been shown.

One node is not one lorry#

The depot has one pass per lorry, and a lorry is one vehicle in one place. A single machine in a cluster runs forty containers belonging to nine teams, and they share a kernel.

Everything that identifies a workload on that node is enforced by software on that node. If the node’s kernel is compromised, every identity on it is compromised together, and the attacker does not need to steal anything: they simply ask the local agent for each identity in turn, using whatever proof the agent accepts. The isolation between workloads on a shared node is real and useful and it is not the same kind of boundary as the wall between two buildings.

The honest version: workload identity within a node is enforced at the strength of the container boundary, which is a software boundary. Identity between nodes can be anchored in hardware. Designs that need the stronger property either give the sensitive workload its own node, or move it into an enclave where even the operating system is outside the trust boundary.

The gate cannot see how the pass was obtained#

The last one is subtle and it undoes a lot of otherwise good designs.

When the gate reads a valid pass, it learns that the presenter holds a valid pass. It does not learn how they came to hold it. A pass obtained by a legitimate lorry through attestation and a pass obtained by an attacker who compromised the issuing office are indistinguishable at the gate, because they are the same pass.

This means the quality of every identity in the system is capped by the quality of the issuance path, and the issuance path is usually the least examined part. Teams spend months on the strength of the token and ten minutes on the question of who can ask for one.

The honest version: the strength of a workload identity is the strength of its weakest issuance route, not its strongest. If there are three ways to get an identity and one of them is a static token in a configuration file, the identity is worth exactly that.

The technical version#

What a workload identity is, and what it is bound to#

A workload is a running unit of software that needs to authenticate on its own behalf: a service, a batch job, a function, a daemon, an agent. A workload identity is a name for that unit plus a credential that lets it prove the name.

The name is the easy part. The interesting engineering is the binding: the mechanism by which the credential is only obtainable by the thing that is supposed to have it. Bindings in production use fall into five families.

Possession of a secret binds nothing. It says only that the caller has some bytes, and bytes travel. Platform assertion means the infrastructure that created the instance signs a statement about it; you are trusting the cloud provider, which you already do. Orchestrator assertion means a component such as a kubelet, which placed the container, vouches for what it placed. Process introspection means a local agent asks the kernel which process opened this socket and what its executable path, user and container identifier are. Hardware measurement means a chip on the board signs a statement about the boot state.

Real systems combine these. The strongest common pattern is hardware measurement for the node, then orchestrator or kernel introspection for the workload on that node, then a short-lived credential issued only after both succeed. That is precisely the SPIFFE model and we will come to it.

The second design question is what the name means. A workload identity that names a deployment is stable, writable in policy, and survives restarts. A workload identity that names a process is precise and useless for policy, because the policy would have to be rewritten every time the process restarts. Almost every production system names something at the deployment level: a Kubernetes service account, an AWS role, a SPIFFE path. The consequence, stated honestly, is that all replicas of a deployment share one identity and are interchangeable to an attacker inside any one of them.

The API key, and every path by which it escapes#

An API key is a long random string that a service presents to prove it is a particular client. It is a bearer credential, meaning whoever holds it is treated as the client, with no further check. In its raw form it is the machine equivalent of a shared password that is never changed.

Good key formats do three things that bad ones do not. They carry enough entropy that guessing is hopeless, typically 128 bits or more of randomness, encoded as base32 or base62 so that it is copy-safe. They carry a fixed, greppable prefix so that automated scanners can find them with certainty rather than by heuristics: GitHub’s tokens begin ghp_, gho_, ghs_ and similar, Stripe’s live secret keys begin sk_live_, AWS access key identifiers begin AKIA for long-lived users and ASIA for temporary session credentials. They carry a checksum, so that a scanner can tell a real key from a random string of the right shape without calling the provider.

Those three properties exist for one reason: keys leak, constantly, and the only workable response at scale is to find them automatically. The leak paths are well enumerated.

Two of these deserve names and dates, because they are the canonical cases.

In October 2016, attackers obtained credentials to Uber’s private GitHub organization, found an AWS access key inside a private repository, and used it to reach an unencrypted storage bucket holding records on approximately 57 million riders and drivers, including around 600,000 driving licence numbers. The technically interesting part is that the repository was private. Private did not help, because the key was valid, long-lived and unscoped, and the repository’s privacy was one compromised account away from being irrelevant. The subsequent cover-up led to the conviction of Uber’s former chief security officer, Joseph Sullivan, on 5 October 2022, on charges of obstruction of a Federal Trade Commission proceeding and misprision of a felony.

Between 31 January 2021 and 1 April 2021, an attacker repeatedly modified the Bash Uploader script distributed by Codecov, a code coverage service. The modification added a single line that posted the contents of the build environment to a server the attacker controlled. Codecov disclosed the compromise on 15 April 2021. Because continuous integration environments habitually hold every credential a build might need as environment variables, the exfiltrated material could include cloud access keys, deployment keys, service account tokens and API keys across thousands of customers at once. The lesson is not about Codecov. It is that the build system is a credential concentrator, and any code it runs inherits every secret in the environment.

Scanning, rotation, and the arithmetic of sprawl#

Secret scanning is now a standard control rather than a novelty, and there are two places to put it. Scanning at rest walks repositories, registries, ticket systems and chat archives looking for material that matches known patterns, then optionally calls the provider to test whether the credential is still live. Scanning in the path blocks the commit or the push before the secret ever lands, which is the only version that prevents rather than reports.

The scale is worth stating with real figures, because the arguments about budget are always won or lost on numbers. GitGuardian’s State of Secrets Sprawl 2026, published on 17 March 2026, reported that 28.65 million new hardcoded secrets were added to public GitHub commits during 2025, a 34 per cent increase on the previous year, against a backdrop of about 1.94 billion public commits, itself up 43 per cent. The same report put the leak rate at about 1.5 secrets per hundred commits across all public commits, rising to 3.2 per hundred for commits made with the assistance of a coding agent. It found internal repositories roughly six times more likely than public ones to contain a hardcoded secret, and about 28 per cent of incidents originating entirely outside repositories, in tools such as Slack, Jira and Confluence.

The number that should change behaviour is a different one. The same report retested a set of credentials that had been confirmed valid in 2022. In January 2025, nearly 70 per cent were still valid. Retested in January 2026, more than 64 per cent were still valid. Four years after public exposure, roughly two thirds of leaked credentials still worked.

That figure is the whole argument for workload identity in one line. Detection has improved enormously. Remediation has barely moved, because remediation requires knowing which running systems use a credential, and almost nobody knows that.

The arithmetic of rotation is simple and unforgiving. If you hold N long-lived credentials and your policy is to rotate every T days, you are performing N divided by T rotations per day, forever, each with a small probability of breaking something. At a thousand credentials and a ninety-day policy that is eleven rotations a day. Nobody does eleven careful rotations a day, so the policy silently becomes a document rather than a practice. The only way out of the arithmetic is to make the number of long-lived credentials small, which means issuing short-lived ones automatically instead, which is the rest of this chapter.

Mutual TLS for workloads, and certificates at machine scale#

Mutual TLS, usually written mTLS, is ordinary TLS with the optional client certificate made mandatory. In a normal web connection the server proves its identity with a certificate and the client proves nothing at the TLS layer. In mutual TLS the server sends a CertificateRequest message, the client responds with its own certificate chain and a CertificateVerify message containing a signature over the handshake transcript made with the private key matching that certificate, and the server checks the chain against its trust store. Both ends finish the handshake knowing a verified name for the other.

The security property that matters is that the client’s private key never crosses the wire. Unlike a bearer token, a certificate cannot be replayed by someone who observed the traffic, because the proof is a fresh signature over data unique to this handshake. That single property is why certificates beat API keys for service-to-service work, and it is worth more than any amount of key rotation discipline.

At machine scale, certificates change character. The public web PKI issues a certificate to a name a human proved they control, and the certificate lasts months. An internal workload PKI issues a certificate to a process that was created eleven seconds ago and will be gone in an hour. The lifetimes differ by three orders of magnitude and so do the operational problems.

The public web is nonetheless moving hard in the same direction, and the schedule is now fixed. Ballot SC-081v3 of the CA/Browser Forum, whose voting period closed on 11 April 2025 with 25 certificate issuers in favour and none against, sets this timetable in the TLS Baseline Requirements.

As of August 2026 the first step is in force: a publicly trusted TLS certificate may not exceed 200 days. The second column matters more than the first for anyone running infrastructure, because it is the reuse of validation evidence, not the certificate lifetime, that determines how much of the process can be cached. When domain validation evidence may only be reused for 10 days, manual issuance stops being viable in any form.

Inside a cluster the numbers are far more aggressive. A SPIRE deployment, using its shipped defaults, issues X.509 identity documents with a one hour lifetime, JSON identity tokens with a five minute lifetime, and runs its own intermediate certificate authority with a 24 hour lifetime. Rotation is not an operational event at those numbers; it is a background loop, and it must never involve a human or a restart.

The three things that break at machine scale, in the order they break, are these. Clock skew, because a certificate valid from 03:07:00 is not valid on a machine whose clock says 03:06:55, and at one hour lifetimes a thirty second skew is a large fraction of your safety margin. Trust bundle distribution, because rotating the root means every peer must learn the new root before the old one stops being used, which requires an overlap period and a mechanism to push bundles. Revocation, because certificate revocation lists and online status checks do not work at this rate and are not used; the actual revocation mechanism for a one hour certificate is waiting one hour, which is why the lifetime is one hour.

SPIFFE: the identity, the trust domain, and the two document formats#

SPIFFE, the Secure Production Identity Framework For Everyone, is a set of open specifications for naming and issuing workload identities. It was accepted into the Cloud Native Computing Foundation sandbox in 2018, moved to incubation in 2020, and graduated on 20 September 2022 together with its reference implementation, SPIRE.

A SPIFFE ID is a URI. The specification defines it as an RFC 3986 compliant URI comprising a trust domain name and a path, in the form spiffe://trust-domain-name/path. The scheme must be spiffe, the trust domain must be non-empty, and there must be no query or fragment component.

The trust domain is the authority component, and it is the root of trust of a system: everything within one trust domain is issued by the same authority and can validate everything else in that domain without further configuration. The specification restricts it to lowercase letters, digits, dots, dashes and underscores, forbids percent-encoding, forbids userinfo and port components, and caps it at 255 bytes. The path identifies the particular workload; its segments may not be empty, may not be . or .., may not be percent-encoded, and there is no trailing slash. Implementations must support IDs up to 2048 bytes.

The specification’s own examples show the range of conventions in use:

spiffe://staging.example.com/payments/mysql
spiffe://k8s-west.example.com/ns/staging/sa/default
spiffe://example.com/9eebccd2-12bf-40a6-b262-65fe0487d453

For Harbourline the invoice service is named:

spiffe://harbourline.example/ns/billing/sa/invoice-api

and the ledger it calls is:

spiffe://harbourline.example/ns/finance/sa/ledger

A SPIFFE ID by itself is only a name. The credential that proves it is an SVID, a SPIFFE Verifiable Identity Document, and there are two encodings.

An X509-SVID is an X.509 certificate carrying the SPIFFE ID as a URI entry in the Subject Alternative Name extension. The specification is strict: the certificate must contain exactly one URI SAN, and therefore exactly one SPIFFE ID, and a validator that sees more than one URI SAN must reject the certificate. Key usage must be present and marked critical. A leaf must set digitalSignature, may set keyEncipherment or keyAgreement, and must not set keyCertSign or cRLSign. A signing certificate must set keyCertSign and may set cRLSign. Basic constraints must set cA to true on signing certificates and false on leaves. This is the format used for mutual TLS, and its whole point is that it drops straight into any TLS library without modification.

A JWT-SVID is a signed JSON Web Token. The specification requires exactly three claims: sub set to the workload’s SPIFFE ID, aud present with one or more values, and exp present, with validators required to reject any token lacking it. The permitted signature algorithms are the nine RSA, ECDSA and RSA-PSS options from RFC 7518, namely RS256, RS384, RS512, ES256, ES384, ES512, PS256, PS384 and PS512; a token whose alg is anything else must be rejected. Note what is absent: there is no iat requirement, and there is no permitted symmetric or unsecured algorithm, which closes the alg: none and HMAC-confusion families of attack by construction.

The practical difference between the two is that an X509-SVID proves possession of a key and cannot be replayed, while a JWT-SVID is a bearer token and can be, which is exactly why its default lifetime is five minutes and why the aud claim is mandatory. You use the X.509 form when you control both ends of the connection and the transport is TLS. You use the JWT form when you must cross something that terminates TLS, such as a load balancer or a gateway you do not own.

Workloads obtain their SVIDs from the Workload API, a gRPC service exposed on a local endpoint. The address is passed in the environment variable SPIFFE_ENDPOINT_SOCKET, and the schemes permitted are unix for a Unix domain socket and tcp. The methods are FetchX509SVID and FetchX509Bundles for certificates and trust bundles, and FetchJWTSVID, FetchJWTBundles and ValidateJWTSVID for tokens.

The design decision that makes the whole thing work is this: the Workload Endpoint specification states that the endpoint must not require any direct authentication of its clients, and that whatever mechanism is used to identify the caller must not require the workload to actively participate. The workload does not present a password to get its identity. It simply connects, and the endpoint works out what it is by looking at it from outside, using kernel introspection or by interrogating the orchestrator. That is what removes the secret zero from the workload itself: there is nothing to steal from the process, because the process never had anything.

Streams are long-lived. Every response carries the complete current state rather than a delta, so a client that reconnects is immediately correct, and rotation arrives as a new message on an open stream rather than as a polling loop.

SPIRE: node attestation, workload attestation, and the chain that connects them#

SPIRE is the reference implementation, and its architecture makes the two-stage nature of the problem explicit.

A SPIRE server is the authority for one trust domain. It holds the signing key, holds the registration entries, and mints SVIDs. A SPIRE agent runs on every node and exposes the Workload API to the workloads on that node.

Node attestation is the first stage: before an agent is allowed to ask for anything, the agent and server together establish the identity of the node the agent is running on. Node attestors are plugins, and the important ones correspond to whatever the node’s platform can prove about itself: the AWS instance identity document, Google Compute Engine instance identity tokens, Azure managed identities, Kubernetes projected service account tokens, a pre-shared join token typed by a human, and TPM-based attestation where hardware is available. On success the agent receives a SPIFFE ID of its own, which becomes the parent identity for every workload on that node.

Workload attestation is the second stage, and it happens on every call to the Workload API. When a process connects to the agent’s socket, the agent interrogates the locally available authorities: the kernel, for the calling process’s user id, group id, executable path and process id; the container runtime, for the container and image; the kubelet, for the pod, namespace and service account. Each fact it discovers is a selector. The agent then compares the discovered selector set against the registration entries it has been given, and issues the SVID for any entry whose selectors are all satisfied.

A registration entry is therefore a rule of the form: any process on a node with parent identity P, exhibiting selectors S, is spiffe://.../this-name. For Harbourline:

TD=spiffe://harbourline.example
AGENT=$TD/spire/agent/k8s_psat/hl-prod-eu/NODE
IMG=harbourline/invoice-api@sha256:9f1c

spire-server entry create \
  -spiffeID $TD/ns/billing/sa/invoice-api \
  -parentID $AGENT \
  -selector k8s:ns:billing \
  -selector k8s:sa:invoice-api \
  -selector k8s:container-image:$IMG \
  -ttl 3600

Read that as a sentence: on a node whose agent attested itself through the Kubernetes projected token attestor in cluster hl-prod-eu, a process in namespace billing, running under service account invoice-api, from a container whose image digest begins 9f1c, is the invoice service, and may hold a one hour certificate saying so.

Note what the workload had to supply: nothing. It connected to a socket. Everything in the decision was observed rather than presented, which is the entire point.

The chain, drawn out, is the thing worth memorizing:

  hardware or cloud platform
        |  signs a statement about the node
        v
  SPIRE agent on that node   (node attestation)
        |  holds an agent SVID, opens a socket
        v
  local kernel / kubelet     (workload attestation)
        |  reports uid, path, pod, image digest
        v
  SPIRE server matches selectors to an entry
        |  mints a 1-hour X509-SVID
        v
  invoice-api holds a certificate it never stored

Each arrow is a place where the chain can be weakened, and the weakest arrow sets the strength of the whole. An agent attested by a join token pasted into a configuration file is exactly as strong as that file’s permissions, no matter how good the rest is.

Cloud instance identity: the document, the metadata service, and IMDSv2#

Every major cloud gives a virtual machine a way to find out who it is, and to prove it. On Amazon Web Services this is the instance metadata service, reachable from inside the instance at the link-local address 169.254.169.254, or fd00:ec2::254 over IPv6. The address is link-local, meaning it is only meaningful and only routable from within the instance itself.

The instance identity document is a JSON object served at /latest/dynamic/instance-identity/document. It contains, among other fields, accountId, architecture, availabilityZone, imageId, instanceId, instanceType, pendingTime, privateIp, region and version. For Harbourline’s node it looks like this:

{
  "accountId": "123456789012",
  "architecture": "x86_64",
  "availabilityZone": "eu-west-1b",
  "imageId": "ami-0c3f1e9a4d2b7e5f1",
  "instanceId": "i-0ab7c19d4e8f23b5c",
  "instanceType": "m6i.2xlarge",
  "pendingTime": "2026-08-18T21:44:03Z",
  "privateIp": "10.42.7.119",
  "region": "eu-west-1",
  "version": "2017-09-30"
}

The document on its own proves nothing, because it is plain JSON. What makes it an identity credential is that the platform signs it, and the signature is available at three sibling paths: /latest/dynamic/instance-identity/pkcs7 for a PKCS#7 signature, /latest/dynamic/instance-identity/signature for a base64-encoded signature, and /latest/dynamic/instance-identity/rsa2048 for an RSA-2048 signature. A verifier outside the instance obtains the AWS public certificate for the relevant region and checks the signature, for instance with openssl smime -verify -in pkcs7 -inform PEM -certfile certificate -noverify. That is how a SPIRE server, or any other authority, can be satisfied that a machine claiming to be instance i-0ab7c19d4e8f23b5c in eu-west-1 really is one.

The same service also serves temporary credentials for whatever role is attached to the instance, at /latest/meta-data/iam/security-credentials/. That is the convenience which caused the accident.

Until 2019 the metadata service answered any plain HTTP GET from inside the instance. That is version 1, and it has exactly the property of the intercom on the pole: anything that can cause an HTTP request to originate inside the instance can read it, including code that has no idea it is doing so.

IMDSv2, announced on 19 November 2019, makes the service session-oriented. Retrieving anything now takes two requests:

TOKEN=$(curl -X PUT \
  "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

curl -H "X-aws-ec2-metadata-token: $TOKEN" \
  http://169.254.169.254/latest/dynamic/instance-identity/document

Four separate defences are stacked in that small change, and it is worth naming each one against the attack it stops.

The first request is an HTTP PUT. The overwhelming majority of vulnerable request-forgery primitives, misconfigured reverse proxies and open web application firewalls will issue a GET on an attacker-supplied URL and will not issue a PUT. Requiring a PUT removes the entire class in one move.

The token must be carried in a request header on every subsequent call. A request-forgery primitive that can only control a URL cannot set a header, so it cannot use a token even if one existed.

The service refuses to issue a session token to any caller that presents an X-Forwarded-For header. That header is what a reverse proxy adds when it forwards somebody else’s request, so its presence is direct evidence that the caller is proxying for an outsider.

The IP packets carrying the token response are sent with a time-to-live of 1. A packet with a TTL of 1 is discarded by the first router that would forward it, so the token physically cannot leave the instance. This is what stops an instance that has been configured as a NAT device, router or VPN endpoint from handing its credentials to the wider network. The hop limit is configurable from 1 to 64, and container workloads generally need 2 because the packet crosses the container network bridge; AMIs marked with imds-support set to v2.0 launch with IMDSv2 required and a hop limit of 2.

The metadata options themselves are set per instance. HttpTokens set to optional permits both versions; set to required it permits IMDSv2 only. AWS announced on 6 November 2023 that from mid-2024, newly released instance types would default to IMDSv2 only, and account-level defaults can now be set so that every launch in an account requires it. As of August 2026, an estate that still permits IMDSv1 is doing so by choice, and it is straightforward to prevent by policy: a deny on ec2:RunInstances when the condition key ec2:MetadataHttpTokens is not required.

Capital One, 2019: the chain worked link by link#

The reason this section exists is that every link in the chain was a component working exactly as designed.

In July 2019, Capital One discovered that data on its customers had been taken from storage buckets in its Amazon Web Services environment. Paige A. Thompson, a former Amazon Web Services engineer using the handle “erratic”, was arrested in July 2019 after Capital One alerted the FBI. According to the Department of Justice, she had built a tool to scan for misconfigured accounts and used it to download data from more than 30 entities. The intrusion into Capital One affected more than 100 million United States customers; contemporaneous reporting and the appellate record put the Capital One figure at 106 million records across the United States and Canada.

The chain ran as follows. A web application firewall in front of a Capital One application was misconfigured in a way that allowed it to be induced to make a request to a URL of the attacker’s choosing. That firewall was running on an instance inside the environment. The request it was induced to make was to the instance metadata service. Because the environment used version 1 of that service, a plain GET returned data, and the path requested was the one that returns temporary credentials for the role attached to the instance. Those credentials had permission to list and read storage buckets. The attacker used them to list the buckets and download the contents.

Note what is not in that chain. No password was guessed. No software vulnerability in the operating system was exploited. No malware ran. Each component did its documented job: the firewall forwarded a request, the metadata service answered a local GET, the credentials granted the access they had been granted. The compromise was in the composition.

The legal aftermath is unusually long and is worth stating accurately, because it is often summarized wrongly. The Office of the Comptroller of the Currency assessed an 80 million dollar civil money penalty against Capital One in August 2020, citing failures in cloud security risk assessment ahead of the migration. Capital One settled customer litigation for 190 million dollars. Thompson was convicted by a federal jury on 17 June 2022 of wire fraud, five counts of unauthorized access to a protected computer, and damaging a protected computer, and was acquitted of access device fraud and aggravated identity theft. On 4 October 2022 she was sentenced to time served and five years of probation. The government appealed; on 17 March 2025 a divided panel of the Ninth Circuit held the sentence substantively unreasonable and remanded for resentencing. On resentencing in November 2025 the district court again imposed a non-custodial sentence, with time served, five years of supervised release including three years of home confinement, 250 hours of community service, and restitution of 40.7 million dollars.

The single configuration change that would have broken the chain is HttpTokens: required. Not better firewall rules, not smaller bucket permissions, though both would have reduced the damage. The request-forgery primitive could issue a GET; it could not issue a PUT and then set a header.

Kubernetes service accounts and projected tokens#

Inside a Kubernetes cluster the workload identity is the ServiceAccount, a namespaced object. A pod that does not name one is assigned the default service account in its namespace by the ServiceAccount admission controller, which also injects a volume containing a token and mounts it at /var/run/secrets/kubernetes.io/serviceaccount in every container that does not already have something mounted there.

The mechanism behind that token changed decisively in Kubernetes v1.22, when the bound service account token volume and the TokenRequest API both reached stable. Before then, the control plane created a Secret holding a token that never expired and had no audience restriction; anyone who read that Secret from etcd, or from a compromised pod, held a permanent cluster credential. Kubernetes v1.24 stopped auto-generating those Secrets. From v1.29 legacy auto-generated tokens that go unused for a configurable period, defaulting to one year, are marked invalid, and the legacy token cleaner, stable since v1.30, deletes them.

The modern mechanism is a projected volume. The kubelet obtains a token from the TokenRequest API, writes it into the volume, and refreshes it before it expires. The token is bound to the pod object, carries the API server as its audience, and expires after a defined lifespan which defaults to one hour. Deleting the pod expires its tokens. The injected volume looks like this:

- name: kube-api-access-x7q2n
  projected:
    defaultMode: 420
    sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          name: kube-root-ca.crt
          items:
            - key: ca.crt
              path: ca.crt
      - downwardAPI:
          items:
            - fieldRef:
                fieldPath: metadata.namespace
              path: namespace

Three sources, three files. The token, the cluster’s certificate authority bundle so the pod can verify the API server rather than trusting whatever answers, and the namespace name.

The token itself is a JWT whose Kubernetes-specific claims live under a kubernetes.io key. Decoded, Harbourline’s invoice pod holds something with this shape:

{
  "aud": ["https://kubernetes.default.svc.cluster.local"],
  "exp": 1787116020,
  "iat": 1787112413,
  "iss": "https://oidc.hl-prod-eu.harbourline.example",
  "jti": "aed34954-b33a-4142-b1ec-389d6bbb4936",
  "kubernetes.io": {
    "namespace": "billing",
    "node": {
      "name": "ip-10-42-7-119",
      "uid": "646e7c5e-32d6-4d42-9dbd-e504e6cbe6b1"
    },
    "pod": {
      "name": "invoice-api-7d4f9c6b8-x2vlq",
      "uid": "5e0bd49b-f040-43b0-99b7-22765a53f7f3"
    },
    "serviceaccount": {
      "name": "invoice-api",
      "uid": "14ee3fa4-a7e2-420f-9f9a-dbc4507c3798"
    }
  },
  "nbf": 1787112413,
  "sub": "system:serviceaccount:billing:invoice-api"
}

Read the binding out of that. The sub claim names the service account in the standard system:serviceaccount:NAMESPACE:NAME form. The private claims name the exact pod and its unique identifier, so the token is useless once that pod is gone. Since Kubernetes v1.32 the node name and node identifier are embedded as well, though the API server does not verify the node claim; it is there so that a consumer can see the placement without fetching more objects. Node-bound tokens, where deleting the Node genuinely revokes the token, became generally available in v1.33.

Two consequences follow, and both catch people out. First, an offline validator that checks the signature and the expiry is not checking whether the bound pod still exists; a token for a deleted pod will still verify offline until it expires. Only the TokenReview API expands and checks the bound claims. Second, there is no revocation list for these tokens. To invalidate every token for a service account you must delete and recreate the service account, which changes its uid and breaks the binding on all of them.

The other half of the design is that the API server publishes an OpenID Connect discovery document and a JSON Web Key Set for its service account issuer. That is what turns an internal Kubernetes token into something an external party can verify, and it is the hinge on which the next section turns.

Workload identity federation: replacing static cloud keys#

The historical way to let a pod call a cloud API was to create a user in that cloud, generate a long-lived access key, put it in a Secret, and mount it. That is the pass in the glovebox, with the additional charm that it is now also in your version control system, your Helm values file and your team’s password manager.

Workload identity federation replaces it. The pod already holds a short-lived, signed, audience-restricted token proving what it is. The cloud already knows how to validate an OpenID Connect token against a published key set. So the pod presents its Kubernetes token to the cloud’s security token service, the cloud validates the signature against the cluster’s published keys, checks the claims against a trust policy, and returns short-lived cloud credentials. No static key exists anywhere in the system.

On AWS the pattern is IAM roles for service accounts. Amazon EKS hosts a public OpenID Connect discovery endpoint for each cluster, containing the signing keys for projected tokens; the public keys rotate every seven days. You register that issuer as an IAM identity provider, annotate the service account with the role to assume, and the mutating webhook injects a second projected token with the audience sts.amazonaws.com at /var/run/secrets/eks.amazonaws.com/serviceaccount/token, along with the environment variables AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE. Every AWS SDK’s default credential chain knows to read those two variables and call AssumeRoleWithWebIdentity.

The service account carries one line:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: invoice-api
  namespace: billing
  annotations:
    eks.amazonaws.com/role-arn: >-
      arn:aws:iam::123456789012:role/hl-invoice-writer

And the role’s trust policy carries the condition that does the real work:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/ISS"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "ISS:aud": "sts.amazonaws.com",
        "ISS:sub":
          "system:serviceaccount:billing:invoice-api"
      }
    }
  }]
}

In a real policy ISS stands for the cluster’s OpenID Connect issuer host, which EKS publishes per cluster and which is long enough that it is shortened here to fit the page.

The sub condition is not optional decoration. Omit it, or write it with a wildcard, and any service account in the cluster can assume the role, which converts a per-service credential back into a cluster-wide one. This is the single most common misconfiguration in the pattern, and it is invisible until somebody looks.

Amazon later introduced EKS Pod Identity as a simpler alternative that does not require an OIDC provider per cluster. A DaemonSet called the Pod Identity Agent runs on every node, listening on the link-local address 169.254.170.23, and the association between a service account and a role is configured in EKS rather than in IAM. The trust policy becomes a single universal principal, pods.eks.amazonaws.com, and credentials are assumed once per node rather than once per pod, which scales better. It is limited to Linux EC2 nodes, and to 5,000 associations per cluster.

On Google Cloud the equivalent is a workload identity pool with an OIDC provider, and this is the worked configuration Harbourline uses for its analytics table. Three commands and one file:

gcloud iam workload-identity-pools create harbourline-k8s \
  --location="global" \
  --display-name="Harbourline clusters"

gcloud iam workload-identity-pools providers create-oidc \
  hl-prod-eu \
  --location="global" \
  --workload-identity-pool="harbourline-k8s" \
  --issuer-uri="ISSUER" \
  --attribute-mapping="\
google.subject=assertion.sub,\
attribute.namespace=assertion['kubernetes.io']['namespace'],\
attribute.sa=assertion['kubernetes.io']['serviceaccount']['name']" \
  --attribute-condition="\
assertion['kubernetes.io']['namespace'] in ['billing']"

POOL=projects/482915602371/locations/global
POOL=$POOL/workloadIdentityPools/harbourline-k8s
SA=invoice-writer@hl-analytics.iam.gserviceaccount.com

gcloud iam workload-identity-pools create-cred-config \
  $POOL/providers/hl-prod-eu \
  --service-account="$SA" \
  --credential-source-file=/var/run/service-account/token \
  --credential-source-type=text \
  --output-file=credential-configuration.json

The attribute mapping is the interesting part. google.subject=assertion.sub says that Google’s notion of the principal is the Kubernetes sub claim, which for our pod is system:serviceaccount:billing:invoice-api. The other two mappings lift the namespace and service account name out of the Kubernetes private claims into attributes that can be used in conditions and in policy bindings. The attribute condition is evaluated at token exchange time and rejects anything from outside the billing namespace before a token is ever minted, which is a cheaper and blunter control than an IAM binding and should be used as well as one, not instead.

The resulting file that ships in the container is a credential configuration, not a credential:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/482915602371
      /locations/global/workloadIdentityPools
      /harbourline-k8s/providers/hl-prod-eu",
  "subject_token_type":
      "urn:ietf:params:oauth:token-type:jwt",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": {
    "file": "/var/run/service-account/token",
    "format": { "type": "text" }
  },
  "service_account_impersonation_url":
      "https://iamcredentials.googleapis.com/v1/projects/-
      /serviceAccounts/invoice-writer@hl-analytics
      .iam.gserviceaccount.com:generateAccessToken"
}

Two of those values are shown wrapped across lines so that they fit the page. In the real file each is a single unbroken string with no spaces in it.

Every field in it is public. If it leaks, nothing happens, because the only thing that turns it into access is a live, unexpired, audience-matched Kubernetes token that only the kubelet can produce and only for the right pod. That is the whole prize: the artefact you ship is no longer a secret.

The identity may be granted directly to the federated principal without any intermediate service account, using the principal form

principal://iam.googleapis.com/projects/482915602371/
  locations/global/workloadIdentityPools/harbourline-k8s/
  subject/system:serviceaccount:billing:invoice-api

which removes even the impersonation step.

The same pattern federates continuous integration. GitHub Actions issues an OpenID Connect token from the issuer token.actions.githubusercontent.com to any workflow that declares permissions: id-token: write, retrievable through the environment variables ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN. Its sub claim historically took the forms repo:ORG/REPO:ref:refs/heads/BRANCH, repo:ORG/REPO:environment:NAME, repo:ORG/REPO:pull_request and repo:ORG/REPO:ref:refs/tags/TAG, and it carries claims including repository, repository_id, repository_owner, actor, workflow_ref, job_workflow_ref, run_id and event_name.

That list of forms is exactly why trust policies must be written against the full subject and not a prefix. A policy that trusts repo:harbourline/* trusts every branch of every repository in the organization, including a branch a contractor opens on a fork. A policy that trusts repo:harbourline/billing:environment:production trusts one deployment environment, which is what was meant.

One recent change matters for anyone writing these policies now. GitHub reports that repositories created after 15 July 2026 use an immutable default subject format that embeds numeric identifiers, of the form repo:OWNER@OWNER-ID/REPO@REPO-ID:ref:refs/heads/BRANCH. The reason is name recycling: if an organization is deleted and somebody else registers the same name, the old string-based subject would match the new owner. Embedding the identifier removes that. Policies written against the old format will not match repositories created after that date, which is a migration to plan rather than discover.

Hardware roots of trust: TPM, quotes, and remote attestation#

Everything above eventually rests on something. In a public cloud it rests on the provider’s signature over the instance identity document. On your own hardware it rests on a chip.

A Trusted Platform Module is a small, tamper-resistant device on the board, or a firmware equivalent inside the processor. The current specification is the TCG Trusted Platform Module 2.0 Library Specification, Version 185, published in March 2026. Its lineage is unusually easy to date because the Trusted Computing Group publishes the whole list.

The 2.0 library was also standardized internationally as ISO/IEC 11889:2015, in four parts: Architecture, Structures, Commands and Supporting Routines. The current TCG publication is in four documents titled Introduction, Architecture, Structures and Commands.

Three TPM features do the work.

Platform Configuration Registers, or PCRs, are registers that cannot be written, only extended. The extend operation replaces the register’s contents with the hash of the old contents concatenated with a new measurement. Because hashing is one-way and the operation is chained, the final value depends on every measurement and on their order; the specification is explicit that extending A then B gives a different result from B then A. You cannot roll a PCR back to a value you liked better without resetting the machine.

The PC Client platform profile assigns the low registers to firmware. PCR 0 holds platform firmware code and the core root of trust for measurement, PCR 1 holds platform configuration and settings, PCR 2 holds option ROM code, PCR 3 holds option ROM configuration, PCR 4 holds the boot manager and OS loader code, PCR 5 holds their configuration, PCR 6 holds manufacturer-specific measurements, and PCR 7 holds Secure Boot policy. The profile states that PCRs 0 to 7 are exclusively for platform firmware and that operating system software should never use them; the static OS set begins at PCR 8.

The endorsement key is a key pair created inside the TPM at manufacture, whose private half never leaves the chip, accompanied by an endorsement certificate signed by the manufacturer. It is the chip’s birth certificate. Because using it directly for signing would make every operation linkable to one physical machine, in practice an attestation key is created and certified against the endorsement key, and the attestation key does the signing.

Remote attestation is then a challenge and response. A verifier sends a fresh random nonce. The machine asks its TPM to produce a quote: a structure containing the selected PCR values, the nonce, and a signature over both made with the attestation key. The verifier checks the signature, checks that the nonce is the one it just sent, and compares the PCR digest against a set of values it considers acceptable. Alongside the quote the machine usually supplies an event log, so the verifier can replay the individual measurements and see which one differs rather than merely learning that the total does not match.

In command terms, with the standard tooling:

tpm2_createek  -c ek.ctx -G rsa -u ek.pub
tpm2_createak  -C ek.ctx -c ak.ctx -u ak.pub -n ak.name
tpm2_quote     -c ak.ctx -l sha256:0,1,2,3,4,5,6,7 \
               -q $NONCE -m quote.msg -s quote.sig \
               -o pcrs.out

The verifier then checks quote.sig over quote.msg using ak.pub, confirms the nonce inside quote.msg, and compares pcrs.out against expected values.

Secure enclaves attack a different problem. A TPM measures the platform; an enclave removes the platform from the trust boundary. Intel SGX, AMD SEV-SNP and Arm Confidential Compute Architecture each create a region of memory that the processor encrypts and integrity-protects, such that the hypervisor and the host operating system can see only ciphertext. Each provides its own attestation: the processor signs a report describing what was loaded into the enclave, and a remote party verifies that report against the vendor’s certificate chain before releasing any secret to it. This is what “confidential computing” names, and its honest description is a narrower trust boundary, not an absence of trust; you still trust the silicon vendor and the microcode, and there is an active and productive research literature on side-channel attacks against every one of these designs.

The practical link back to the rest of the chapter is short. A TPM-based node attestor lets a SPIRE server verify that an agent is running on a specific, known, correctly booted physical machine before it will parent any workload identity to it. That closes the loop: the workload’s certificate rests on the agent’s identity, the agent’s identity rests on the machine’s boot state, and the machine’s boot state rests on a key in silicon that cannot be copied.

The 03:07 invoice, traced end to end#

Now put the whole thing together on Harbourline’s invoice.

At 03:07:11 the invoice-api process finishes computing INV-2026-08-14771. It holds no secrets. Its container image contains no credentials. Its environment contains three file paths and two public identifiers.

It opens a connection to the local Workload API socket named by SPIFFE_ENDPOINT_SOCKET and calls FetchX509SVID. The agent looks at the far end of the socket, asks the kernel for the calling process’s user, path and container, asks the kubelet for the pod, namespace and service account, and finds selectors matching the registration entry. It returns a certificate whose single URI SAN is spiffe://harbourline.example/ns/billing/sa/invoice-api, valid for one hour, together with the trust bundle.

It dials ledger over TLS. ledger requests a client certificate; invoice-api presents its SVID and signs the handshake transcript. invoice-api in turn checks that ledger’s certificate carries the URI SAN spiffe://harbourline.example/ns/finance/sa/ledger and chains to the same trust bundle. Both ends now know a verified name for the other. What invoice-api is then allowed to read from ledger is an authorization question, and chapter 46 is where it belongs.

It reads the file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token, which the kubelet refreshed forty minutes ago, and its AWS SDK calls AssumeRoleWithWebIdentity. The security token service validates the signature against the cluster’s published key set, checks that aud is sts.amazonaws.com and that sub is system:serviceaccount:billing:invoice-api, and returns session credentials with an access key identifier beginning ASIA. The PDF is written to harbourline-invoices-eu. The credentials expire within the hour.

It reads the same style of token from the second projected volume, hands it to the Google security token service using the credential configuration, receives an access token for the impersonated analytics service account, and appends the row.

For the payment provider, which knows nothing about any of this, it falls back to the old world: a bearer API key held in a secrets manager, fetched at startup using the AWS session credentials it just derived, held only in memory, never written to disk, and rotated by the provider’s rotation Lambda every thirty days.

03:07:11  invoice-api asks local agent for identity
03:07:11  agent inspects caller; matches selectors
03:07:11  SPIRE issues X509-SVID, expires 04:07:11
03:07:12  mTLS to ledger; both ends verify SAN
03:07:13  read projected token (aud sts.amazonaws.com)
03:07:13  AssumeRoleWithWebIdentity -> ASIA... session
03:07:14  PUT invoice PDF to harbourline-invoices-eu
03:07:15  read projected token (aud //iam.googleapis...)
03:07:15  STS token exchange -> Google access token
03:07:16  append analytics row
03:07:17  fetch payment API key from secrets manager
03:07:18  POST collection request; key stays in memory

Count the long-lived secrets in that sequence. One: the payment provider’s API key, which exists because the payment provider does not support federation, and which is held in a managed store rather than in a file. Everything else was minted for this hour and will not exist by breakfast.

Count the places a compromise of the pod gets an attacker. The ledger service, at exactly the permissions invoice-api has. The one storage bucket. The one analytics table. The payment key, which is the real prize and the reason the remaining work is to get the payment provider onto federation.

That is what good looks like as of August 2026. It is not exotic. Every component named here is generally available and documented, and the reason most estates do not look like this is not technical difficulty; it is that nobody has ever been given the job of counting the machine identities.

47.98 Common wrong ideas#

Wrong: Machine credentials matter less than human ones because there is no person to phish. Right: They matter more, because there is nobody to notice. A phished human often reports a strange login within hours; a stolen service credential is used by a program that behaves exactly like the program it replaced, and the median time to discovery is measured in months. The absence of a person removes the best detector you had, it does not remove the risk.

Wrong: Rotating a leaked key fixes the leak. Right: Rotation stops future use of that credential and nothing else. Everything the holder did while it was valid still stands, including any second credential, trusted key or access grant they created for themselves. A rotation without an investigation of what was done during the exposure window is a change of locks after the burglar has already photographed the safe combination.

Wrong: A private repository is a safe place for a credential. Right: Private controls who may read the repository today, not who may read it after any single account is compromised, and it does nothing about the credential’s own lifetime or scope. Uber’s 2016 breach began with an AWS key in a private repository. Internal repositories are, by GitGuardian’s March 2026 figures, about six times more likely than public ones to contain a hardcoded secret, precisely because people relax when nobody outside can see.

Wrong: Mutual TLS means the connection is secure, so the request is authorized. Right: Mutual TLS establishes a verified name for each end of the connection and nothing more. A compromised service presents an entirely valid certificate and every check passes. Authentication of the peer is the floor; deciding what that peer may ask for is a separate mechanism, covered in chapter 46 on authorization.

Wrong: The instance metadata service is safe because it is only reachable from inside the instance. Right: “Inside the instance” includes every piece of software on it that will fetch a URL on somebody else’s behalf, which is what a proxy, a web application firewall, a link previewer or a vulnerable importer does for a living. That is exactly the composition that produced the 2019 Capital One intrusion. Requiring IMDSv2, which needs a PUT and then a header, removes the class rather than mitigating it.

Wrong: A TPM proves the machine is trustworthy. Right: A TPM proves what was measured into its registers during boot, signed by a key that cannot be extracted. That is a strong statement about the launch state and a weak implication about the present. Software compromised after boot produces identical measurements. Attestation must be repeated on a schedule and combined with runtime evidence if you want a claim about now rather than about start-up.

Wrong: SPIFFE and similar systems eliminate secrets. Right: They eliminate secrets from the workload, which is the population you cannot govern, and concentrate the remaining trust in one attested root you can watch. There is always a bottom to the chain: a cloud provider’s signing key, a TPM endorsement key, or a join token somebody typed. A design that claims no root has a root it has not shown you.

Wrong: Short-lived credentials are strictly better, so make them as short as possible. Right: Every reduction in lifetime moves risk from confidentiality to availability. One hour certificates make your certificate authority a load-bearing hourly dependency of everything you run, and an outage there arrives everywhere simultaneously. Choose a lifetime short enough that the exposure window is acceptable, then make the issuance path more available than anything that depends on it.

Wrong: A federation trust policy that names the right repository or cluster is enough. Right: It must name the exact subject. An AWS trust policy without a sub condition lets any service account in the cluster assume the role. A GitHub Actions policy matching repo:org/repo:* accepts any branch, including one pushed to a fork by anybody with write access. The condition on the subject claim is the control; the rest is plumbing.

Wrong: Kubernetes service account tokens can be revoked. Right: There is no revocation list for tokens issued through the TokenRequest API. A pod-bound token is invalidated by deleting the pod, and every token for a service account is invalidated by deleting and recreating the service account, because that changes its uid. An offline validator checking only the signature and expiry will happily accept a token whose pod no longer exists; only the TokenReview API expands and checks the bound claims.

47.99 Chapter summary in 20 lines#

1. The overwhelming majority of authentications happening at any moment are one program proving itself to another, with no human present at either end.
2. Survey figures published by Palo Alto Networks in May 2026 put machine identities at 109 for every human identity, up from 82 to one the year before, and should be read as a vendor survey rather than a census.
3. A workload identity is a name plus a credential, and the engineering that matters is the binding: what mechanism makes that credential obtainable only by the intended software.
4. Bindings range from possession of a secret, which binds nothing, through platform and orchestrator assertions and kernel introspection, to hardware measurement, which is the only one an attacker cannot satisfy by copying a file.
5. Long-lived API keys leak by at least seven routes, and GitGuardian’s March 2026 report counted 28.65 million new hardcoded secrets in public GitHub commits during 2025 alone.
6. Detection has improved far faster than remediation: of credentials confirmed valid in 2022, more than 64 per cent were still valid when retested in January 2026.
7. The arithmetic of rotation defeats policy at scale, because a thousand credentials on a ninety-day cycle requires eleven careful rotations every day forever.
8. Mutual TLS gives each end a verified name for the other and cannot be replayed, because the proof is a fresh signature over the handshake transcript rather than a reusable secret.
9. Public certificate lifetimes are falling on a fixed schedule set by CA/Browser Forum ballot SC-081v3: 200 days from 15 March 2026, 100 days from 15 March 2027, and 47 days from 15 March 2029.
10. A SPIFFE ID is a URI of the form spiffe://trust-domain/path, where the trust domain is the root of trust and the path names the workload.
11. An X509-SVID carries that identity as exactly one URI entry in the Subject Alternative Name extension, and a JWT-SVID carries it in the sub claim with a mandatory audience and expiry.
12. The SPIFFE Workload API deliberately requires no authentication from the workload, because the endpoint identifies the caller by inspecting it rather than by asking it for a password.
13. SPIRE separates node attestation, which establishes what machine the agent runs on, from workload attestation, which establishes what process is calling the socket right now.
14. A cloud instance identity document is only a credential because the platform signs it, and the signature is retrievable in PKCS#7, base64 and RSA-2048 forms alongside the document itself.
15. IMDSv2, announced on 19 November 2019, requires a PUT to obtain a session token, requires that token in a header, refuses callers presenting X-Forwarded-For, and sets the packet time-to-live to 1 so the token cannot leave the instance.
16. The 2019 Capital One intrusion chained a request-forgery primitive to a version 1 metadata service to role credentials to storage buckets, and every component behaved exactly as documented.
17. Kubernetes projected service account tokens, stable since v1.22, are bound to a pod, carry an explicit audience, expire in about an hour and are refreshed by the kubelet without restarting anything.
18. Workload identity federation exchanges that Kubernetes token for cloud credentials, so that the artefact shipped in the container is a public configuration file rather than a secret.
19. Federation is only as good as the subject condition in the trust policy, and a missing or wildcarded subject silently converts a per-service credential into a cluster-wide one.
20. A TPM quote binds selected boot measurements and a fresh nonce under a key that never leaves the chip, which proves how the machine started and says nothing about what it has done since.

Chapter sources: SPIFFE specifications, namely the SPIFFE ID specification sections 2, 2.1, 2.2 and 2.3, the X.509 SVID specification sections 2, 4.1 and 4.3, the JWT SVID specification sections 2.1, 3.1, 3.2 and 3.3, the SPIFFE Workload API specification and the SPIFFE Workload Endpoint specification; Cloud Native Computing Foundation announcement of SPIFFE and SPIRE graduation, 20 September 2022; SPIRE server configuration reference for default_x509_svid_ttl, default_jwt_svid_ttl and ca_ttl defaults; RFC 3986 for URI syntax and RFC 7518 for JSON Web Algorithms; Amazon Web Services EC2 User Guide pages on instance metadata retrieval, instance metadata options, instance identity documents and their verification; the AWS Security Blog post “Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the EC2 Instance Metadata Service”, 19 November 2019; the AWS News Blog post “Amazon EC2 Instance Metadata Service IMDSv2 by default”, 6 November 2023; United States Department of Justice, Western District of Washington press releases of 17 June 2022 and 4 October 2022 in the prosecution of Paige A. Thompson, together with the Ninth Circuit opinion in case 22-30179 of 17 March 2025 and the November 2025 resentencing; Office of the Comptroller of the Currency civil money penalty against Capital One, August 2020; United States Department of Justice, Northern District of California press release of 5 October 2022 on the conviction of Joseph Sullivan; Codecov security update of 15 April 2021 and Rapid7 analysis of 16 April 2021 on the Bash Uploader compromise of 31 January to 1 April 2021; Kubernetes documentation on managing service accounts, bound service account tokens, the TokenRequest API and projected volumes, covering the v1.22, v1.24, v1.28, v1.29, v1.30, v1.32, v1.33 and v1.36 feature states; Amazon EKS User Guide on IAM roles for service accounts and on EKS Pod Identity; Google Cloud IAM documentation on workload identity federation with Kubernetes; GitHub Actions documentation on security hardening with OpenID Connect, including the immutable subject format for repositories created after 15 July 2026; CA/Browser Forum ballot SC-081v3, voting period closed 11 April 2025, and the TLS Baseline Requirements relevant dates for 15 March 2026, 15 March 2027 and 15 March 2029; Trusted Computing Group TPM 2.0 Library Specification Version 185 of March 2026 with its published revision history back to Revision 00.96 of March 2013, and ISO/IEC 11889:2015 Parts 1 to 4; TCG PC Client Platform Firmware Profile specification for PCR 0 to 7 assignments and the extend operation; GitGuardian State of Secrets Sprawl 2026, published 17 March 2026; Palo Alto Networks 2026 Identity Security Landscape report, published May 2026.