TL;DR
- AWS Nitro Sandboxes provisions lightweight Firecracker microVMs with dedicated guest kernels for opted-in Kubernetes pods on Amazon EKS.
- Enabling microVM isolation requires adding
runtimeClassName: isolated-sandboxorruntimeClassName: confidential-sandboxto the pod specification across all commercial AWS regions at no additional feature charge. - Isolated mode protects the parent host while retaining operational capabilities like
kubectl exec; confidential mode treats the host as untrusted, enforcing Cedar security policies and cryptographic PCR attestation via the Nitro Secure Module. - Linux core scheduling prevents sandboxes from sharing physical CPU cores on hyperthreaded x86 nodes, cutting usable vCPU capacity by roughly half, whereas single-threaded AWS Graviton nodes are unaffected.
- Firecracker lacks host device passthrough capabilities at launch, excluding GPU-accelerated workloads from Nitro Sandbox isolation.
Container isolation in Kubernetes traditionally relies on Linux kernel constructs such as namespaces, control groups (cgroups), and Linux Security Modules (LSMs). While effective for application workloads sharing an enterprise trust domain, this model shares a single host operating system kernel across all containers running on a node. When executing untrusted code—such as multi-tenant software-as-a-service (SaaS) execution, continuous integration (CI) build tasks, or autonomous artificial intelligence (AI) coding agents—a single zero-day kernel privilege escalation vulnerability can compromise the underlying host and adjacent pods sharing the node.
To eliminate this shared-kernel attack surface, AWS launched aws nitro sandboxes, a feature designed to run targeted workloads inside hardware-assisted microVMs with separate guest kernels carved out directly by the AWS Nitro hypervisor. On Amazon Elastic Kubernetes Service (EKS), platform teams opt into this boundary by specifying runtimeClassName: isolated-sandbox or runtimeClassName: confidential-sandbox in the pod spec. The feature is available across all commercial AWS regions at no extra charge, bringing microVM isolation directly to EKS workloads without requiring manual hypervisor orchestration.
By enforcing hardware virtualisation boundaries at the pod level, platform engineering teams can run untrusted or multi-tenant workloads without exposing the parent host to OS-level compromise. However, moving from shared-kernel containerisation to hardware-assisted isolation introduces operational prerequisites, core scheduling constraints, and mandatory changes to credential management systems like IAM Roles for Service Accounts (IRSA).
Hardware-Assisted Pod Isolation on Amazon EKS
Standard Linux container isolation creates logical boundaries within a shared kernel instance. If an attacker triggers a privilege escalation vulnerability in the host kernel, every container on that host is compromised. This shared-kernel risk is particularly acute when running autonomous AI coding agents, remote IDE environments, or multi-tenant SaaS code execution engines, where arbitrary code runs on demand.
According to InfoQ, AWS Nitro Sandboxes shifts the security boundary from software-level namespace isolation to hardware-assisted virtualisation. Each opted-in pod boots its own dedicated guest operating system kernel carved out of the parent EC2 instance by the AWS Nitro hypervisor. The containers inside the pod run inside this lightweight Firecracker microVM, ensuring that neither memory mappings nor kernel state are shared with the host or adjacent sandboxed pods.
AWS provides two distinct RuntimeClasses to select the appropriate security boundary:
isolated-sandbox: Designed for workloads where the tenant or executed code is untrusted, but the parent EC2 host and underlying infrastructure are trusted.confidential-sandbox: Designed for environments where the parent EC2 host and host infrastructure are treated as untrusted, enforcing strict memory encryption and cryptographic hardware attestation.
The feature is available across all commercial AWS regions without additional AWS feature charges; standard EC2 instance compute fees remain unchanged. However, transitioning workloads into microVM sandboxes requires a clear understanding of the underlying shim process, communication boundaries, and node provisioning requirements.
Architecture and Runtime Execution Flow
When an application pod with a sandbox RuntimeClass is scheduled onto an EKS worker node, the standard host container runtime delegates execution to a specialized shim named containerd-shim-nitro-sandbox-v2. This shim communicates with the AWS Nitro hypervisor to instantiate a Firecracker microVM as a first-level guest on the host EC2 instance.
Inside the newly booted microVM, a lightweight guest agent initializes an internal, dedicated containerd instance. This internal runtime manages the container lifecycle, pulling image layers and launching application processes inside the isolated guest environment. Communication between the host parent instance and the guest microVM is strictly restricted to low-level virtio and vsock virtual interfaces.
+-------------------------------------------------------------------+
| EC2 Host Instance (Parent Host / AWS Nitro Hypervisor) |
| |
| +-------------------------------------------------------------+ |
| | Host Containerd Daemon | |
| +------------------------------+------------------------------+ |
| | |
| v |
| +----------------------------------+ |
| | containerd-shim-nitro-sandbox-v2 | |
| +----------------+-----------------+ |
| | |
| virtio / vsock |
| | |
| +----------------------------v--------------------------------+ |
| | Firecracker MicroVM (First-Level Guest) | |
| | | |
| | +-------------------------------------------------------+ | |
| | | Guest Linux Kernel | | |
| | +---------------------------+---------------------------+ | |
| | | | |
| | v | |
| | +-------------------------------------------------------+ | |
| | | Internal Guest Agent & Guest Containerd | | |
| | +---------------------------+---------------------------+ | |
| | | | |
| | v | |
| | +-------------------------------------------------------+ | |
| | | Pod Application Containers | | |
| | +-------------------------------------------------------+ | |
| +-------------------------------------------------------------+ |
+-------------------------------------------------------------------+
Node provisioning models differ between EKS execution modes:
- EKS Auto Mode: Pre-configures both
isolated-sandboxandconfidential-sandboxRuntimeClasses out of the box. When a sandboxed pod is created, EKS Auto Mode automatically provisions sandbox-capable EC2 worker nodes on demand. - Standard EKS: Requires platform engineers to build custom Amazon Machine Images (AMIs) containing the necessary Firecracker and containerd shim binaries. Node groups or Karpenter NodePools must explicitly reference these custom AMIs and underlying launch templates.
Isolated vs Confidential Mode Security Boundaries
The choice between isolated-sandbox and confidential-sandbox depends on the operational trust model and required administrative visibility. Each mode enforces a fundamentally different threat model.
Isolated Mode (`isolated-sandbox`)
Isolated mode protects the parent host from untrusted workloads running inside the guest microVM. Because the parent instance is treated as trusted, standard Kubernetes administrative utilities remain fully functional. Platform engineers can execute kubectl exec, stream logs via kubectl logs, and inspect active processes on the host. However, as AWS Container Hero Sandip Das notes, protection in isolated mode operates in one direction. A compromised parent host can inspect or modify unencrypted network traffic, alter the guest pod’s /etc/hosts configuration, or mint Pod Identity credentials for any pod running on that node.
Confidential Mode (`confidential-sandbox`)
Confidential mode treats the parent host as potentially compromised or untrusted. When a pod is submitted under confidential-sandbox, an automated Cedar security policy is generated directly from the pod specification. This policy pins container image digests, command-line arguments, environment variables, directory mounts, and Linux capabilities. Interactive features such as kubectl exec, kubectl attach, and interactive standard input (stdin) are explicitly blocked.
During microVM boot, the Nitro Secure Module (NSM) measures the container image layers and the synthesized Cedar policy into Platform Configuration Registers (PCRs). In addition, image layers can be encrypted on disk, allowing the untrusted parent host to transfer ciphertext without accessing unencrypted content.
Critical Attestation Requirement
In confidential mode, the Nitro Secure Module records PCR attestation measurements. However, according to the AWS EKS documentation, unless an AWS Key Management Service (KMS) key policy or external attestation verifier explicitly validates these PCR values prior to releasing secrets, the attestation measurement remains unverified, rendering confidentiality guarantees incomplete.
Network Isolation Annotation
Operators can apply a network-isolation annotation to completely detach the sandbox from virtual network interfaces. This establishes a strict offline execution environment for sensitive computation.
Impact of Network Isolation
Applying the network-isolated annotation strips all virtual network devices from the microVM guest. This disables DNS resolution, NTP time synchronization, EC2 Instance Metadata Service (IMDS) access, EKS Pod Identity credential retrieval, and HTTP/TCP liveness or readiness probes.
| Feature / Attribute | Isolated Mode (isolated-sandbox) |
Confidential Mode (confidential-sandbox) |
|---|---|---|
| Host Trust Assumption | Parent host is trusted; guest microVM is untrusted. | Parent host is untrusted; guest microVM is isolated. |
| Kubectl Exec / Attach | Supported. | Blocked by Cedar policy. |
| Cedar Policy Generation | Not generated. | Automated from pod spec (pins digests, args, env, mounts). |
| Hardware Attestation | None. | Nitro Secure Module (NSM) measures policy and images into PCRs. |
| Image Layer Encryption | Standard unencrypted layers. | Encrypted layers supported; host handles ciphertext. |
| Secret Protection | Standard Kubernetes secrets / Pod Identity. | KMS key policies release secrets based on PCR attestation. |
Density Constraints, Core Scheduling, and Resource Overhead
Running containers inside dedicated microVMs changes compute density calculations and hypervisor scheduling dynamics compared to standard Kubernetes nodes.
Density Caps and Scaling Limits
AWS enforces two hard caps on sandbox density per worker node:
- vCPU Capacity Cap: The combined vCPU allocations of all active sandboxes on a node cannot exceed 5x the host node’s physical vCPU count.
- Total Sandbox Count Cap: The maximum number of concurrently running sandboxes on a node is capped at 5x the node’s vCPU count.
On EC2 instance types that utilize CPU overcommit, these scaling caps are halved to 2.5x the node vCPU count. For example, a standard non-overcommitted 4-vCPU EC2 node can host a maximum of twenty 1-vCPU sandboxes or five 4-vCPU sandboxes.
Linux Core Scheduling Penalties on x86 vs Graviton
To prevent microVM side-channel exploits across hardware threads, AWS enforces Linux core scheduling on hyperthreaded x86 EC2 instances. Core scheduling prevents two distinct sandboxes from executing concurrently on the sibling hyperthreads of a single physical CPU core. As a result, approximately 50% of advertised vCPUs on hyperthreaded x86 nodes become unavailable for sandbox scheduling.
Architecture Recommendation: AWS Graviton
AWS Graviton processors utilize single-threaded physical CPU cores. Because there are no sibling hyperthreads, Graviton nodes incur zero core scheduling capacity penalties, preserving 100% of the allocated vCPUs for sandbox execution.
Hardware and GPU Restrictions
Because Firecracker does not support passing host PCI devices directly into guest microVMs, GPU passthrough and generic host device passthrough are unsupported at launch. Applications requiring CUDA or specialized hardware acceleration cannot run within AWS Nitro Sandboxes.
Mitigating Boot Latency with Warm Pools
Instantiating a Firecracker microVM and booting a guest Linux kernel adds cold-start latency compared to launching standard container namespaces. To reduce startup delays for ephemeral tasks, platform teams can utilize the open-source agent-sandbox project. By maintaining warm pools of pre-booted microVMs, agent-sandbox delivers ready-to-execute environments in approximately two seconds.
Note: Exact I/O latency overhead and networking throughput benchmarks comparing Nitro Sandboxes to standard containerd runtimes have not been disclosed by AWS and should be evaluated via production-like benchmarking.
Operational Prerequisites and Node Configuration
Deploying AWS Nitro Sandboxes on EKS requires specific control plane, node template, and identity configurations.
System Prerequisites
- Kubernetes Version: Amazon EKS cluster running Kubernetes 1.36 or later.
- EC2 Launch Templates: Worker node launch templates must explicitly enable Nitro Enclaves support on Managed Node Groups.
- Identity Architecture: Workloads must adopt EKS Pod Identity. Legacy IAM Roles for Service Accounts (IRSA) relying on OIDC web identity tokens are unsupported inside sandboxed microVMs.
Handling Pending Pod Status
If a pod configured with a sandbox RuntimeClass remains in a Pending state, it indicates that no registered nodes meet the sandbox capability criteria. Common root causes include node groups deployed without Nitro Enclaves enabled in their launch templates or standard EKS clusters missing the custom AMI built with the containerd-shim-nitro-sandbox-v2 runtime binaries.
Example Manifest: Sandbox Pod Deployments
The following example demonstrates an isolated-sandbox deployment alongside a confidential-sandbox deployment featuring network isolation:
apiVersion: v1
kind: Pod
metadata:
name: isolated-agent-worker
namespace: tenant-workloads
spec:
runtimeClassName: isolated-sandbox
containers:
- name: code-runner
image: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/agent-runner:v1.4.2
resources:
limits:
cpu: "2"
memory: "4Gi"
requests:
cpu: "1"
memory: "2Gi"
---
apiVersion: v1
kind: Pod
metadata:
name: confidential-secure-builder
namespace: tenant-workloads
annotations:
nitro-sandbox.amazonaws.com/network-isolated: "true"
spec:
runtimeClassName: confidential-sandbox
containers:
- name: sensitive-build-task
image: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/secure-builder@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
resources:
limits:
cpu: "4"
memory: "8Gi"
requests:
cpu: "2"
memory: "4Gi"
Architectural Comparison: EKS Sandboxes vs GKE Sandbox
Google Cloud Platform (GCP) and AWS address microVM pod isolation through distinct runtime architectures. GKE Sandbox offers two runtime classes: a user-space system call interception architecture powered by gVisor, and a microVM architecture using Kata Containers with Cloud Hypervisor.
gVisor intercepts application system calls in user space using a dedicated kernel emulator written in Go. This allows cold starts under 200 milliseconds and supports CUDA passthrough on select GPUs. However, because gVisor emulates system calls rather than running an authentic Linux kernel, strict kernel compatibility and performance parity vary depending on syscall patterns.
Conversely, microVM isolation—used by AWS Nitro Sandboxes and GKE’s Kata/Cloud Hypervisor option—boots a genuine Linux guest kernel. GKE Kata microVMs impose a fixed overhead allocation of 250 mCPU and 130 MiB of RAM per pod, with start times between one and two seconds. AWS Nitro Sandboxes manages hardware limits using node vCPU ratio caps (5x limits) and Linux core scheduling policies rather than static per-pod mCPU deductions.
| Dimension | AWS Nitro Sandboxes | GKE Sandbox (gVisor) | GKE Sandbox (Kata / Cloud Hypervisor) |
|---|---|---|---|
| Isolation Mechanism | Firecracker MicroVM (Nitro Hypervisor) | User-Space Syscall Interception (gVisor) | Kata Containers + Cloud Hypervisor MicroVM |
| Kernel Architecture | Dedicated Linux Guest Kernel | Emulated User-Space Kernel (Sentry) | Dedicated Linux Guest Kernel |
| Startup Latency | 1–2 seconds (2s with warm pools) | < 200 milliseconds | 1–2 seconds |
| GPU Acceleration | Unsupported (Firecracker device limit) | Supported (CUDA on select GPUs) | Unsupported |
| Resource Overhead Model | 5x vCPU density caps; Core scheduling limits | Minimal user-space process overhead | Fixed per-pod allocation (250 mCPU + 130 MiB) |
Implementation Checklist and Migration Guidance for Platform Teams
Platform teams migrating untrusted workloads to AWS Nitro Sandboxes should follow a structured deployment methodology to ensure security, identity, and capacity requirements are met.
1. Migration Audit Phase
- Audit target workloads for host device, PCI, or GPU dependencies. Move GPU workloads to dedicated non-sandboxed node pools.
- Identify services relying on legacy IRSA (OIDC token volume projections). Upgrade cluster configurations to adopt EKS Pod Identity across all namespaces.
- Review probe definitions. If planning to use the
network-isolatedannotation, replace HTTP/TCP liveness and readiness probes with exec-based file checks or rely on external status monitoring.
2. Node Group Infrastructure Setup
For standard EKS clusters using Karpenter, ensure the NodePool specifications select AMIs that include the containerd-shim-nitro-sandbox-v2 binaries and configure EC2 Launch Templates with Nitro Enclaves enabled:
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
name: nitro-sandbox-nodepool
spec:
template:
spec:
requirements:
- key: kubernetes.io/arch
operator: In
values: ["arm64"] # Prefer Graviton for core scheduling efficiency
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c7g", "m7g", "r7g"]
nodeClassRef:
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
name: sandbox-node-class
---
apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
name: sandbox-node-class
spec:
amiFamily: AL2023
launchTemplate:
nitroEnclavesEnabled: true
role: KarpenterNodeRole-ClusterName
3. KMS Policy Construction for Confidential Mode
To enforce complete confidentiality, configure AWS KMS key policies to validate Nitro Secure Module PCR measurements before releasing secrets or decryption keys:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "EnablePCRBoundSecretDecryption",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789012:role/SandboxExecutionRole"
},
"Action": "kms:Decrypt",
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:RecipientAttestation:PCR0": "1a2b3c4d5e6f...",
"kms:RecipientAttestation:PCR1": "9f8e7d6c5b4a..."
}
}
}
]
}
4. Diagnostics and Cluster Verification
Verify that nodes are registered with sandbox capabilities and monitor pod lifecycle events using the following diagnostic commands:
# Verify RuntimeClasses registered in the cluster
kubectl get runtimeclass
# Inspect node status for Nitro Sandbox capability labels
kubectl get nodes -o custom-columns=NAME:.metadata.name,SANDBOX:.metadata.labels.node\.kubernetes\.io/instance-type
# Describe pending sandboxed pod to diagnose scheduling failures
kubectl describe pod -n
# Verify containerd-shim processes on worker host (requires SSH/SSM host access)
ps aux | grep containerd-shim-nitro-sandbox-v2
Key Takeaways for Platform and Security Engineering
AWS Nitro Sandboxes brings hardware-assisted microVM isolation to Kubernetes workloads on Amazon EKS without requiring custom hypervisor management layers. By leveraging Firecracker microVMs and dedicated guest kernels, platform and security engineering teams can isolate untrusted code, tenant-submitted scripts, and autonomous AI agents.
When planning deployment strategies, platform leaders should prioritize the following operational principles:
- Match Trust Models to Runtime Classes: Use
isolated-sandboxwhen operational visibility (such askubectl execand logging) is required and the parent host is trusted. Adoptconfidential-sandboxwhen treating the host as untrusted, ensuring KMS key policies strictly check NSM PCR attestation values. - Optimize Instance Architecture for Density: Standardize sandbox node pools on single-threaded AWS Graviton instances (such as
c7gorm7gfamilies) to avoid the 50% core scheduling vCPU penalty incurred on hyperthreaded x86 architectures. - Address Hardware Limitations Early: Exclude GPU-accelerated and host-device-dependent workloads from Nitro Sandboxes due to Firecracker passthrough constraints.
- Standardize on EKS Pod Identity: Migrate existing IAM Roles for Service Accounts (IRSA) setups to EKS Pod Identity before executing workloads inside sandboxed environments.
Frequently asked questions
How does AWS Nitro Sandboxes differ from standard container isolation in Kubernetes?
Standard container isolation relies on Linux cgroups, namespaces, and LSMs sharing the underlying parent host operating system kernel. AWS Nitro Sandboxes provisions a dedicated Firecracker microVM with its own isolated guest kernel for each opted-in pod via the AWS Nitro hypervisor, establishing a hardware-assisted boundary that prevents shared-kernel privilege escalation attacks.
Can I run GPU workloads inside an EKS pod using AWS Nitro Sandboxes?
No. GPU and host PCI device passthrough are unsupported at launch because the Firecracker microVM hypervisor cannot pass physical host devices directly into guest microVMs. Pods requiring GPU acceleration must run using standard container runtimes without the sandbox RuntimeClass.
What is the difference between isolated-sandbox and confidential-sandbox modes?
Isolated mode trusts the parent EC2 host and isolates the untrusted guest, retaining operational tools like kubectl exec and streaming logs. Confidential mode treats the parent host as untrusted, generating an automated Cedar security policy to block exec and interactive stdin while measuring images and policies into PCRs via the Nitro Secure Module for KMS cryptographic attestation.
Why does my sandboxed pod stay in a Pending state on standard EKS node groups?
A pod stays in a Pending state when no nodes satisfy the sandbox RuntimeClass requirements. On standard EKS, this occurs if the worker node group launch template lacks Nitro Enclaves enablement, or if the underlying nodes were built without custom AMIs containing the containerd-shim-nitro-sandbox-v2 binaries and guest agent dependencies.
How does core scheduling affect pod density on x86 EC2 instances versus AWS Graviton?
On hyperthreaded x86 instances, Linux core scheduling stops two sandboxes from sharing sibling hyperthreads on the same physical core to prevent microarchitectural side-channel attacks, reducing usable node vCPU capacity by roughly half. Single-threaded AWS Graviton processors have no sibling hyperthreads, completely avoiding core scheduling penalties and preserving full vCPU capacity.
Sources and further reading
This article was drafted with AI assistance from the publicly cited sources and passed automated checks for structure and sourcing. Verify critical details against the vendor advisories linked above before acting.