TL;DR
- Replacing legacy bastion host port-forwarding with namespace-scoped Envoy mTLS gateways reduces command latency for unauthenticated TCP protocols from 1–2 seconds to under 100 ms and application installation times from 3–4 minutes to under 30 seconds.
- Short-lived 12-hour mTLS certificates issued via cert-manager and AWS Private CA rotate automatically at the 4-hour remaining window using Envoy Secret Discovery Service (SDS) without dropping active TCP connections or restarting pods.
- Default-deny ingress NetworkPolicies are mandatory in every tenant namespace; without kernel-level network isolation, unencrypted TCP ports remain exposed to cross-namespace pod traffic across the flat cluster network.
- Migrating WebRTC media relay from self-managed STUNner gateways to Amazon Kinesis Video Streams eliminates custom token-exchange services and prevents WebRTC session drops during Availability Zone pod migrations.
- The architecture decouples legacy TCP authentication (ADB, VNC, RDP, MQTT) from application code modifications by establishing a zero-trust mTLS proxy boundary tied to an enterprise Identity Provider (IdP) JWT chain.
Legacy TCP-based protocols such as Android Debug Bridge (ADB), Virtual Network Computing (VNC), and Remote Desktop Protocol (RDP) present persistent security and operational challenges in containerised platform environments. Designed for local execution or isolated enterprise networks, these protocols lack native support for modern federated authentication standards like JSON Web Tokens (JWT) and routinely default to unencrypted transport channels. In a shared multi-tenant kubernetes mtls environment, this architectural gap exposes cleartext traffic across tenant boundaries and forces platform teams to maintain complex, high-latency jump-host infrastructure.
According to the AWS Security Blog, traditional access patterns that route legacy protocol traffic through bastion hosts and two-hop Kubernetes API server port-forwarding create severe operational bottlenecks. Under a legacy bastion architecture deployed for virtual Android devices (Google Cuttlefish), ADB commands suffered latency of 1 to 2 seconds per round trip, while application installations took up to 4 minutes. Furthermore, centralising proxy traffic through the Kubernetes API server limits system throughput, introduces control plane rate-limiting risks, and obscures granular audit trails.
Modern platform engineering requires replacing shared bastion hosts with a zero-trust network model. By combining namespace-scoped Envoy mTLS gateways, automated short-lived certificate management via cert-manager and AWS Private CA, and managed TURN media relays, engineering teams can secure legacy TCP streams, enforce identity attribution down to AWS Security Token Service (STS) role sessions, and cut gateway processing overhead under 50 ms.
The Legacy Bastion Bottleneck in Multi-Tenant Kubernetes
Legacy TCP protocols operate on trust models that fail to meet modern zero-trust standards. Android Debug Bridge (ADB), for instance, relies on a client-server daemon architecture over unencrypted TCP sockets. Authentication is restricted to local RSA key-pair checks without support for centralized single sign-on (SSO), multi-factor authentication (MFA), or OAuth2/OIDC identity tokens. Protocols like VNC and RDP share similar limitations: they depend on local credential stores or domain-joined configurations that are difficult to manage dynamically inside short-lived container instances. Wrapping these protocols individually by patching upstream source code is unmaintainable across enterprise software lifecycles.
In standard multi-tenant Kubernetes clusters—including Amazon Elastic Kubernetes Service (Amazon EKS)—the underlying overlay network (such as the AWS VPC CNI plugin) creates a flat network by default. Pods in any namespace can send IP packets directly to open ports on pods in any other namespace unless explicitly restricted. If an unauthenticated ADB service listens on TCP port 5555 inside Tenant Namespace A, an infected or misconfigured container in Tenant Namespace B can establish an unencrypted socket to that port across namespace boundaries.
To prevent raw network exposure, early remote-access architectures relied on shared bastion hosts and Kubernetes control plane proxying. A developer CLI authenticated against a custom management API, which checked user permissions against an enterprise Identity Provider (IdP). Upon validation, the CLI opened a two-hop port-forwarding tunnel:
- Hop 1: Direct SSH or HTTPS tunnel from the developer workstation to a central bastion host.
- Hop 2: A secondary
kubectl port-forwardconnection initiated by the bastion to the target pod via the Amazon EKS control plane API server.
+------------------+ Hop 1 (HTTPS/SSH) +------------------+
| Developer Laptop | ------------------------> | Bastion Host |
+------------------+ +------------------+
|
| Hop 2 (API Server Proxy)
v
+------------------+ Pod-to-Pod Overlay +------------------+
| Target Tenant Pod| <----------------------- | EKS Control |
| (ADB Port 5555) | | Plane API Server |
+------------------+ +------------------+
This design introduces critical operational performance drags. Routing raw binary protocol streams through the Kubernetes API server serialises data through the control plane’s API server process, causing CPU saturation and socket exhaustion under high concurrency. According to the AWS Security Blog, this two-hop port-forwarding model inflated ADB command latency to 1–2 seconds per round trip and prolonged mobile application package installations to 3–4 minutes.
Audit logging also breaks down in this pattern. Because the API server acts as the network proxy for the final hop, traffic reaching the target pod originates from control plane internal interfaces. Unless custom logging layers are built to map every TCP frame back to an identity session, cloud audit trails (such as AWS CloudTrail) only log the bastion host’s service account, obscuring individual developer session accountability.
Architectural Overview: Namespace Isolation, mTLS, and Identity Chains
To eliminate performance bottlenecks while maintaining strict security isolation, platform teams can implement a zero-trust architecture based on three reusable platform components:
- Namespace-Scoped Envoy Gateways: Dedicated Envoy proxy instances deployed per tenant namespace, serving as layer 4 (L4) mTLS termination points.
- Automated Certificate Lifecycles: cert-manager integrated with the AWS Private CA issuer plugin (using
AWSPCAClusterIssuer) to provision short-lived X.509 certificates. - Managed Media Relays: Amazon Kinesis Video Streams operating as a serverless STUN/TURN provider for WebRTC media streams.
The architecture enforces a strict separation between the HTTP control plane and the mTLS TCP data plane. The control plane manages user identity validation and certificate issuance, while the data plane handles direct, high-throughput mTLS proxying between the client CLI and the tenant workload.
+-----------------------------------------------------------------------------------+
| CONTROL PLANE (HTTPS / REST) |
| |
| Developer CLI --> Enterprise IdP (OIDC/JWT) |
| | |
| +----------> Custom Auth REST API (Behind AWS WAF) |
| | |
| +--> Validates JWT & Maps User to Tenant Namespace |
| +--> Assumes IAM Role (STS RoleSessionName = User ID) |
| +--> Fetches Namespace mTLS Client Cert from Vault/Secret |
| +--> Returns Short-Lived mTLS Credentials to CLI |
+-----------------------------------------------------------------------------------+
+-----------------------------------------------------------------------------------+
| DATA PLANE (mTLS / TCP) |
| |
| Developer CLI ==================================================> Envoy Gateway |
| (presents mTLS Cert) Direct mTLS Connection (Port 443) (Tenant Namespace)|
| | |
| | Cleartext
| | TCP |
| v |
| Target Pod |
| (Port 5555) |
+-----------------------------------------------------------------------------------+
When a developer requests access to a target container, the custom CLI initiates an authentication request to the custom REST API operating behind AWS WAF. The REST API validates the user’s JWT issued by the enterprise IdP. Upon successful authentication, the API maps the user’s corporate identity to their assigned Kubernetes namespace and assumes an AWS IAM role. By passing the developer’s unique user identifier into the AWS STS RoleSessionName parameter, every subsequent management call in AWS CloudTrail is immutably linked to an individual employee.
Once identity and authorization are verified, the REST API returns short-lived mTLS client credentials (issued via cert-manager and AWS Private CA) directly to the CLI over HTTPS. The CLI then opens a direct, single-hop TCP mTLS socket to the namespace-scoped Envoy Gateway, completely bypassing the Kubernetes control plane API server.
Implementing Namespace-Scoped mTLS Proxying with Envoy Gateway
Envoy Gateway—a Kubernetes API gateway built on the open-source Envoy proxy—acts as the ingress controller for incoming TCP streams. Rather than deploying a monolithic, cluster-wide gateway instance that bridges all network traffic, platform engineers deploy namespace-scoped Envoy Gateway instances. Each tenant namespace hosts its own Envoy Gateway deployment, ensuring complete control plane and data plane separation between tenants.
When the client CLI initiates a connection to the Envoy Gateway endpoint, a mutual TLS (mTLS) handshake occurs:
- Server Authentication: The client CLI verifies the Envoy Gateway’s server certificate against the root CA bundle to ensure it is communicating with a legitimate infrastructure endpoint.
- Client Authentication: Envoy Gateway verifies the X.509 client certificate presented by the CLI against the trusted CA. Envoy validates that the Subject Alternative Name (SAN) or Common Name (CN) matches the target namespace pattern (e.g.,
tenant-a.eks.internal).
Once the mTLS handshake succeeds, Envoy Gateway strips the mTLS encryption layer and proxies the underlying raw TCP stream directly to the target pod’s IP within the local namespace. Envoy uses TCP connection pooling and keep-alive optimization to maintain efficient socket reuse across sessions.
According to the AWS Security Blog, replacing double-hop API server port-forwarding with direct namespace-scoped Envoy mTLS proxying reduces total gateway processing overhead to under 50 ms. Total ADB command execution latency drops below 100 ms (measured as the round-trip differential compared to a locally attached hardware device), while container application installation times are cut from 3–4 minutes to under 30 seconds.
Automating Dynamic Certificate Lifecycle with cert-manager and AWS Private CA
Managing mTLS at scale across dozens or hundreds of tenant namespaces requires full lifecycle automation for internal X.509 certificates. Manually issuing static certificates creates administrative operational overhead and introduces security risks associated with long-lived credentials.
The automated certificate pipeline uses cert-manager paired with the official aws-privateca-issuer plugin. To prevent duplicating resource configurations inside every tenant namespace, platform operators deploy a single, cluster-scoped AWSPCAClusterIssuer resource. This cluster issuer grants cert-manager permissions to request private X.509 certificates from a central AWS Private CA instance.
apiVersion: awspca.cert-manager.io/v1beta1
kind: AWSPCAClusterIssuer
metadata:
name: aws-privateca-issuer
spec:
arn: arn:aws:acm-pca:eu-west-1:123456789012:authority/12345678-1234-1234-1234-123456789012
region: eu-west-1
Each tenant namespace defines a cert-manager Certificate manifest referencing the central AWSPCAClusterIssuer. To minimize exposure in the event of client credential exfiltration, certificates are issued with a short lifetime of 12 hours.
Certificate Renewal Window Mechanics
cert-manager evaluates certificate renewal timing based on total lifetime duration. By default, renewal begins at the two-thirds mark of a certificate’s lifespan. For a 12-hour certificate, automated renewal triggers when 4 hours of validity remain (8 hours after issuance). This 4-hour window provides cert-manager with multiple retry attempts against the AWS Private CA API in the event of transient network disruptions before the active certificate lapses.
Certificates enforce two critical operational configuration requirements:
- Naming Conventions: The certificate Common Name (CN) and Subject Alternative Names (SANs) use a dedicated internal domain convention, such as
<tenant-namespace>.eks.internal. This prevents DNS naming collisions with public corporate domains and allows Envoy to validate tenant ownership during mTLS handshakes. - Key Usages: The certificate specification must include both
clientAuth(Extended Key Usage for TLS Web Client Authentication) andserverAuth(Extended Key Usage for TLS Web Server Authentication) because the same certificate authority infrastructure generates both client CLI certificates and gateway server termination certificates.
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: envoy-mtls-cert
namespace: tenant-alpha
spec:
secretName: envoy-mtls-tls-secret
duration: 12h
renewBefore: 4h
subject:
organizations:
- ITSTCK Tenant Infrastructure
commonName: tenant-alpha.eks.internal
dnsNames:
- tenant-alpha.eks.internal
- *.tenant-alpha.eks.internal
usages:
- digital signature
- key encipherment
- server auth
- client auth
issuerRef:
name: aws-privateca-issuer
kind: AWSPCAClusterIssuer
group: awspca.cert-manager.io
When cert-manager renews a certificate, it updates the corresponding Kubernetes Secret object (envoy-mtls-tls-secret). Envoy Gateway dynamically detects updated X.509 certificate pairs in Kubernetes Secrets via Envoy’s native Secret Discovery Service (SDS). SDS hot-reloads the new TLS certificates directly into Envoy’s active memory buffers without dropping existing long-lived TCP sockets, terminating active ADB debug sessions, or requiring pod restarts.
Because X.509 certificates expire automatically within 12 hours, compromised client keys are rendered useless shortly after exfiltration. This short lifespan eliminates the requirement to deploy complex, heavy Certificate Revocation List (CRL) or Online Certificate Status Protocol (OCSP) verification infrastructure inside the Kubernetes cluster.
Eliminating Relaying Friction: Shifting from STUNner to Kinesis Video Streams
Where TCP control protocols like ADB require raw stream proxying, interactive media streaming protocols (such as WebRTC for VNC screen rendering or remote desktop access) require NAT traversal utilities. When developer workstations and target container pods sit behind separate NAT gateways or firewalls, direct peer-to-peer UDP/TCP paths cannot be established, requiring a Traversal Using Relays around NAT (TURN) media relay.
Early iterations of containerised remote desktop platforms relied on self-managed relay gateways, such as STUNner (an open-source Kubernetes STUN/TURN gateway). According to the AWS Security Blog, self-managed TURN infrastructure introduces significant operational overhead:
- Custom Token Services: Because STUNner lacked native integration with enterprise IdPs and AWS IAM, platform engineering teams had to construct and maintain custom microservices to exchange IdP identity tokens for time-limited STUN/TURN credentials.
- Connection Instability During Migrations: When Kubernetes node autoscaling or spot instance interruptions caused container pod migrations across Availability Zones (AZs), self-managed STUNner required manual routing table updates, resulting in dropped WebRTC media streams for active users.
- Operational Maintenance: Scaling, patching, and monitoring clusters of TURN EC2 instances created recurring maintenance overhead for platform SRE teams.
To eliminate this friction, platform teams can replace self-managed TURN infrastructure with Amazon Kinesis Video Streams WebRTC. Kinesis Video Streams provides managed STUN/TURN relay infrastructure natively integrated with AWS IAM.
| Architectural Dimension | Self-Managed STUNner Gateway | Amazon Kinesis Video Streams (KVS) |
|---|---|---|
| Credential Authentication | Custom token-exchange microservice mapping IdP tokens to TURN static secrets. | Native AWS IAM integration using temporary STS credentials and session policies. |
| Multi-AZ Migration Resilience | Connection drops during pod rescheduling; requires manual route table updates. | Managed global endpoints maintain session state across Availability Zone shifts without drops. |
| Operational Overhead | High: ongoing OS patching, TURN daemon tuning, capacity management. | Zero server management; fully managed AWS infrastructure. |
| Billing & Cost Structure | Fixed hourly EC2/EKS instance compute, NAT Gateway egress, network interface costs. | Pay-per-use: active signaling channels, signaling messages, and TURN streaming minutes. |
| Control Plane Signaling | Unencrypted custom signaling endpoints required custom TLS termination wrappers. | Integrated signaling channels with granular IAM access authorization. |
Shifting to Kinesis Video Streams simplifies authentication. When a user requests a WebRTC media session, the management API generates short-lived, session-scoped TURN credentials using AWS STS policies. The application passes these temporary credentials directly to the client WebRTC stack. Idle tenant namespaces incur zero cost because Kinesis Video Streams bills strictly on active signaling channels, message counts, and TURN streaming minutes.
Security Boundaries and Mandatory Ingress Controls: NetworkPolicy and SDS
Deploying an mTLS proxy gateway does not automatically secure a multi-tenant cluster. An mTLS gateway is an ingress point; if the surrounding network flatland remains unrestricted, malicious or compromised workloads can bypass the gateway entirely.
Mandatory Defense-in-Depth Prerequisite
An mTLS gateway terminates mutual TLS authentication at the edge of the namespace. However, the final network hop—from the Envoy Gateway pod to the target application pod—occurs as unencrypted TCP over the namespace container network interface (CNI). Default-deny NetworkPolicies are strictly required in every tenant namespace to block direct pod-to-pod cross-tenant access. Without these policies, any pod in the cluster can bypass mTLS controls by connecting directly to the target pod’s open TCP port.
To establish a secure boundary, platform teams must deploy a default-deny ingress NetworkPolicy inside every tenant namespace prior to issuing certificates or provisioning gateways. Once default isolation is active, an explicit allow rule is applied to permit ingress traffic to the application’s unencrypted target port (e.g., ADB port 5555) only if the traffic originates from the namespace-scoped Envoy Gateway pod.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all-ingress
namespace: tenant-alpha
spec:
podSelector: {}
policyTypes:
- Ingress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-envoy-to-adb-target
namespace: tenant-alpha
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: cuttlefish-adb-device
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: envoy-gateway-proxy
ports:
- protocol: TCP
port: 5555
Platform security architects must carefully evaluate security boundaries based on their organization’s data classification standard:
- Namespace-Scoped Gateway Isolation (Standard Enterprise): Envoy terminates mTLS at the namespace boundary and forwards cleartext TCP over the isolated pod network. Suitable for non-sensitive dev/test environments and standard multi-tenant workloads. NetworkPolicies guarantee that cross-namespace traffic cannot reach port 5555.
- Pod-Level Sidecar mTLS (High Regulatory Compliance): For strict compliance frameworks (e.g., PCI-DSS, HIPAA, FedRAMP High) that prohibit unencrypted network hops even on loopback or local CNI bridges, an Envoy sidecar proxy or pod-level certificate must be deployed directly inside the workload pod to terminate mTLS inside the pod’s network namespace.
Step-by-Step Implementation Guide for Platform and SRE Teams
Engineering teams adapting this pattern to secure unauthenticated TCP protocols (ADB, VNC, RDP, MQTT) should execute implementation across four distinct phases.
Phase 1: Cluster Infrastructure & Network Isolation
- Deploy
cert-managerand theaws-privateca-issuerplugin to the EKS cluster. - Provision an AWS Private CA instance in your AWS account and configure an
AWSPCAClusterIssuerresource in Kubernetes. - Apply a default-deny ingress
NetworkPolicyto all target tenant namespaces. Verify that inter-namespace pod communication on protocol target ports (e.g., TCP 5555, 5900, 3389) is completely blocked.
Phase 2: Management Control Plane & Authentication
- Deploy the custom management REST API behind AWS WAF. Configure WAF rulesets to protect against standard web application vectors.
- Integrate the REST API with your enterprise IdP (e.g., Okta, Entra ID) to validate incoming OIDC JWT bearer tokens.
- Configure AWS IAM role-mapping logic within the REST API, ensuring that calls pass the authenticated user’s ID into the AWS STS
RoleSessionNameparameter for audit traceability.
Phase 3: Data Plane & Envoy Gateway Deployment
- Install Envoy Gateway in the cluster and provision a dedicated, namespace-scoped Envoy instance for each tenant namespace.
- Apply a cert-manager
Certificateresource in each namespace referencing theAWSPCAClusterIssuer. Ensure the certificate duration is configured for 12 hours, with a 4-hour renewal window, containing bothclientAuthandserverAuthkey usages. - Configure Envoy Gateway route objects to map incoming TLS connections on port 443 to the target pod’s internal unencrypted TCP port.
Phase 4: Client Distribution & SDS Verification
- Distribute the custom developer CLI tool configured to interact with the management REST API over HTTPS.
- Validate end-to-end mTLS handshakes: ensure the CLI retrieves short-lived client certificates from the management REST API and connects successfully through Envoy Gateway.
- Verify zero-downtime certificate rotation: trigger a manual certificate renewal via
cert-manager(usingcmctl renew) and verify via Envoy SDS metrics that certificates update dynamically without terminating active TCP connections or dropping packets.
Common Implementation Mistakes
- Omitting Extended Key Usages: Specifying only
serverAuthin the cert-manager manifest causes client CLI authentication attempts to fail during mTLS handshakes. BothclientAuthandserverAuthmust be defined. - Deploying Gateways Before NetworkPolicies: Provisioning Envoy Gateway without enforcing default-deny NetworkPolicies leaves the target pod ports exposed across the flat pod network during the transition window.
- DNS Domain Collisions: Using public domain extensions (e.g.,
tenant.company.com) for internal certificate Common Names can cause split-horizon DNS conflicts. Always use an internal-only naming convention like.eks.internal.
Key Takeaways for Securing Unauthenticated Protocols at Scale
Securing legacy, unauthenticated TCP protocols in multi-tenant Kubernetes does not require modifying application source code or maintaining high-latency bastion hosts. By decoupling network authentication from the application layer, platform teams can enforce zero-trust security controls while improving execution performance.
- Decouple Auth via Gateway Proxies: Wrap cleartext TCP protocols (ADB, VNC, RDP, MQTT) in namespace-scoped Envoy mTLS gateways to add enterprise identity controls without altering upstream application code.
- Automate Short-Lived Certificate Lifecycles: Combine cert-manager and AWS Private CA to issue 12-hour mTLS certificates. Use Envoy Secret Discovery Service (SDS) to hot-reload credentials dynamically without dropping TCP sessions or restarting pods.
- Enforce Mandatory Kernel-Level Isolation: Pair mTLS ingress gateways with namespace-level default-deny NetworkPolicies. Without NetworkPolicies, mTLS protection can be bypassed over the internal cluster network.
- Offload Relaying to Managed Cloud Infrastructure: Replace operational burdens like self-managed STUNner EC2 clusters with managed media relays like Amazon Kinesis Video Streams to ensure zero-downtime cross-AZ transitions and native IAM credential management.
Frequently asked questions
Why wrap unauthenticated TCP protocols in mTLS instead of modifying the source protocol?
Modifying upstream protocol code (such as patching ADB or VNC daemons) requires maintaining custom forks, complicates software update lifecycles, and risks breaking compatibility with standard client tools. Terminating mTLS at an ingress gateway like Envoy isolates protocol security to the infrastructure layer, allowing developers to use off-the-shelf CLI binaries without code modifications.
How does Envoy Gateway reload mTLS certificates without dropping active TCP connections or restarting pods?
Envoy Gateway integrates with Kubernetes Secrets using Envoy’s Secret Discovery Service (SDS) API. When cert-manager rotates a certificate, Envoy detects the updated secret in memory and swaps the active X.509 certificate pointers for new TLS handshakes while maintaining open TCP socket file descriptors for established streaming sessions.
Why are default-deny NetworkPolicies strictly required when implementing an mTLS gateway?
In Kubernetes, pod overlay networks are flat by default, allowing any pod to route traffic directly to open ports on pods in other namespaces. Because Envoy Gateway forwards decrypted TCP traffic to the target pod inside the namespace, default-deny NetworkPolicies are necessary to prevent unauthorized intra-cluster pods from bypassing Envoy entirely.
How does short-lived certificate issuance eliminate the need for CRL or OCSP infrastructure?
By issuing certificates with short lifespans (such as 12 hours) and rotating them automatically every 8 hours, the exposure window of an exfiltrated private key is strictly time-bounded. If a client key is compromised, it expires rapidly without requiring the maintenance, latency, and operational bloat of Certificate Revocation Lists (CRL) or OCSP stapling servers.
Can this mTLS proxy architecture be used for other legacy protocols like VNC, RDP, or MQTT?
Yes. The combination of namespace-scoped Envoy mTLS gateways, cert-manager automation, and default-deny NetworkPolicies operates at Layer 4 (TCP). It applies directly to any binary or cleartext TCP protocol—including VNC, RDP, MQTT, and SIP—replacing local passwords or unauthenticated channels with enterprise mTLS identity validation.
Sources and further reading
- Addressing protocol security gaps in multi-tenant Kubernetes | Amazon Web Services (AWS Security Blog)
This article was drafted with AI assistance from the publicly cited sources and passed automated checks for structure and sourcing. Verify critical details against the vendor advisories linked above before acting.