NVIDIA Cuts Confidential Computing Overhead in TensorRT-LLM Below Five Percent
NVIDIA released architectural mitigations in TensorRT-LLM for confidential computing on DGX B200 systems. It preserves over 96 percent throughput on DeepSeek-R1 while keeping token latency overhead below five percent.
Impact: High
Why it matters
You can now deploy regulated, privacy-sensitive LLM workloads on enterprise GPUs without paying a crippling performance tax.
TL;DR
- 01Confidential computing on NVIDIA B200 retains 96.1% to 98.2% of baseline output token throughput.
- 02TensorRT-LLM avoids bounce-buffer bottlenecks by swapping pinned memory for pageable memory in CVMs.
- 03Kernel autotuning under confidential computing requires using the GPU %globaltimer instead of CUDA events.
Key facts
- Throughput Retained
- 96.1% to 98.2% of CC-off baseline
- Latency Overhead (TPOT)
- 1.2% to 4.3% increase
- Hardware Tested
- 1x NVIDIA DGX B200 (8x B200 GPUs)
- Software Release
- TensorRT-LLM 1.3.0rc22
Eliminating Bottlenecks in Protected Enclaves
Confidential computing on NVIDIA DGX B200 environments isolates tenant weights and prompts using memory-encrypted CVMs and encrypted NVLink connections. However, hardware isolation alters standard memory access paradigms. TensorRT-LLM introduces targeted runtime patches to counteract data movement penalties:
- Host-to-Device Paging: Because protected CVM memory blocks direct GPU access, host copies use encrypted software bounce buffers. Pinned memory no longer guarantees asynchronous execution and can stall threads. TensorRT-LLM selects pageable memory dynamically on affected pipelines.
- Asynchronous Token Offloading: Pulling sampling data and generated tokens back to the host blocks the primary decode loop. TensorRT-LLM delegates these transfers to dedicated asynchronous workers via PR #11573.
- Hardware Clock Autotuning: Standard CUDA events generate jitter inside confidential environments, skewing kernel tactic comparisons. TensorRT-LLM switches tactic evaluation to the internal
%globaltimerregister via PR #11657.
Benchmarked Throughput and Latency
Under low-concurrency, long-context evaluation—specifically chosen to highlight encryption stalls during prefill and per-token decode overhead—an 8-GPU B200 DGX cluster running TensorRT-LLM 1.3.0rc22 retained between 96.1% and 98.2% of unencrypted token throughput. Output token generation latency (TPOT) increased by only 1.2% to 4.3% relative to the non-confidential baseline.
Platform teams planning confidential deployments must account for multi-GPU communication constraints: NVLink SHARP (NVLS) multicast is disabled in B200 confidential modes. Workload managers must configure NCCL collectives to bypass multicast paths and avoid unneeded synchronization penalties.
Try it in 2 minutes
docker pull nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22bash
✓ When to use
- Deploying proprietary model weights or enterprise user prompts in compliance-heavy healthcare or financial clouds.
- Running DeepSeek-R1 and similar reasoning models in zero-trust multi-tenant GPU clouds.
✕ When NOT to use
- When running local development environments without regulatory or multi-tenant privacy isolation needs.
- When relying on unsupported multi-GPU collective multicast mechanisms such as NVLink SHARP.
What to do today
- Upgrade production inference containers to TensorRT-LLM 1.3.0rc22 or later when running in confidential VMs.
- Disable NVLink SHARP collective paths in NCCL configuration when launching multi-GPU confidential environments.
- Benchmark Time Per Output Token across concurrency levels 1 to 16 to measure encryption overhead on custom sequence lengths.
Sources