Oleksandr Kuzmenko, EditorPublished 6 min readAI-assisted · editor-reviewed
TL;DR
- Confidential computing on NVIDIA B200 retains 96.1% to 98.2% of baseline output token throughput.
- TensorRT-LLM avoids bounce-buffer bottlenecks by swapping pinned memory for pageable memory in CVMs.
- Kernel autotuning under confidential computing requires using the GPU %globaltimer instead of CUDA events.
Key facts
- Throughput Retained
- 96.1% to 98.2% of CC-off baseline
- Latency Overhead (TPOT)
- 1.2% to 4.3% increase
- Hardware Tested
- 1x NVIDIA DGX B200 (8x B200 GPUs)
- Software Release
- TensorRT-LLM 1.3.0rc22
Eliminating Bottlenecks in Protected Enclaves
Confidential computing on NVIDIA DGX B200 environments isolates tenant weights and prompts using memory-encrypted CVMs and encrypted NVLink connections. However, hardware isolation alters standard memory access paradigms. TensorRT-LLM introduces targeted runtime patches to counteract data movement penalties:
- Host-to-Device Paging: Because protected CVM memory blocks direct GPU access, host copies use encrypted software bounce buffers. Pinned memory no longer guarantees asynchronous execution and can stall threads. TensorRT-LLM selects pageable memory dynamically on affected pipelines.
- Asynchronous Token Offloading: Pulling sampling data and generated tokens back to the host blocks the primary decode loop. TensorRT-LLM delegates these transfers to dedicated asynchronous workers via PR #11573.
- Hardware Clock Autotuning: Standard CUDA events generate jitter inside confidential environments, skewing kernel tactic comparisons. TensorRT-LLM switches tactic evaluation to the internal
%globaltimerregister via PR #11657.
Benchmarked Throughput and Latency
Under low-concurrency, long-context evaluation—specifically chosen to highlight encryption stalls during prefill and per-token decode overhead—an 8-GPU B200 DGX cluster running TensorRT-LLM 1.3.0rc22 retained between 96.1% and 98.2% of unencrypted token throughput. Output token generation latency (TPOT) increased by only 1.2% to 4.3% relative to the non-confidential baseline.
Platform teams planning confidential deployments must account for multi-GPU communication constraints: NVLink SHARP (NVLS) multicast is disabled in B200 confidential modes. Workload managers must configure NCCL collectives to bypass multicast paths and avoid unneeded synchronization penalties.
Try it in 2 minutes
docker pull nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc22When to use
- Deploying proprietary model weights or enterprise user prompts in compliance-heavy healthcare or financial clouds.
- Running DeepSeek-R1 and similar reasoning models in zero-trust multi-tenant GPU clouds.
When NOT to use
- When running local development environments without regulatory or multi-tenant privacy isolation needs.
- When relying on unsupported multi-GPU collective multicast mechanisms such as NVLink SHARP.
What to do today
- Upgrade production inference containers to TensorRT-LLM 1.3.0rc22 or later when running in confidential VMs.
- Disable NVLink SHARP collective paths in NCCL configuration when launching multi-GPU confidential environments.
- Benchmark Time Per Output Token across concurrency levels 1 to 16 to measure encryption overhead on custom sequence lengths.
Step names are taken from What to do today.
Step names are taken from What to do today. Upgrade production inference containers to TensorRT-LLM 1.3.0rc22 or later when running in confidential VMs.. Disable NVLink SHARP collective paths in NCCL configuration when launching multi-GPU confidential environments.. Benchmark Time Per Output Token across concurrency levels 1 to 16 to measure encryption overhead on custom sequence lengths.
- Upgrade production inference containers to TensorRT-LLM 1.3.0rc22 or later when running in confidential VMs.
- Disable NVLink SHARP collective paths in NCCL configuration when launching multi-GPU confidential environments.
- Benchmark Time Per Output Token across concurrency levels 1 to 16 to measure encryption overhead on custom sequence lengths.