Search the publication

Results for “Batching”

Showing 1–8 of 8 · latest 8 stories

Batching

04Local LLMs

Sep 5, 20262 min read

Migrating Local LLM Workloads from Ollama to vLLM

A practical architectural guide details when engineering teams should transition local inference from Ollama to vLLM. While Ollama streamlines developer workstations, vLLM becomes necessary for concurrent agent batching and multi-tenant serving.

01Token & cost optimization

Aug 22, 20262 min read

OpenAI Cuts GPT-5.6 Sol API and Codex Credit Pricing by 20%

OpenAI has lowered API and credit pricing for its frontier GPT-5.6 Sol model by more than 20% across API endpoints and eligible ChatGPT Work plans. The promotional discount is active through at least November 21, 2026.

01Agents & MCP

Jul 13, 20262 min read

Analyzing Claude Code Token Overhead and Caching Costs Against OpenCode

A systematic analysis of developer agents reveals that Claude Code injects around 33,000 tokens of system prompts, tool schemas, and reminders before reading your query, compared to OpenCode's minimal 7,000 tokens. This overhead escalates with system instructions, custom AGENTS.md files, and active Model Context Protocol servers, heavily impacting prompt-caching efficiency.

02Local LLMs

Jul 3, 20262 min read

Interfaze Open-Sources Multilingual Speech-to-Text Model Powered by Parallel Diffusion

Interfaze has open-sourced `diffusion-gemma-asr-small`, a multilingual speech-to-text model built on Google's DiffusionGemma-26B. It transcribes 6 languages in parallel using a tiny 42M-parameter adapter, processing entire transcripts bidirectionally.

03Tutorials & guides

Aug 30, 20262 min read

Build Custom Batched Ensemble Weather Forecasting with NVIDIA Earth2Studio

Learn how to install NVIDIA Earth2Studio in Google Colab while preserving your CUDA PyTorch environment, load the FCN prognostic model, and implement custom wind-power diagnostics. The guide walks through building a coordinate-aware Zarr data backend and executing batched ensemble forecasts.

07Token & cost optimization

Aug 15, 20262 min read

Achieving a 232x Faster GPU Kernel Using Codex in an Auto-Research Loop

A GPU Mode contest participant used Codex in a tight automated feedback loop to optimize a batched QR decomposition CUDA kernel, achieving a 232x speedup over baseline. The approach highlights how agentic loop engineering and automated benchmarking allow developers to iterate rapidly on high-performance code.

08Tools & releases

Aug 3, 20262 min read

JFrog Exposes Batch of Fabricated SQLite CVEs Generated by LLMs

Security researchers discovered that over 50 GitHub vulnerability advisories for SQLite were entirely AI-generated LLM slop. The cited code, functions, and patch references did not exist in the target versions.

08Token & cost optimization

Jul 30, 20262 min read

GPU Management: Why Idle Hardware is the Next Enterprise Bottleneck

Enterprise AI is shifting from model intelligence to hardware utilization constraints. Much like aircraft fleet economics, GPUs accrue fixed costs by the calendar hour while output depends purely on continuous workload scheduling across training, inference, and fine-tuning.