fork() for AI agents.When your agent forks N ways to explore a problem, thaw skips the cold prefill and runs them in parallel from one shared memory. The substrate for RL rollouts, multi-agent reasoning, and parallel coding agents.
Snapshot the engine, restore N children that share the parent's KV cache at the fork point, diverge. Same process or across workers.
from thaw_vllm import LLM, fork # A live vLLM engine, already serving traffic.parent = LLM("meta-llama/Llama-3.1-8B")parent.generate(prefix_messages) # warms the KV cache + prefix-hash table # Snapshot mid-stream → fork into N divergent children.children = fork(parent, n=4) # 0.88s median on H100 · steady state # Each child resumes from the parent's exact context.# No re-prefill. Same trunk, different futures.for child, suffix in zip(children, suffixes): print(child.generate(suffix))N rollouts × prefill_timeN rollouts × memcpy_timeFork a reasoning trunk into N parallel hypotheses mid-conversation. Each child inherits the parent's KV cache at the fork point, runs concurrently on the same GPU, then you pick the winner. The expensive trunk only gets paid for once.
One parent engine, N children, zero re-prefill — but the magic is in four specific tricks. Watch the loop: scattered KV slabs coalesce, ping-pong through two pinned buffers, land on children via PCIe Gen5, then get tagged with prefix hashes so the next request is a cache hit. Real mechanics, real numbers.
KV slabs gather into one contiguous tensor
Two pinned buffers ping-pong · 86% of PCIe Gen5 ceiling
Chunks land on child GPUs · parallel verification
Prefix hashes re-inserted · children skip prefill
H100 80GB PCIe · Llama-3.1-8B · 5 rounds × 4 branches × 64 tokens
H100 SXM · pinned host memory · double-buffered pipelined DMA
2× H100 SXM · Llama-3.1-70B TP=2 · bit-identical across 8 architectures
Pinecone became a company by being the easiest vector DB the moment RAG mattered. LangChain became a company by being the orchestration layer everyone calls. thaw becomes a company by being the fork primitive every agent framework — TRL, slime, verl, LangGraph, Temporal — calls when an agent loop needs to branch.
The fork() runtime is Apache-2.0, written in Rust + CUDA, integrated with the engines real production traffic runs on. Anyone can pip install thaw-vllm and snapshot a live agent session.
Working with Courier (SLC) on MLX-side inference for the Apple-silicon agentic stack — paid integration sprint validates the snapshot/restore semantics against a non-vLLM engine in production. RFC #34303 upstream with vLLM.
The OSS primitive is the wedge. The company is the framework-layer integrations — TRL, slime, verl, LangGraph, Temporal — that make fork() a first-class verb in every agent loop. Agent branching, RL rollouts, parallel coding agents, multi-agent reasoning. Not a platform. A primitive.
First-class integration via vLLM's load_format extension. Snapshot a live vLLM engine, restore into a fresh process, prefix-cache hash table rebuilt on the way back. RFC #34303 in flight upstream for sleep-mode integration.
RFC #34303 ↗from vllm import LLMfrom thaw_vllm import fork parent = LLM(model="meta-llama/Llama-3.1-8B", load_format="thaw") children = fork(parent, n=4) # PCIe Gen5 line-rateClass-passthrough loader keeps SGLang's scheduler intact and snapshots the runtime state directly. Same fork() ergonomics as vLLM. Validated on H100 SXM with Llama, Qwen, and DeepSeek architectures.
from sglang import Enginefrom thaw_native import ThawEngine parent = ThawEngine(Engine, "…/Qwen2.5-7B")parent.generate(prefix) children = parent.fork(n=4)Drop-in LangChain BaseChatModel — every existing LangGraph node works unchanged. fork_fanout(llm, prefix, [suffixes]) exposes explicit fan-out for tool-use branching and parallel reviewers. PR-review demo: 4 reviewers, 1.43s median round.
from thaw_vllm.langgraph import ChatThaw, fork_fanout llm = ChatThaw(model="…/Llama-3.1-8B") reviews = fork_fanout( llm, diff_messages, suffixes=["security", "perf", "tests", "style"])Two PyPI packages: thaw-vllm for first-class vLLM integration, thaw-native for the underlying Rust runtime. CUDA wheels published for Python 3.10–3.12. Apache-2.0.
pip install thaw-vllmpip install thaw-native