ModelRein (formerly LMCP) is an open protocol and agent runtime for controlling and adapting open-source foundation models. It defines a single HTTP contract that lets an agent reach into any Hugging Face model — residual stream, attention heads, individual transformer layers, final logits — and compose heterogeneous models (text, image, depth, segmentation, audio, video) into one pipeline call.
One protocol. One agent. Any open-source model. That's the whole pitch. Agents written against ModelRein don't have to be rewritten each time a new model drops; they call the same 27 primitives and the server adapts to the model via runtime introspection.
ModelRein is a specification plus a reference server. The specification is an HTTP+JSON contract. The reference server is a FastAPI application that wraps transformers, diffusers, and nnsight and exposes 27 primitives grouped into 9 pillars. Any model with a loadable Hugging Face checkpoint can be driven by the same client code.
config.json and probes layer counts, head counts, head dims, embedding dims at load time.${step.field} template binding; outputs of step k become inputs to step k+1.ModelRein ships adapters for six model families. The HTTP surface is identical; only the adapter plumbing differs.
Qwen, Llama, DeepSeek, Mistral, Gemma, GLM, Phi, Yi, Falcon, GPT-NeoX
Stable Diffusion 1.5 / 2.x variants
SDXL / SDXL-Turbo, PixArt-Sigma, FLUX
Qwen2-VL, LLaVA, BLIP, BLIP-2
MusicGen, AudioLDM, Stable Audio
CogVideoX, Zeroscope, ModelScope T2V
Every open-source model drop today ships with its own repo, its own README, its own quirks about tokenizer handling, generation config, and how to reach into intermediate activations. Agents built on top of a single model's API break the moment you switch to another.
ModelRein decouples the agent from the model. An agent asks ModelRein: "give me the residual stream of layer 22, inject this concept vector, generate 80 tokens." The agent doesn't care whether that model is Qwen 2.5 or Llama 3.1 or DeepSeek V3. The server adapts. Add a new model to Hugging Face Hub, point ModelRein at the ID, done.
Different animals. MCP (Model Context Protocol) is a protocol for feeding external tools and context into a chat model via a client-server split. ModelRein (formerly LMCP) is for reaching inside the model: activations, heads, layers, logits — things MCP has no notion of. A single system can use both: MCP for tool invocation, ModelRein for the model-internals it uses to serve those tools.
The v0.4 technical report documents 8 cached model families running concurrently through POST /lmcp/pipeline: Qwen2.5-1.5B-Instruct → SDXL-turbo → Depth-Anything-V2 → SAM-ViT-Base → BLIP-base → MusicGen-small → Whisper-base → CogVideoX-2b. 22-step stress test, 27.5s combined wall clock on one RTX PRO 6000 Blackwell GPU, peak 26.6 GB VRAM.
This is the point: the same compositional primitive drives all of them. No per-model code. A single JSON body tells the server to render a scene with SDXL, extract depth with Depth-Anything, segment with SAM, caption each region with BLIP, summarize with Qwen, score with MusicGen, verify with Whisper, animate with CogVideoX — in sequence, in one wall-clock, with field-level binding between steps.