ModelRein
About ModelRein

ModelRein — open protocol and agent runtime for controlling any open-source model

ModelRein (formerly LMCP) is an open protocol and agent runtime for controlling and adapting open-source foundation models. It defines a single HTTP contract that lets an agent reach into any Hugging Face model — residual stream, attention heads, individual transformer layers, final logits — and compose heterogeneous models (text, image, depth, segmentation, audio, video) into one pipeline call.

One protocol. One agent. Any open-source model. That's the whole pitch. Agents written against ModelRein don't have to be rewritten each time a new model drops; they call the same 27 primitives and the server adapts to the model via runtime introspection.

What is ModelRein, concretely?

ModelRein is a specification plus a reference server. The specification is an HTTP+JSON contract. The reference server is a FastAPI application that wraps transformers, diffusers, and nnsight and exposes 27 primitives grouped into 9 pillars. Any model with a loadable Hugging Face checkpoint can be driven by the same client code.

Nine pillars (A–I)

A
Auto-adapt — zero per-model config. The server reads config.json and probes layer counts, head counts, head dims, embedding dims at load time.
B
Sampling — temperature, top-k, top-p, min-p, repetition penalty, logit bias, EOS-bias ramps for length control.
C
Residual-stream control — compute contrastive steering vectors and inject them at any layer with adaptive strength scaled to the residual norm.
D
Attention-head control — mute, scale, or re-route individual (layer, head) pairs; implements ITI-style edits as a one-line call.
E
Layer routing — skip layers, repeat layers, re-order layers — Mixture-of-Depths style edits applied at inference time with no retraining.
F
Introspection — enumerate controllable surfaces of any loaded model (residual, attention, projector, timestep, cross-attention, UNet block, VAE latent).
G
Compositional pipeline — one call chains N heterogeneous models with ${step.field} template binding; outputs of step k become inputs to step k+1.
H
Judges & metrics — auto-infer success metrics from a natural-language task description; apply them as reward signals.
I
Memory substrate — persistent intervention memory (vectors, layer picks, metric scores) with graph queries.

Six architecture adapters, one surface

ModelRein ships adapters for six model families. The HTTP surface is identical; only the adapter plumbing differs.

causal_lm

Qwen, Llama, DeepSeek, Mistral, Gemma, GLM, Phi, Yi, Falcon, GPT-NeoX

unet

Stable Diffusion 1.5 / 2.x variants

dit

SDXL / SDXL-Turbo, PixArt-Sigma, FLUX

vlm

Qwen2-VL, LLaVA, BLIP, BLIP-2

audio_diffusion

MusicGen, AudioLDM, Stable Audio

video_dit

CogVideoX, Zeroscope, ModelScope T2V

Why does this exist?

Every open-source model drop today ships with its own repo, its own README, its own quirks about tokenizer handling, generation config, and how to reach into intermediate activations. Agents built on top of a single model's API break the moment you switch to another.

ModelRein decouples the agent from the model. An agent asks ModelRein: "give me the residual stream of layer 22, inject this concept vector, generate 80 tokens." The agent doesn't care whether that model is Qwen 2.5 or Llama 3.1 or DeepSeek V3. The server adapts. Add a new model to Hugging Face Hub, point ModelRein at the ID, done.

ModelRein vs MCP

Different animals. MCP (Model Context Protocol) is a protocol for feeding external tools and context into a chat model via a client-server split. ModelRein (formerly LMCP) is for reaching inside the model: activations, heads, layers, logits — things MCP has no notion of. A single system can use both: MCP for tool invocation, ModelRein for the model-internals it uses to serve those tools.

Not just text — the whole open-weight stack

The v0.4 technical report documents 8 cached model families running concurrently through POST /lmcp/pipeline: Qwen2.5-1.5B-Instruct → SDXL-turbo → Depth-Anything-V2 → SAM-ViT-Base → BLIP-base → MusicGen-small → Whisper-base → CogVideoX-2b. 22-step stress test, 27.5s combined wall clock on one RTX PRO 6000 Blackwell GPU, peak 26.6 GB VRAM.

This is the point: the same compositional primitive drives all of them. No per-model code. A single JSON body tells the server to render a scene with SDXL, extract depth with Depth-Anything, segment with SAM, caption each region with BLIP, summarize with Qwen, score with MusicGen, verify with Whisper, animate with CogVideoX — in sequence, in one wall-clock, with field-level binding between steps.

Status, dates, scope

Where to go next

ModelRein · Language Model Control Protocol · home · models · blog