ModelRein
Supported models

ModelRein: any open-source model, one protocol

ModelRein's six architecture adapters cover every major open-weight model family. Pick any model from Hugging Face Hub, point the server at its repo ID, and all 27 ModelRein primitives (sampling, logit bias, residual-stream steering, attention-head masking, layer routing, cross-model pipelines) work against it. Below: the current support matrix.

6
adapters
27
primitives
40+
model families
0
per-model code
0
closed APIs
0
cloud dependencies

Causal language modelscausal_lm

Transformer decoder-only LLMs. Full ModelRein surface: sampling, logit bias, EOS ramps, residual steering (any layer), attention head masking (any layer, any head), layer routing (skip / repeat), introspection.

FamilySizesVendorModelRein capabilitiesStatus
Qwen 2.5 / Qwen 30.5B – 72BAlibabageneratesteerhead_masklayer_routelogit_biasvalidated
Llama 3 / 3.1 / 3.2 / 3.31B – 405BMetageneratesteerhead_masklayer_routelogit_biasvalidated
DeepSeek V3 / R17B – 685BDeepSeek AIgeneratesteerhead_masklayer_routevalidated
Mistral / Mixtral7B – 8×22BMistral AIgeneratesteerhead_masklayer_routevalidated
Gemma 2 / Gemma 32B – 27BGooglegeneratesteerhead_masklayer_routevalidated
GLM 49B – 32BZhipu AIgeneratesteerhead_maskvalidated
Phi 3 / 3.5 / 41.3B – 14BMicrosoftgeneratesteerhead_maskvalidated
Yi 1.56B – 34B01.AIgeneratesteerlayer_routetested
Falcon 31B – 10BTIIgeneratesteertested
OLMo 21B – 13BAI2generatesteertested
Any AutoModelForCausalLManyHugging Facegeneratesteerhead_masklayer_routeby introspection

Image generation — UNet backbonesunet

Stable Diffusion lineage. ModelRein exposes residual edits at each UNet block, cross-attention head control, timestep interventions, VAE latent probes.

ModelResolutionVendorModelRein capabilitiesStatus
Stable Diffusion 1.5512Runwayimagexattn_masktimestepvalidated
Stable Diffusion 2.x768Stability AIimagexattn_masktimestepvalidated

Image generation — DiT backbonesdit

Diffusion transformers. Fastest adapter: SDXL-turbo generates a 1024×576 scene in ~0.6 s on an RTX PRO 6000, drives the parallax videos and game-sprite pipeline.

ModelResolutionVendorModelRein capabilitiesStatus
SDXL-turbo1024Stability AIimagexattn_masktimestepvalidated
SDXL base / refiner1024Stability AIimagexattn_maskvalidated
PixArt-Sigma1024 / 2KHuawei Noah's Arkimagexattn_maskvalidated
FLUX.1-dev / schnell1024Black Forest Labsimagexattn_masktested

Vision + Languagevlm

Vision-language models and captioners. Used in the compositional pipeline to describe SDXL renders, SAM regions, and video keyframes.

ModelSizesVendorModelRein capabilitiesStatus
Qwen2-VL2B / 7B / 72BAlibabacaptiongenerateprojectorvalidated
LLaVA 1.5 / 1.67B / 13B / 34BMicrosoft/UWcaptiongeneratevalidated
BLIP / BLIP-2base / largeSalesforcecaptionvalidated · heavily used
InternVL 2.51B – 78BShanghai AI Labcaptiongeneratetested

Depth / Segmentation / Perception

Not language models, but ModelRein wraps them as pipeline tasks so a single call can chain render → depth → segment → caption.

ModelTaskVendorCapabilitiesStatus
Depth-Anything-V2monocular depthByteDance / TikTokdepthvalidated · heavily used
SAM / SAM-ViT-BasesegmentationMeta AIsegmentvalidated · heavily used
RMBG-2.0 / 1.4background removalBria AImatteoptional

Audio / Musicaudio_diffusion

ModelTaskVendorCapabilitiesStatus
MusicGen small / mediumtext → musicMetamusicconditioningvalidated · heavily used
Whisper base / largespeech → textOpenAI (open-weight)transcribevalidated
AudioLDM 2text → audioUMusic / HKUSTaudiotested
Stable Audio Opentext → stemsStability AIaudiotested

Video generationvideo_dit

ModelResolutionVendorCapabilitiesStatus
CogVideoX-2b / 5b720×480 · 6sZhipu AI / THUDMvideotimestepvalidated
Zeroscope v2576×320cerspensevideotested
ModelScope T2V256×256Damo-ViLabvideotested

How to add a new model

If the model loads via AutoModelForCausalLM.from_pretrained(repo_id) or DiffusionPipeline.from_pretrained(repo_id), no new code is required — ModelRein introspects it at load time. Just add the repo ID to the server's allow-list and all primitives work. For architectures that don't match any of the six adapters, implement the adapter interface (one Python file, ~150 lines) and open a PR against the reference server.

ModelRein · Supported Models · home · about · blog