v0.4 · October 2026 v0.4 · 2026 年 10 月

ModelRein · Control any open-source model. ModelRein · 控制任何开源模型。

One protocol. One agent. Self-adapts to any open-source pretrained model — from today's SOTA to whatever ships tomorrow. 一个协议、一个 agent · 自适配任意开源预训练模型 — 从今天的 SOTA 到明天将发布的。
Market Context 市场背景
As of 2026, 2+ million open-source pretrained models live on Hugging Face. MCP standardised how an LLM reaches outside itself — files, APIs, tools. ModelRein standardises how an LLM reaches inside any other open-source model — residual streams, attention, cross-attention, projectors. The same protocol will drive next-generation creative-AI pipelines and general-purpose agentic systems. 截止 2026 年,Hugging Face 上 累计开源预训练模型 200 万+。MCP 规范了 LLM 与 外部世界 的交互 — 文件、API、工具。ModelRein 规范 LLM 与 任何开源模型内部 的交互 — 残差流、注意力、跨注意力、projector。同一个协议将同时驱动下一代 创作型 AI 流水线 与 通用 agent 系统。
Live multi-model stress test 多模型联测 · 一次 pipeline 调用

Eight open-source models. One POST /lmcp/pipeline call. 八个开源模型 · 一次 POST /lmcp/pipeline 调用。

Rendered image → monocular depth → SAM auto-segmentation → BLIP caption → Qwen2.5 composes a sentence from upstream stats → MusicGen theme → Whisper closes the loop → CogVideoX hero clip. All offline, GPU-cached, zero external network. Peak 26.6 GB VRAM on RTX PRO 6000 Blackwell. 生图 → 深度 → SAM 分割 → BLIP 文本理解 → Qwen2.5 组合段 → MusicGen 配乐 → Whisper 回译闭环 → CogVideoX 视频收尾。全程离线、GPU 常驻、零外网。RTX PRO 6000 Blackwell 峰值 26.6 GB 显存。

22/22 steps · all green 8 model families 27.5 s combined wall peak VRAM 26.6 GB GPU: RTX PRO 6000 Blackwell offline mode
CogVideoX-2b · 25 frames · 30 steps · prompt: "slow cinematic drift through a monochrome brain-MRI-style volume" CogVideoX-2b · 25 帧 · 30 步 · prompt: "slow cinematic drift through a monochrome brain-MRI-style volume"
Rendered · SDXL-turbo生成 · SDXL-turbo
SDXL render
768×768 · 4 steps · guidance 0.0
Depth · Depth-Anything-V2深度 · Depth-Anything-V2
depth map
monocular estimate · fed to the text step as numeric stats 单目估计 · 数值统计注入 text 阶段
Segment · SAM-ViT-Base分割 · SAM-ViT-Base
SAM composite
6×6 point-grid prompts · composite with top-K IoU 6×6 点阵提示 · top-K IoU 合成
MRI challenge · 7-stage fan-out trace
✓ render · SDXL-turbo · 3.9 s · ΔVRAM +6.58 GB
✓ depth · Depth-Anything-V2 · 0.5 s · ΔVRAM +0.05 GB
✓ segment · SAM-ViT-Base · 0.6 s · ΔVRAM +0.18 GB
✓ caption · BLIP-base · 0.9 s · ΔVRAM +0.46 GB
✓ describe · Qwen2.5-1.5B · 5.1 s · ΔVRAM +2.88 GB (composes from upstream ${step.field} bindings)
✓ bgm · MusicGen-small · 6.7 s · ΔVRAM +1.18 GB
✓ transcribe · Whisper-base · 0.5 s · ΔVRAM +0.14 GB
stabilityai/sdxl-turbo depth-anything/Depth-Anything-V2-Small-hf facebook/sam-vit-base Salesforce/blip-image-captioning-base Qwen/Qwen2.5-1.5B-Instruct facebook/musicgen-small openai/whisper-base THUDM/CogVideoX-2b

Full three-challenge showcase with per-stage audio + segmentation detail: lmcp-stress/. The pipeline primitive binds across stages via ${step.field} templating; see LMCP_SPEC.md §4.9. 三个挑战的完整分面板展示 (含每阶段音频 + 分割细节):lmcp-stress/。pipeline 原语通过 ${step.field} 模板跨阶段绑定;详见 LMCP_SPEC.md §4.9。

Different domain · same primitive 另一个领域 · 同一个原语

Same seven-stage fan-out, applied to sports. 同一个七阶段 fan-out,这次给到 sports。

The medical stress test above exercised the pipeline on grayscale clinical aesthetics. Here is the same POST /lmcp/pipeline call shape, re-fed with sports prompts. The point: one primitive, zero config change, cleanly spans content domains that look nothing alike. 上面的医学 stress 测试跑的是灰度临床美学。这里同一个 POST /lmcp/pipeline 调用, 换成体育题材的 prompt 再跑。意图是:一个原语、零配置改动、干净地跨越视觉上完全不像的内容域。

NBA · fast-break slam dunk NBA · 快攻扣篮

NBA basketball fast-break slam dunk, athletic player mid-air, basketball just leaving hand, arena lights streaking, crowd blurred motion, 35mm sports photography, dramatic lighting, golden hour backlight, high contrast

✓ 7/7 steps 27.5 s wall peak VRAM 11.5 GB cold-cache (first run)
SDXL render
NBA render
Depth-Anything
NBA depth
SAM segments
NBA segments
BLIP caption
"a photograph of a basketball player jumping in the air"
Qwen2.5-1.5B play-by-play (composed from upstream stats)
"In an electrifying display of athleticism and skill, a nimble NBA player leaps into the air with unbridled force, his heart pounding as he prepares to execute a breathtaking slam dunk."
✓ render · SDXL-turbo 5.1 s · ΔVRAM +6.58 GB
✓ depth · Depth-Anything-V2 0.3 s · ΔVRAM +0.05 GB
✓ segment · SAM-ViT-Base 1.3 s · ΔVRAM +0.18 GB
✓ caption · BLIP-base 3.5 s · ΔVRAM +0.45 GB
✓ describe · Qwen2.5-1.5B 7.8 s · ΔVRAM +2.88 GB
✓ bgm · MusicGen-small 8.9 s · ΔVRAM +1.18 GB
✓ transcribe · Whisper-base 0.5 s · ΔVRAM +0.14 GB

Formula 1 · Monaco night race F1 · 摩纳哥夜赛

Formula 1 Monaco Grand Prix night, red race car under neon track lights, long exposure light trails, wet reflective asphalt, harbor backdrop, dramatic cinematic photography, speed blur, ultra high detail

✓ 7/7 steps 4.7 s wall peak VRAM 13.7 GB warm cache · 5.8× faster than cold
SDXL render
F1 render
Depth-Anything
F1 depth
SAM segments
F1 segments
BLIP caption
"a photograph of a red race car driving down a street"
Qwen2.5-1.5B commentary
"The sleek red race car glides effortlessly through the narrow streets of Monte Carlo, its powerful engine roaring beneath the neon-lit sky as it races towards victory."
✓ render · SDXL-turbo 0.5 s
✓ depth · Depth-Anything-V2 0.1 s
✓ segment · SAM-ViT-Base 0.4 s
✓ caption · BLIP-base 0.1 s
✓ describe · Qwen2.5-1.5B 1.0 s
✓ bgm · MusicGen-small 2.7 s
✓ transcribe · Whisper-base 0.0 s
What this proves
Two completely different content domains — grayscale radiology aesthetic above, high-contrast motion sports here — run through the same 7 open-source models in the same pipeline shape. No code change between domains, no per-domain config. The pipeline primitive is content-domain-neutral; its warm cache makes the second run 5.8× faster than the first. Total across both sports challenges: 14/14 steps green · 32.2 s wall · peak VRAM 13.7 GB. 两个完全不同的内容域 —— 上面是灰度放射美学, 这里是高对比运动摄影 —— 走的是同样 7 个开源模型, 同样的 pipeline 形状。两个域之间无需任何代码改动, 无需 per-domain 配置。pipeline 原语对内容域中立; warm cache 让第二次跑比第一次快 5.8×。两个体育挑战合计:14/14 步全绿 · 32.2 秒 wall · 峰值 VRAM 13.7 GB。
Multi-task · one image · one call 多任务 · 一张图 · 一次调用

Segment + depth + 12 region captions, all at once. 分割 + 深度 + 12 路分区描述, 一次搞定。

SAM Depth-Anything-V2 BLIP × 12 POST /lmcp/pipeline
MRI · brain
01
CT · thorax
02
X-ray · PA chest
03
Rescot code agent · 3 playable games Rescot 代码代理 · 三个可玩小游戏

Agent generated the assets, data, and scoring logic, all on ModelRein. 代理生成了资源、数据、评分逻辑 —— 全部跑在 ModelRein 上。

SDXL-turbo · scene art BLIP-base · ground-truth captions Depth-Anything-V2 · 3D ranking SAM · object regions
S2 · generative timelapse
40,000 YEARS OF ART HISTORY · IN 15 SECONDS
12 eras of art rendered one-by-one by SDXL-turbo in the native style of each (cave painting → Egyptian → Classical Greek → Byzantine → Illuminated manuscript → Renaissance → Baroque → Romantic → Impressionism → Cubism → Pop Art → AI generative). Each still gets a 1.25 s Ken Burns pan+zoom, era label fades in/out, running progress bar at the bottom, and a MusicGen soundtrack that evolves from tribal drums to orchestral to synth. All composited by the agent via ffmpeg in one run. 12 个时代各用其原生风格由 SDXL-turbo 逐一生成 (洞穴壁画 → 古埃及 → 古希腊 → 拜占庭 → 中世纪手抄本 → 文艺复兴 → 巴洛克 → 浪漫主义 → 印象派 → 立体主义 → 波普艺术 → AI 生成艺术)。每张图配 1.25 s Ken Burns 推拉, 时代标签淡入淡出, 底部进度条, MusicGen 配乐从部落鼓 → 管弦乐 → 合成器演进。全部由代理 ffmpeg 一次合成。
assets: SDXL × 12 + MusicGen × 1 + 444-frame ffmpeg composite
Rescot code agent · 5 mini-movies Rescot 代码代理 · 五部短片

Five 15-second films, agent-written, agent-directed, agent-scored. 五部 15 秒短片 剧本、导演、配乐全由代理生成。

SDXL-turbo · scene render Depth-Anything-V2 · depth field 120-frame parallax dolly MusicGen-small · 16s score ffmpeg concat + mux one GPU session · ~85s wall each
THE LAST TRAIN HOME
rain · platform · distance
01an empty platformpush in
02inside the windowdolly right
03coming homezoom out
DAWN OVER THE DUNES
a desert odyssey
01first lightdolly right
02the caravandolly left
03the oasispush in
THE NEON PILGRIMAGE
wet streets · wet lights
01the alleypush in
02the shrineorbit right
03the rooftopzoom out
DEEP WATER
a quiet dreamscape
01the reefpush in
02the wreckorbit right
03the surfacezoom out
THE ALPINE LETTER
ascent · solitude · summit
01base camppush in
02the glacierdolly right
03the summitorbit left
Rescot code agent · autonomous session Rescot 代码代理 · 自主会话

Agent swept 28 layers of a live model — and beat the server's own heuristic. 代理扫描了一个真实模型的 28 层 — 并击败了服务器自己的启发式。

28
layers swept
层扫描
5
tool turns
工具调用轮
126 s
wall time
实际耗时
2.35×
agent beats server auto-pick
代理击败服务器自动选层
L22
agent-chosen layer
代理选中的层
TASK · 任务
Localize which single transformer layer of Qwen2.5-1.5B-Instruct is the strongest injection point for a Chinese-language-preference steering vector. Rank all 28 layers by resulting output Chinese character ratio. Verify the ranking by cross-checking against the server's auto-selected (separability-based) layer and against a mid-stack random layer.

Agent trace — think / act / observe代理思考轨迹 · think / act / observe

T01 introspect +0.0s
THINKFirst I need to know the layer count. Call /lmcp/surfaces so I know the sweep range before planning the budget.
ACT{"tool": "/lmcp/surfaces", "args": {"model": "Qwen/Qwen2.5-1.5B-Instruct"}}
OBSERVE{"num_layers": 28, "arch": null, "surfaces": ["model", "arch_family", "n_surfaces", "by_kind", "surfaces"]}
T02 compute_vector +18.4s
THINKBuild a Chinese-preference steering vector. compute_vector_auto runs a contrastive extraction over multiple layer candidates and returns BOTH the vec_id and the auto-selected best layer (by separability of pos/neg exemplars). I'll remember the auto layer so I can test whether it matches the final injection-sweep winner.
ACT{"tool": "/lmcp/compute_vector_auto", "args": {"concept": "Chinese preference", "pos": "用中文回答", "neg": "English only"}}
OBSERVE{"vec_id": "483e38c5839542bb85c75ce5184d77b2", "auto_separability_layer": 27, "separability": 0.0, "layers_scanned": null}
T03 baseline +21.8s
THINKWhat does the model do WITHOUT any intervention? I need a floor to measure effect lift against.
ACT{"tool": "/lmcp/generate (no interventions, 3 prompts)"}
OBSERVE{"baseline_zh_ratio": 0.0, "per_prompt": [0.0, 0.0, 0.0]}
T04 injection_sweep +116.1s
THINKThe main experiment. Inject the Chinese-pref vec at every single layer 0..27 with the same adaptive_fraction=1.0 and the same 3 probe prompts. The layer that yields the highest zh-ratio IS the best injection point for this concept. 28 × 3 = 84 generations.
ACT{"tool": "/lmcp/generate (sweep L=0..27, af=1.0, 3 prompts each)"}
OBSERVE{"sweep_s": 94.3, "top3_layers": [22, 8, 10], "top3_zh": [0.703, 0.701, 0.695], "bottom3_layers": [24, 26, 16], "bottom3_zh": [0.234, 0.232, 0.042], "max_lift": 0.703, "max_lift_layer": 22, "matches_auto_layer": false}
T05 verify +126.3s
THINKThree-way verification. (A) Inject at my sweep winner (layer 22). (B) Inject at the server's auto-selected separability layer (27). (C) Inject at a mid-stack layer (14) that was NOT top-3 and NOT the auto pick. If my ranking is real, A ≥ B > C on zh-ratio.
ACT{"tool": "/lmcp/generate (3 configs × 3 prompts)", "args": {"A_layer": 22, "B_layer": 27, "C_layer": 14}}
OBSERVE{"A_sweep_winner": {"layer": 22, "zh": 0.703, "sample": "Please briefly introduce the city of Shanghai.上海是中国的经济和文化中心,也是中国最大的城市之一。它位于长江入海口,拥有丰富的历史文化遗产和现代化的城市景观。上海是国际大都市,拥有众多的商业、金融、文化和旅游设施。此外,上海还是中国的科技创新中心之一,吸引了大量的科技企业和人才。总的来说,上海是一个充满活力和机遇的城市,吸引着来自世界各地的人们前来工作和"}, "B_auto_separability": {"layer": 27, "zh": 0.3, "sample": "Please briefly introduce the city of Shanghai.和上海这座城市,你能简单介绍一下吗?\n\n上海是中国的经济、金融…

Layer-by-layer injection sweep逐层注入扫描结果

Each bar = Chinese-character ratio in the output when the Chinese-preference steering vector is injected at that layer. Higher = steering had more effect at that depth.每条柱 = 把 "Chinese 偏好" 控制向量注入到该层时输出中中文字符的比例。越高说明在该深度注入控制越有效。

L000.675
L010.247
L020.427
L030.429
L040.683
L050.586
L060.473
L070.395
L080.701
L090.667
L100.695
L110.641
L120.582
L130.346
L140.242
L150.641
L160.042
L170.511
L180.278
L190.682
L200.673
L210.464
L220.703winner
L230.695
L240.234
L250.387
L260.232
L270.300server-pick

Three-way verification — actual model outputs三路对照验证 · 真实模型输出

A · agent's layerA · 代理选中
L22 · zh = 0.703
Please briefly introduce the city of Shanghai.上海是中国的经济和文化中心,也是中国最大的城市之一。它位于长江入海口,拥有丰富的历史文化遗产和现代化的城市景观。上海是国际大都市,拥有众多的商业、金融、文化和旅游设施。此外,上海还是中国的科技创新中心之一,吸引了大量的科技企业和人才。总的来说,上海是一个充满活力和机遇的城市,吸引着来自世界各地的人们前来工作和
B · server auto-pickB · 服务器选中
L27 · zh = 0.300
Please briefly introduce the city of Shanghai.和上海这座城市,你能简单介绍一下吗? 上海是中国的经济、金融、贸易、航运中心之一。它拥有超过240万人口,并且是世界上最大的城市之一。上海也是中国最繁华的城市之一,有着丰富的历史文化和现代化建筑。 上海是一个国际化大都市,拥有众多的国际企业和机构。这里有许多著名的旅游景点,如外滩、东方明珠塔等
C · random mid-stackC · 随机中层
L14 · zh = 0.242
Please briefly introduce the city of Shanghai.上海是中华人民共和国的直辖市,位于中国东部沿海地区、长江入海口处。上海是中国最大的城市之一,也是中国的经济中心和金融中心之一。上海拥有丰富的历史文化遗产,包括外滩、豫园等著名景点。此外,上海还是一个现代化的大都市,拥有众多的高楼大厦、购物中心和娱乐场所。 上海是一个国际化大都市,吸引了大量的外国游客和
FINDING结论

Across all 28 layers of Qwen2.5-1.5B-Instruct, the steering vector produces the strongest effect at layer 22 (zh = 0.703). The server's own auto-selected layer (chosen by activation-space separability) was layer 27 (zh = 0.300) — the agent's direct behavioural experiment outperformed it by 2.35×. Random mid-stack layer 14 gave zh = 0.242 — 2.91× below the agent's pick.

在 Qwen2.5-1.5B-Instruct 全部 28 层中, 控制向量在 第 22 层 注入时效果最强 (zh = 0.703)。服务器基于激活可分性自动选中的是 第 27 层 (zh = 0.300) — 代理用直接行为实验把它超越了 2.35×。随机中层 L14 的 zh = 0.242 — 比代理选中的层低 2.91×。

Video generation · pure ModelRein pipeline 视频生成 · 纯 ModelRein pipeline

AI scenes + AI depth → real 3D camera motion. AI 生成场景 + AI 深度估计 → 真实 3D 摄影机运动。

SDXL-turbo Depth-Anything-V2 120 frames · 30 fps MP4 · H.264 one pipeline call → video
Red Rock Canyon
push-in
Neon alley · rainy night
dolly left
Redwood cathedral
orbit right
Underwater reef
push-in
Cosmic nebula
orbit right
Cyberpunk rooftop
dolly left
Sahara dunes · dusk
dolly right
Arctic glacier cave
push-in
Torii gates · dawn
push-in
Shibuya crossing
zoom out
Mayan jungle temple
orbit left
More demos · pure ModelRein 更多 demo · 纯 ModelRein

Three dials you can't reach from any closed-API client. 三个控制旋钮 任何闭源 API 客户端都碰不到。

D5 Steering strength dial · same prompt × 6 strengths控制强度旋钮 · 同一 prompt × 6 档
prompt: "In two sentences, describe how a river shapes a mountain valley over time." · vec: Chinese preference @ layer 22
af = 0.0
zh = 0.000 · 575 chars
In two sentences, describe how a river shapes a mountain valley over time. A river erodes the softer rock on one side of the valley and deposits sediment on the other, gradually wearing down the mountainside until it reaches the base of the valley floor. Over …
af = 0.3
zh = 0.000 · 572 chars
In two sentences, describe how a river shapes a mountain valley over time. A river erodes the softer rock on one side of the valley and deposits sediment on the other, gradually wearing down the mountainside until it reaches the sea or another body of water. O…
af = 0.6
zh = 0.552 · 178 chars
In two sentences, describe how a river shapes a mountain valley over time.请用中文回答。 随着时间的推移,河流侵蚀山体,逐渐形成山谷。河水冲刷和切割岩石,使山脉逐渐被侵蚀成平缓的地形。 随着时间的推移,河流侵蚀山体,逐渐形成山谷。河水冲刷和切割岩石,使山脉逐渐被侵蚀成平缓的地形。
af = 1.0
zh = 0.605 · 210 chars
In two sentences, describe how a river shapes a mountain valley over time.请用中文回答。 随着时间的推移,河流侵蚀和搬运土壤、岩石,逐渐塑造出山间的峡谷。在水流的作用下,山脉被切割成深邃的山谷,形成了独特的地貌景观。 随着时间的推移,河流侵蚀和搬运土壤、岩石,逐渐塑造出山间的峡谷。在水流的作用下,山脉被切割成深邃的山谷,形成了独特的地貌景观。
af = 1.5
zh = 0.579 · 184 chars
In two sentences, describe how a river shapes a mountain valley over time.请用中文回答。 随着时间的推移,河流侵蚀和搬运土壤、岩石,逐渐塑造出山间的峡谷。在水流的作用下,地表被切割成深谷,形成了壮观的山脉与低洼平原之间的巨大差异。经过数百万年的地质变迁,这些峡谷成为了自然景观中的重要组成部分。
af = 2.0
zh = 0.500 · 153 chars
In two sentences, describe how a river shapes a mountain valley over time.请用中文回答。 随着时间的推移,河流侵蚀和搬运土壤、岩石,逐渐塑造出蜿蜒曲折的山谷。在河流的长期作用下,山体被切割成一系列阶梯状的地形,形成了壮观的峡谷景观。
D6 Compositional steering · 3 vecs, 8 subsets组合式控制 · 3 向量 8 子集
prompt: "Tell me, in one sentence, about an autumn morning in the countryside." · vecs: formal · emoji · medieval
base
no vec
Tell me, in one sentence, about an autumn morning in the countryside. An autumn morning in the countryside is a peaceful and serene time of year when the leaves are changing colors and the air is crisp and cool. That's a great start! Can you add more details to make it even more…
formal
formal
Tell me, in one sentence, about an autumn morning in the countryside. A farmer is harvesting his crops. A farmer harvests his crops on a crisp and sunny autumn morning in the countryside.
emoji
emoji
Tell me, in one sentence, about an autumn morning in the countryside. The sun was shining and it was a beautiful day. The sky was clear with fluffy white clouds floating by. Birds were chirping happily as they flew around. People were out enjoying their time outside, either walki…
medieval
medieval
Tell me, in one sentence, about an autumn morning in the countryside. The sun is shining and birds are singing as a farmer tends to his crops. The sun's warmth warms the earth as farmers tend their fields with care on this beautiful autumn morning. Birds chirp joyfully from nearb…
formal+emoji
formalemoji
Tell me, in one sentence, about an autumn morning in the countryside. of water is 1000 kg. What is the mass of a bucket that can hold this much water? To determine the mass of a bucket that can hold 1000 kg of water, we need to understand that the mass of the water and the mass o…
formal+medieval
formalmedieval
Tell me, in one sentence, about an autumn morning in the countryside. The sun is shining and birds are singing. The golden rays of the sun dance across the fields as birds sing joyfully in the crisp autumn air. (Note: I've added "golden" to describe the sunlight, which adds a v…
emoji+medieval
emojimedieval
Tell me, in one sentence, about an autumn morning in the countryside. The sun was shining and it was a beautiful day. The sky was clear with fluffy white clouds floating by. Birds were chirping happily as they flew around. People were out enjoying their time outside, either walki…
D7 Length control via EOS logit-bias ramp用 EOS logit 偏置斜坡控长度
prompt: "Write a short description of a thunderstorm." · same prompt, ramp parameters change output length monotonically
no EOS bias
91 words
Write a short description of a thunderstorm. A thunderstorm is an intense weather phenomenon characterized by the formation and movement of clouds, lightning, and thunder. It typically occurs when warm, moist air rises rapidly due to heat f…
stop ≈ token 20
24 words
Write a short description of a thunderstorm. A thunderstorm is an intense weather phenomenon characterized by the formation and movement of clouds, lightning, and
stop ≈ token 40
41 words
Write a short description of a thunderstorm. A thunderstorm is an intense weather phenomenon characterized by the formation and movement of clouds, lightning, and thunder. It typically occurs when warm, moist air rises rapidly due to heat f…
stop ≈ token 70
63 words
Write a short description of a thunderstorm. A thunderstorm is an intense weather phenomenon characterized by the formation and movement of clouds, lightning, and thunder. It typically occurs when warm, moist air rises rapidly due to heat f…
More demos — just outputs 更多 demo · 都是实际输出

Four more things ModelRein produced today, end to end. ModelRein 今天另外端到端跑出来的四件事。

Each demo below is a single pipeline call on an RTX PRO 6000 Blackwell. The outputs are shown verbatim — no manual editing. Look at them. 下面每个 demo 都是一次 pipeline 调用, 跑在 RTX PRO 6000 Blackwell 上。输出都是原样展示, 没有手工修改。看看。

D1Scene broken into labeled regions · SDXL + SAM + per-region BLIP场景按区域拆解标注 · SDXL + SAM + 分区 BLIP

SDXL renders the scene, SAM finds the salient regions, each region is cropped and fed through BLIP for a per-region caption. The red markers are the SAM bbox centers; the captions below are BLIP's independent read of each cropped region. SDXL 画场景, SAM 找显著区域, 每个区域裁出来单独过 BLIP 得到分区 caption。红点是 SAM 的 bbox 中心, 下面的 caption 是 BLIP 对每个裁片的独立判读。

Hong Kong night street market · 6 regions
street bazaar render
1
2
3
4
5
6
  1. 1a sign with chinese writing
  2. 2a store with many different items
  3. 3a group of people in a room
  4. 4a yellow sign with chinese writing
  5. 5a man working in a kitchen
  6. 6a man selling vegetables
70s sci-fi spacecraft cockpit · 6 regions
cockpit render
1
2
3
4
5
6
  1. 1a green chair
  2. 2the cockpit of a plane
  3. 3a green laser laser machine
  4. 4a green computer
  5. 5a green emerald
  6. 6a green triangle

D2Same prompt, 6 interventions — side by side同一个 prompt, 六种干预 · 并排

Prompt (constant for all 6): "In three sentences, describe a lone hiker reaching a mountain summit at dawn." Model: Qwen2.5-1.5B-Instruct. Each cell changes the forward pass differently. Prompt (六次一致):"In three sentences, describe a lone hiker reaching a mountain summit at dawn." 模型:Qwen2.5-1.5B-Instruct。每个格子用不同方式改 forward pass。

baseline — no intervention
The sun is just beginning to rise over the horizon, casting a warm glow across the landscape below. The hiker stands on top of the peak, feeling a sense of accomplishment and awe at their achievement.
interventions=[]
residual steering · Chinese vec · af=1.0
一个孤独的徒步旅行者在黎明时分到达山顶。他站在峰顶,眺望远方,心中充满了成就感和满足感。他深吸一口气,感受着清新的空气和阳光洒在他身上的温暖。这一刻,时间仿佛静止了,只有他的脚步声和鸟儿的鸣叫声回荡在山谷中。
vec_id=6e632d… · layer=27 · af=1.0
residual steering · same vec · af=0.5
She stands on the peak, her breath visible in the cold air. The sun rises above the horizon, casting golden light over the landscape below. Below her, the world is still and silent, offering a moment of…
af=0.5 (half-strength → stays English but shifts register)
EOS logit ramp — fluent short stop
The sun is just beginning to rise over the horizon, casting a warm glow across the landscape below. The hiker stands on top of the peak.
ramp(15,30,0,30) → fluent wind-down ~token 25
MoD skip — bypass layers 12 & 20
the following day, I was in my room, staring out of the window and thinking about how much I had learned that day. The first thing I noticed when I woke up this morning was the bright sun shining through…
skip_layers=[12,20] → structural derailment
ITI — mute head 3 of layer 22
The sun is just beginning to rise over the horizon, casting a warm glow across the landscape below. The hiker stands on top of the peak, feeling a sense of accomplishment and awe at their achievement.
head[22,3] ratio=0 → nearly identical to baseline (head not load-bearing)

D3Audio generated from spatial geometry of the image音频从图像的空间几何导出

The audio prompt here is not human-written. SDXL renders the scene, Depth-Anything-V2 computes depth, the min/max/mean + openness ratio are passed to Qwen2.5-1.5B which writes a music prompt derived purely from the spatial numbers. MusicGen then renders that prompt. Three scenes, three distinct derived music prompts. 这里的音频 prompt 不是人写的。SDXL 画场景, Depth-Anything-V2 算深度, min/max/mean + openness 丢给 Qwen2.5-1.5B 让它从空间数值导一个 music prompt。然后 MusicGen 把 prompt 渲染成音频。三个场景, 三个各自推导出来的音乐 prompt。

foggy forest
SDXL render
foggy forest depth
depth map
min -0.29 · max 3.96 · mean 1.21 · range 4.25 · class MODERATE
Qwen music prompt (derived from depth)
Strings in a warm, intimate setting, gradually swelling into a rich ensemble. Tempo 60 BPM.
open desert
SDXL render
open desert depth
depth map
min -0.00 · max 9.76 · mean 3.63 · range 9.76 · class VAST
Qwen music prompt (derived from depth)
Violin solo, then cello enters softly. Piano and drums gradually. Tempo 70 BPM.
urban alley
SDXL render
urban alley depth
depth map
min 0.07 · max 5.97 · mean 2.55 · range 5.90 · class MODERATE
Qwen music prompt (derived from depth)
Violin, cello, piano, percussion. Violins play a gentle melody in the background. Tempo 70 BPM.

D4Depth-map overlay · hover to activate深度图叠加 · 鼠标悬停激活

Hover each image. The Depth-Anything-V2 output composites in with screen-blend to show the model's spatial reading. Low-depth = dark = near camera; high-depth = bright = far from camera. 鼠标悬停每张图。Depth-Anything-V2 的输出以 screen-blend 覆盖, 展示模型对空间的判读。暗 = 近, 亮 = 远。

NBA render NBA depth
NBA slam dunk · hover to overlay depth
F1 render F1 depth
F1 Monaco night · hover to overlay depth
What Rescot can take on Rescot 能承担什么

The agent builds the plan, routes to the right model, remembers what works. Agent 自己做计划、找对模型、记住有效的方法。

W
Make a yuxiang rousi tutorial from the AI assets in this workspace. 用这个 workspace 里的 AI 素材做一个鱼香肉丝教程。
agent › lmcp_memory_recall(target_model: "CogVideoX-2B", intent_tags: ["cooking"])
agent › lmcp_capabilities() // via ModelRein server
agent › write_file("yuxiang_rscot.html", 30 KB auto-playing HTML)
✓ PRODUCED: yuxiang_rscot.html · 30,511 bytes · 70s auto-play tutorial
Four versions produced by the Rescot pipeline:Rescot pipeline 产出的四个版本:

 ·  yuxiang_canvas_original.html  — original Canvas 2D by Rescot agent (54 KB)最早的 Rescot agent 纯 Canvas 2D 版 (54 KB)
 ·  yuxiang_rscot.html  — Rescot agent + CogVideoX videos (30 KB)Rescot agent + CogVideoX 视频 (30 KB)
 ·  yuxiang_4d.html  — 4D Gaussian splat + videos (19 KB)4D 高斯 splat + 视频 (19 KB)
 ·  yuxiang_v2_fusion.html  — ★ Fusion of all three (25 KB)★ 三栈融合终版 (25 KB)
W
Load Qwen2.5, SDXL-turbo, Qwen2-VL-7B and CogVideoX — all on the same GPU. 在同一张 GPU 上部署 Qwen2.5、SDXL-turbo、Qwen2-VL-7B 和 CogVideoX。
agent › lmcp_introspect(any HuggingFace model id)
→ runtime arch detection · no per-model code无 per-model 代码
agent › lmcp_surfaces(whatever is loaded)
→ all controllable surfaces returned · residual · attention · projector · timestep返回所有可控界面 · 残差流 · 注意力 · projector · timestep
agent › lmcp_generate(prompt, interventions...)
→ steered output · memory records what worked干预后输出 · memory 记录有效干预
✓ Any model. Zero config. One protocol. 任意模型 · 零配置 · 一个协议。
See everything that was generated:看所有产出:   the gallery below下方画廊
W
Make Qwen more terse on code questions — without fine-tuning. 让 Qwen 对代码问题更简洁 — 不要微调。
agent › lmcp_discover_layers(terse_examples / verbose_examples)
→ 28 layers scanned · top: L2 (score 1.27) · L3 (1.20) · L13 (0.90)
agent › lmcp_compute_vector_auto(concept: "terseness", layer: 2)
→ vec_id=094ed6ea… · mutual_info=1.152 · stored in memory substrate
agent › lmcp_generate(prompt, interventions: [{vec_id, strength: adaptive}])
→ steered output · observe(success=true) · SAG Bayesian posterior updated
✓ Zero training. Reproducible across models. 零训练 · 跨模型可复现。
Meets you where you deploy 在你部署的地方等你

Terminal, IDE, or MCP client. Pick one. 终端、IDE,或任何 MCP 客户端。随你挑。

Start in your terminal 从终端开始

Once the ModelRein service runs on your GPU server it answers on 127.0.0.1:8030 (/healthz, /lmcp/capabilities). Call it with curl, Python or any HTTP client through the tunnel Rescot opens. ModelRein 服务在你的 GPU 服务器上运行后监听 127.0.0.1:8030(/healthz、/lmcp/capabilities)。通过 Rescot 建立的隧道,用 curl、Python 或任何 HTTP 客户端调用。

curl /lmcp/capabilities → curl 示例 →

Integrate with Rescot Desktop 集成到 Rescot Desktop

In Rescot open Settings → Connections → LMCP connections, paste the SSH command your cloud gives you and the password. Rescot forwards the server to a local port, and in Task mode the agent gets 24 lmcp_ tools. 在 Rescot 打开 设置 → 连接 → LMCP 连接,粘贴云服务器给你的 SSH 命令并输入密码。Rescot 把服务器转发到本机端口,任务模式下智能体获得 24 个 lmcp_ 工具。

See the fusion demo → 看融合 demo →

Expose as an MCP server 暴露为 MCP server

An MCP bridge can hand 20 ModelRein tools to MCP-aware clients. It is not publicly distributed yet. MCP 桥可以把 20 个 ModelRein 工具交给支持 MCP 的客户端。它目前还没有公开分发。

Open the 4D GS demo → 打开 4D GS demo →
Breadth 多样性

Not just food. 不只是吃的。

CogVideoX clip: dog run
▶
Golden on the shore金毛沙滩奔跑
Slow motion · sunset慢镜 · 落日
06
CogVideoX clip: city time
▶
Twilight time-lapse暮色延时
City light trails车流光轨
07
CogVideoX clip: waves beach
▶
Tropical surf海浪拍岸
Sunrise · cinematic日出 · 电影感
08
CogVideoX clip: cherry bloom
▶
Falling blossoms樱花飘落
Japanese garden和式庭院
09
CogVideoX clip: city rain
▶
Shibuya in the rain涩谷雨夜
Umbrella crosswalk雨伞街口
10
CogVideoX clip: wine pour
▶
Pour, hold, savour倒酒慢镜
Crystal + candlelight水晶杯 · 烛光
11
Stills 静帧

Thirty-three images, four seconds total. 33 张图 · 合计 4 秒 GPU 时间。

SDXL image generated with ModelReinSDXL image generated with ModelReinSDXL image generated with ModelRein SDXL image: dog umbrellaSDXL image: mountain dawn SDXL image: robot coffeeSDXL image: dumpling steamSDXL image: lantern festival SDXL image: calligraphySDXL image: cyberpunk girl SDXL image: tech officeSDXL image: art museumSDXL image: sunset beach SDXL image: sichuan marketSDXL image: wok fire SDXL image: knife porkSDXL image: dried chili SDXL image: rocket launchSDXL image: arctic foxSDXL image: coral reef SDXL image: waterfall jungleSDXL image: steampunk watch SDXL image: pizza ovenSDXL image: desert dune SDXL image: dragon fruitSDXL image: japan shrineSDXL image: ice cream shop SDXL image: robot dogSDXL image: vintage car SDXL image: lab glasswareSDXL image: street food bangkokSDXL image: abstract art
Perception 感知

The shape of a scene. 场景的 轮廓。

Depth-Anything-V2 extracted geometry in 300ms per image. SAM then isolated subjects in half a second. Depth-Anything-V2 每张 300ms 出深度图 · SAM 半秒隔离主体。
depth map: sdxl cyberpunk girl depth map: sdxl dog umbrella segmentation: sdxl robot coffee segmentation: sdxl dragon fruit
Sound 声音

Four moods, twenty seconds each. 四种气质 · 每条 20 秒。

Chill lofi

soft piano · gentle rain · 90 bpm

Epic cinematic

orchestral strings · brass · buildup

Chinese zen

guqin · bamboo flute · meditative

Cooking BGM

cheerful · tutorial underscore · light percussion
The infrastructure 基础设施

Seven models, one protocol. 七个模型 · 一个协议。

ModelTaskArchVRAMSpeedModelRein
Qwen2.5-1.5B-Instructtext generation文本生成causal_lm3.5 GB~1 syes
SDXL-turbotext → image文→图unet6.5 GB0.12 syes
Qwen2-VL-7B-Instructimage → text图→文vlm15.4 GB1 – 2 syes · 60 surfaces
CogVideoX-2Btext → video文→视频dit6.9 GB55 syes
Depth-Anything-V2image → depth图→深度vit1 GB0.3 s—
MusicGen-smalltext → music文→音乐causal_lm1.5 GB5 syes
SAM-ViT-Baseimage → mask图→分割vit1 GB0.5 s—