vLLM/Recipes
Thinking Machines Lab

thinkingmachines/Inkling-Small

Natively multimodal 276B-parameter MoE from Thinking Machines Lab — 12B active parameters, text/image/audio in, text out, and up to 1M context.

276B total / 12B active Inkling architecture for lower-cost, lower-latency deployment

View on HuggingFace
moe276B / 12B1,048,576 ctxvLLM 0.26.0+multimodal
Guide

Overview

TML Inkling-Small is a natively multimodal 276B-parameter Mixture-of-Experts model from Thinking Machines Lab. It activates 12B parameters per token and uses the same Inkling architecture and vLLM serving configuration as its larger 975B-parameter sibling, while substantially reducing the deployment footprint.

Inkling-Small accepts text, image, and audio inputs and generates text. It supports Inkling's relative-attention and short-convolution serving path, tool and reasoning parsers, and MTP speculative decoding.

Variants

  • NVFP4 (default): Requires at least 180 GB aggregated VRAM. Run W4A4 with TP1 on B300/GB300 or TP2 on B200/GB200. On H200, run W4A16 with TP2; the MoE kernel dequantizes the NVFP4 weights to BF16 on the fly.
  • BF16: Requires at least 600 GB aggregated VRAM. Run TP4 on B300/GB200/GB300 or AMD MI300X/MI325X/MI355X, and TP8 on B200/H200. MI355X TP4 is validated; direct MI300X/MI325X BF16 smoke is pending.
  • MXFP4 (AMD): EmbeddedLLM/Inkling-Small-MXFP4 keeps the full 1M-token context at TP1 on MI355X or TP2 on MI300X. MI355X runs native AITER MXFP4 MoE; MI300X uses the supported Triton on-the-fly MXFP4 emulation path.

Prerequisites

  • Hardware: B300/GB300 TP1 or B200/GB200/H200 TP2 for NVFP4; B300/GB200/GB300/MI300X/MI325X/MI355X TP4 or B200/H200 TP8 for BF16; MI355X TP1 or MI300X/MI325X TP2 for AMD MXFP4.
  • vLLM: A nightly build with Inkling support.
  • Audio: Install the optional vllm[audio] extra only when serving audio inputs.

Client Usage

Launch the NVFP4 checkpoint in W4A4 mode on B300:

export VLLM_USE_V2_MODEL_RUNNER=1
export FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1

vllm serve thinkingmachines/Inkling-Small-NVFP4 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --tensor-parallel-size 1 \
  --kernel-config.enable_flashinfer_autotune=False \
  --trust-remote-code

Select GB300 for W4A4 with TP1, or B200/GB200 for W4A4 with TP2. On H200, the builder emits TP2 and vLLM runs W4A16 with on-the-fly dequantization.

On AMD, select the MXFP4 variant. To fit 1M context, MI300X TP2 and MI355X TP1. MI355X uses --moe-backend aiter; MI300X uses --moe-backend auto because native AITER W4A4 is gated to gfx950, while gfx942 falls back to OCP_MXQuantizationEmulationTritonExperts. Inkling relative attention remains on its model-specific Triton path (INKLING_GFX950_GLUON=0), which was faster than the optional gfx950 Gluon path in the serving benchmark.

Equivalent MI355X launch:

export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export INKLING_GFX950_GLUON=0

vllm serve EmbeddedLLM/Inkling-Small-MXFP4 \
  --tensor-parallel-size 1 \
  --tokenizer-mode inkling \
  --reasoning-parser inkling \
  --tool-call-parser inkling \
  --enable-auto-tool-choice \
  --moe-backend aiter \
  --block-size 128 \
  --trust-remote-code

Query the server with the OpenAI SDK:

from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="thinkingmachines/Inkling-Small-NVFP4",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)
print(response.choices[0].message.content)

References