Inkling-FP8-dynamic

Model Overview

• Model Architecture: InklingForConditionalGeneration
  • Input: Text / Image / Audio
  • Output: Text
• Model Optimizations:
  • Weight quantization: FP8
  • Activation quantization: FP8
• Release Date: 2026-07-15
• Version: 1.0
• Model Developers: RedHatAI

This model is a quantized version of thinkingmachines/Inkling, a 975B total / 41B active parameter multimodal Mixture-of-Experts model that accepts text, image, and audio inputs and generates text outputs.

Model Optimizations

This model was obtained by quantizing the weights and activations of thinkingmachines/Inkling to FP8 data type using dynamic per-token quantization, ready for inference with vLLM. This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.

Weights are quantized statically using per-channel FP8 scaling, and activations are quantized dynamically at inference time using per-token scaling. Only the weights and activations of the linear (attention and MoE expert) layers within the language backbone are quantized using LLM Compressor. The vision encoder, audio encoder, token embedding and unembedding layers, normalization layers, biases, MoE routing/gating logic, shared experts, and the model's early dense MLP layers are kept in their original precision.

Deployment

Use with vLLM

This model can be deployed using vLLM.

Note: Inkling model support in vLLM is being added in vllm-project/vllm#48768. This PR is currently a draft and has not yet been merged, so you will need to install vLLM from that pull request's branch (tml-inkling) to run this model until it lands on main.

1. Start the vLLM server:

vllm serve RedHatAI/Inkling-FP8-dynamic \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.90 \
  --limit-mm-per-prompt '{"image": 4, "audio": 1}'

Tip: For text-only workloads, pass --limit-mm-per-prompt '{"image": 0, "audio": 0}' to skip the vision/audio encoder memory allocation and free up GPU memory for a longer context window.

2. Send requests to the server:

from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://<your-server-host>:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

model = "RedHatAI/Inkling-FP8-dynamic"

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = client.chat.completions.create(
    model=model,
    messages=messages,
)

generated_text = outputs.choices[0].message.content
print(generated_text)