inference-optimization/Ornith-1.0-9B-FP8-Dynamic

Model Overview

• Model Architecture: Qwen3_5ForCausalLM
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Activation quantization: FP8
  • Weight quantization: FP8
• Intended Use Cases: Intended for commercial and research use. Similarly to the base model, this quantized version is intended for agentic coding tasks, terminal-based workflows, and tool-calling applications.
• Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws).
• Version: 1.0
• Model Developers: inference-optimization (Neural Magic / RedHat)

Model Optimizations

This model was obtained by quantizing activations and weights of deepreinforce-ai/Ornith-1.0-9B to FP8 data type. This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x). Weight quantization also reduces disk size requirements by approximately 50%.

Only weights and activations of the linear operators within transformers blocks are quantized. Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme. The llm-compressor library is used for quantization.

Deployment

Using Transformers

This model can be deployed using the Hugging Face Transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "inference-optimization/Ornith-1.0-9B-FP8-Dynamic"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

prompt = "Explain quantum computing in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.6,
        do_sample=True,
        top_p=0.95,
        top_k=20
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using vLLM (Note: Compatibility Issue)

Important: Due to architectural differences between the original multimodal Ornith model and the text-only quantized version, this model currently has compatibility issues with vLLM when the model architecture expects vision components. We recommend using the Transformers library for deployment until this is resolved.

Model Details

• Base Model: deepreinforce-ai/Ornith-1.0-9B (built on Qwen 3.5 and Gemma 4)
• Model Size: ~9B parameters
• Disk Size: ~11 GB (down from ~19 GB in bf16)
• Context Length: Up to 262,144 tokens (reduced context recommended for faster inference)
• Reasoning: Supports <think>...</think> blocks for chain-of-thought reasoning
• Tool Calling: XML-based function calls compatible with OpenAI format

Performance

This FP8 quantized model provides:

• ~50% memory reduction compared to the bf16 base model
• ~2x faster inference for matrix multiplication operations
• Maintained accuracy for reasoning and coding tasks

Limitations

• The quantization process removes multimodal (vision) capabilities present in the base model
• This is a text-only model suitable for language tasks, coding, and reasoning
• vLLM compatibility is currently limited due to vision config expectations

Creation

Creation details

This model was created with llm-compressor by running the code snippet below.

from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "deepreinforce-ai/Ornith-1.0-9B"

# Load model.
model = AutoModelForCausalLM.from_pretrained(
  MODEL_ID,
  low_cpu_mem_usage=True,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
  targets=["Linear"],
  scheme="FP8_DYNAMIC",
  ignore=["lm_head"],
)

# Apply quantization.
oneshot(model=model, recipe=recipe)

# Save to disk in compressed-tensors format.
SAVE_DIR = "Ornith-1.0-9B-FP8-Dynamic"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)

Citation

If you use this model, please cite:

@software{ornith_fp8_dynamic,
  author = {inference-optimization},
  title = {Ornith-1.0-9B-FP8-Dynamic},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/inference-optimization/Ornith-1.0-9B-FP8-Dynamic}
}

Original base model:

@software{ornith,
  author = {DeepReinforce AI},
  title = {Ornith-1.0-9B},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B}
}