← All notes and experiments

How can a compressed reasoning model serve several users on a memory-constrained GPU? This investigation follows the work of adding model support to vLLM, then improving kernels, batching and conversation memory.

Published measurements are snapshots of the stated hardware, settings and revisions.

I wanted to serve PrismML's Ternary Bonsai 2 27B to several users from an RTX A4000 with 16 GB of GPU memory. I chose vLLM for its ability to batch work and serve concurrent requests. When I started, vLLM could not run this model, so the first job was to add support for its PQ2 weight format. The next was to make that support fast enough to be useful.

The model stores most of its 27 billion learned numbers using just three choices: −1, 0 and +1, with a shared scale. This saves space, but the software has to calculate with that compressed representation efficiently. This series follows the work from the first working implementation through faster kernels, concurrent serving and a smaller conversation cache.

The final Triton/Gluon implementation reaches 47.7 output tokens per second for one request on the A4000, or 349.4 tokens per second in total across sixteen concurrent requests. Prompt processing reaches 1,238.0 tok/s for a 1,024-token prompt and 1,221.1 tok/s for a 4,096-token prompt. That brings the work back to the original goal: running the model for several users on the 16 GB card.

The code I developed for this project was co-authored with OpenAI Codex & Claude.

For vLLM contributors

This work is relevant to vLLM’s ternary-support request, #33142. It implements PrismML Ternary Bonsai 2 27B in the PQ2_0 GGUF format, including its Hadamard rotations. The tested setup is text-only, on one NVIDIA Ampere GPU (TP = 1). Supporting other ternary formats such as BitNet, Tequila or Sherry would require their own format and model integration.

Start with the code: PQ2 plugin · vLLM fork · installation guide · final results and limitations.

Evidence from the final Triton/Gluon implementation

These are separate serving measurements. The 3090 Ti run uses FP32 recurrent state, INT8 KV, plain decoding and no prompt caching, with 512-token decode prompts and 1,024 output tokens. Its final GSM8K check scores 120/128 at both one and sixteen concurrent requests. The A4000 run’s full configuration and repeat statistics still need documenting; additional tuning made little difference. The tables below include the intermediate concurrency levels.

What belongs where

The smallest reviewable model-support change would live in the GGUF plugin: recognise PQ2_0, load its packed weights and rotation metadata, apply the Hadamard transforms, and connect those operations to the model’s linear layers and embeddings. A first validation could use plain generation with BF16 KV and FP32 recurrent state, before adding the faster kernels and compressed cache. This is a proposed integration sequence from inspecting the code; a minimal patch on stock vLLM has not been built and tested separately.

The useful next question for maintainers is whether PQ2 support should stay in the GGUF plugin, and which core cache or draft-loading changes would be worth separating into focused upstream contributions.

Final checkpoint

These are the final Triton/Gluon serving results. Prefill is the work of reading a prompt; decode is generating the answer. Concurrent decode rates add the output across all requests.

RTX A4000: deployment results

Additional tuning made little difference in these runs. The measurements establish serving throughput on the target card; the earlier cache-capacity estimates still need memory-allocation measurements.

RTX 3090 Ti: final development results

The final development runs use one RTX 3090 Ti at 300 W, FP32 recurrent state, an INT8 KV cache, plain decoding and no prompt caching. Decode uses 512-token prompts, 1,024 generated tokens and a 16-sequence server limit.

Prompt processing reaches 2,297 / 2,173 / 2,015 tok/s for 1,024 / 4,096 / 16,384-token prompts. Time to first token is 98 ms for a 128-token prompt.

On the 128-question GSM8K check, the final implementation scores 120/128 at both one and sixteen concurrent requests.

Chapter 10 explains the final implementation, its measurement settings and the comparisons used to check the changes.

Validation scope

These measurements come from development (RTX 3090 Ti) and deployment (RTX A4000) hardware. Quality validation uses GSM8K (128 questions) and limited GPQA Diamond testing. Memory capacity figures are estimates based on component budgets; actual allocations depend on workload and may differ.

For production deployment, broader evaluation across reasoning tasks and context lengths is recommended, particularly when using BF16 recurrent state.

Choose a reading path

• Follow the engineering story: read chapters 1–10 in order, from model support to the final Triton/Gluon results. The bottleneck moves from arithmetic and data layout to power, attention and memory.
• See where the implementation stands: start with chapter 10.
• Run the model: use the installation guide, a companion to the investigation. The recipe pins the public revisions; benchmark settings are documented separately.
• Understand the implementation: start with GPU concepts and data flow, then browse the 39 recorded component experiments.

Worked examples introduce the arithmetic. Expandable sections keep the code, detailed measurements and historical settings available alongside the main explanation.

Chapters

Chapters

1. A 27B reasoning model in 1.7 bits per weight

   How ternary weights, rotation and a few full-precision tensors preserve the model’s capabilities in less memory

2. From model size to GPU performance

   A byte budget, a bandwidth ceiling for decode, the PQ2_0 block layout and a checklist for an efficient kernel

3. Establishing a baseline and using integer arithmetic

   Establish the starting performance, then improve it with small integers and weights arranged for faster reads

4. Improving single-request generation

   Repeat calculations with real model inputs, recover weights with fewer instructions, and write a program just for one token

5. Batching, kernel performance and power limits

   A trick that loses at batch 1 and wins at batch 8, why promising kernel gains disappeared in serving, and what power and clock measurements revealed

6. Long-context attention: balancing latency and throughput

   Why attention grows with context, and how using different kernels for prefill and decode avoids a latency trade-off

7. Small batches and speculative decoding

   Make small batches efficient, use them to verify draft tokens, then preserve the gain under load and at long context

8. Reducing memory use with an INT8 KV cache

   Remove wasted cache space, fix a memory-overwrite bug, and nearly double saved-history capacity while checking speed and answer quality

9. The CUDA checkpoint: vLLM and llama.cpp

   The earlier matched comparison, CUDA kernel rounds, BF16 recurrent state, speculative measurements and compile-cache findings

10. From hand-written CUDA to Triton and Gluon

    Replace the CUDA fast paths, tune under serving-like power conditions and measure the final gains at one to sixteen requests

Companion guides

• GPU concepts and data flow

  Understand tiles, layouts, numerical checks and the data flow behind the experiments.

• Running the model locally

  Install and launch the model using a pinned recipe, with optional DFlash support.

• Kernel experiments: recorded evidence

  Browse all 39 component-test records, with diagrams and expandable setup and code.

How to read the results

Chapters 1–9 follow earlier implementation stages, including the CUDA/llama.cpp comparison. Chapter 10 gives the final Triton/Gluon results. Compare numbers within the stated workload and revision; gains from different stages do not add together.

The precision settings apply to different parts of the model:

The recurrent state and KV cache are separate. Changing one does not change the other. Intermediate calculations can also use their own precision.

Measurement hardware

Development measurements use one RTX 3090-series GPU at a time. The paired comparisons identify the card, power limit and workload needed to interpret them. The 16 GB A4000 deployment results are reported separately above; its cache capacities remain estimates pending memory-allocation measurements. The series compares software changes within matched runs.

Development setup and deployment target

The development machine has two RTX 3090 Tis and an RTX 3090. They share the Ampere architecture with the A4000, while memory capacity, bandwidth, multiprocessor counts and power limits differ. Shared instructions make the techniques relevant, while the separate A4000 runs check serving throughput on the target card.

Code and sources

PrismML created the model and its formats. I added the initial PQ2 support to the vLLM GGUF plugin, then developed the kernels, serving changes and measurements described here.

• Model: GGUF model card, MLX model card and PrismML's official release page.
• Implementation: my public PQ2 plugin fork and the vLLM fork with DFlash and quantised-context changes.
• Background: GGUF quantisation formats, vLLM's GGUF documentation, A4000 specifications and Ampere tuning guide.