ManniX PRO

The package_courier test harness does it; it's the only thing an operator has to find, the right floor_tps for that model & quantization.
Looping the test harness at different floor_tps, find the sweet spot.

Fair question — I ran the cells to answer it directly, including the overload run you suggested.

Same fixed workload (10 packages × 55 steps, fixed seed, same model/GPU), both gates 100%: the pre-P7 gate finished with 9 workers admitted, P7 with 8 — essentially equal concurrency — yet P7 delivered the identical workload in 1919 s vs 2187 s (−12% wall) with p50 delivery 764 s vs 1350 s (−43%). If the gain were stricter throttling, equal-or-fewer workers would mean a longer wall; it's shorter because admitted workers spent far less time degraded (time-under-floor −52%, deep sub-floor −83%). Less thrash, not less work.

Overload cell (16 packages, up to 16 workers, ~2× past this GPU's knee): neither gate drops tasks — both score 100% (legacy: wall 3029 s, p50 769 s, workers →10, 68% of samples under the 15 tok/s floor; P7: wall 2999 s, p50 737 s, workers →11, 71% under floor). At full saturation an admission policy can't manufacture throughput — the GPU is the ceiling and both gates pin it; the enforced-admission 429 backstop keeps sessions alive either way, so work gets slow, not lost. I didn't rig a hard deadline to force drops — degradation is graceful by design on both sides. P7's value is the mid-load regime: honoring a tps floor while scheduling headroom exists (that's the −43% p50), plus the idle-gap estimate that treats tool-call idle as free capacity in /capacity projections (projected_idle_estimate: true) — the opposite of counting it as busy.

Raw work-reports and logs available if you want to dig.

https://huggingface.co/ManniX-ITA/opencoti-llamafile/blob/main/bench/courier-admission/

I use HE+ mostly to verify if the quant quality, if it doesn't collapse the quant is good, or to evaluate the loss at 2/3-bit.
Added MPE cause it's another canary and it has a bit more discriminative signal (depends on the mode capabilities with other coding languages than Python).
I don't use LCB because it takes hours to run; with 20-27 quants per model it's a NO GO, while usually HE+/MPE are taking 3-7 minutes each.

I cannot say:

• there's not much overlap at ISO disk size, I should compare Qwen3.5-9B F16 vs v7-coder Q6_K: not really much use for it, nobody would use it at F16
• I didn't run the evaluations for Qwen3.5-9B other than Q6_K and there's no matching quant even at Q8_0 with v7-coder
• they serve different domains; Qwen3.5-9B is an excellent Coder but the thinking is awful. Different tasks. Qwen3.5-9B can be used for simple tasks, completion or as coding assistant, not for agentic coding or any too complex thinking. It has a surprisingly good knowledge of Science but I think the high GPQA score comes more from a training bias than real reasoning capabilities. It scores very good on HE/HE+/MPE, but the LiveCodeBench score is revealing: only 58% vs v7-coder 95-98% and low AIME. Still, it's a 9B model and in that class you can hardly do more.

Yes indeed, I have my canonical OmniMergeKit evaluations.
If you check the v7-coder GGUF repo you can see the scores of Gemma-4 A4B which it's a 26B.
The balanced 184e version was awful compared to that and even miserably beaten in more than a few by the (excellent) Qwen3.5-9B. The speed advantage of A3B wasn't worth at all the 3x times VRAM usage of 27B vs 9B.

Both A4B and A3B untargeted prunes were quite disappointing; not excelling in anything, dipping low in some.
In general scoring at the same level or worse than an unpruned model with the same amount of parameters.
Not worth it.

I can track the context size and context / concurrent agents ratio and plot over time the with/without PolyKV ratio and context size with and without.

Right now I'm validating the fine-tuned model with the harness, once done will make some runs comparing all the uses cases with and without polykv, pooled and static workers.

The drawback of the pruning is clearly state in the verbosity/rumination chapter:

https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder#verbosity--rumination-length-breakdown-per-eval

What you can see in the table is that the routing, despite the pruning, is not really impacting the target usage.
The biggest issue was the collapse of IFEval, which means generically a loss of instruction following capabilities.
But the top-10 recovered it and at the end it works even better than the full 256e on AIME and MATH.
AIME and GSM8K even are scoring slightly higher with the pruned version.

It's a very different behavior from the Gemma-4 A4B pruning; it had much more less runaway both in the base and pruned but the length and rumination ballooned for almost everything.
But the cost is really low, despite that it's still very fast.
The issue with A4B is that it ends up looping which is a base behavior which is inherited and augmented by the pruning.

I will soon try with both REAM instead of REAP as suggested.

Yes, I had a look at the overlap of both; there's not much because the knowledge is very very spread over the experts.
Gemma-4 A4B has only one expert dedicated to coding fully, all the others are scattered over 128e.
Adding additional targeting ends up reducing a lot the good experts selection.
There's a huge base of mandatory experts that must be included for specific reasons and excluding them destroys the model; like those for multi-lingual and agentic end of turn.
For Qwen A3B is a bit different because there are 256 experts instead of 128e.
There's more 'choice' but the pruning hits really hard everything, scores goes down all-around: targeting with a lower weight also MPE partially recovered.
But despite that there was no way to increase MPE score without scarifying too much everything else.

I'm very busy with many other things right now but I'm considering it.
Working on the same type of harness to excercise Agent2Agent protocol that simulates a more "real" usage.
That does have 2 modes with an heavy template and a light one.

Do you mean as context of the agents increases a projection of what would be the context without PolyKV and track the ratio?

I really can't measure all use cases. The pruning is targeted for a good reason!
I have done the 184e pruning non targeted: it was terrible on almost everything.
The coding eval coverage is pretty good and in my experience covers pretty well the domains; python with HE+, multiple languages with MPE and code reasoning with LCB.

As of the targeting, it pays off to target LCB. Adding more specific targeting it's only hurting.
Once he prune is targeted for LCB it covers already almost all the experts for coding.
Adding more coding targeting is going to include low-value experts that only contains residuals in spite of the strong ones already selected by LCB.

The scores with top-8 are much worse, that's why it's needed top-10.
I tried to fix the routing with all other solutions I have, it gets worse in a way or another.

You need to think more "out of the box" and have a boarder vision.
The evaluation harness is limited and exercise a single use case.

The only number that matters is the reduction at P, Step 0.
If you consider also S per step, it's not going to tell you anything really valuable.

This is a fixed, mocked eval harness that runs a static number of steps.
It has its own purpose which is exercising and validating the API and the engine.
Doesn't really reflect a real use case; maybe there are similar ones but it's an edge case.

Every agent in the harness have a very small prefix (P=1216) which is most often not true.
Unless you are using agents to do some very basic and trivial things, hardly a real use case (most probably you don't need an agent for this kind of stuff).

In a real use case the P is what matters most of the time; in my claude-hooks consultants skill the P is gigantic: 60-200K to describe the project, provide the tools, the AST, list of the files to inspect, what was done, test results, etc etc.

Then the agent will start its work and you have no idea how many steps will take, how long will be the output, how much thinking will be needed. Ever spawned agent will have, in non synthetic tasks, most likely a very different behavior. So you can't compute and forecast a ratio of the savings due to the shared P and involve S and the steps.

The capacity lever is at P: let's frame a realistic scenario, 128k of "P" plus max 128k for "S".
If you have space for 1M tokens on your GPU and you need to run agents without PolyKV you can run 4 of them.
With PolyKV you can run 7 agents and they will be faster overall plus the context compaction gotcha.

The context compaction is another big win: without PolyKV every agent needs to be instructed to preserve the P and they all need to process its own P. They could easily mess it up and remove critical information or just make mistakes.
With PolyKV the P is static and it's not reprocessed by the agents, doesn't have a chance to get messed up.

You are really wrong when you say that the fleet size is not a lever: it's exactly the opposite.
The floor tps scheduler is a killer feature just like the shared P.
That's why I uploaded so many reports with different models and different floor tps.
Each combo model/GPU needs the right floor tps otherwise it starts crapping out and the wall time will increase, the quality will decrease. That's the value of the test harness as it is configured now.

You can't quite precisely predict the load of every agent, if you choose a static value that's it. You are bound to it.
If you are running multiples instances of any other inference engine without the floor tps scheduler you have either to under or over subscribe the number of agents.
Unless you put on top a complex custom orchestrator which needs to be managed.

Since opencoti-llamafile with the PolyKV API knows the actual load on the GPU it can auto-balance with just the right tps floor value for the model, independently of any other instance running or not running in parallel.
If there's no other load can scale up the workers up or above the available slots, your decision.
In case there's load the floor tps scheduler will restrict the amount of agents that can be spawned automatically.
I still need to improve concurrency and avoid over subscription at start but the logic is simple and should be clear.

Abou the chart: I will probably replace the graph of tps at step as it's redundant to the tps with context and less granular.
But as I said a "cell reduction" graph as you describe I don't think that really make sense.
Have to think a bit more about it on how to frame it.

You can start wiring it now and test yourself.
The new c4 release provide a very rich and sophisticated API for PolyKV, including a floor-tps scheduler that allows automatic workers allocation based on engine capacity.
Should be properly documented but it's just been done and I didn't have time to double check.

There is also an evaluation benchmark test harness, package-courier, and a few benchmark runs.
You can use the test harness as a reference implementation of the API in Langchain/Langgraph.

You can simulate multi-turn tool usage with it and track the gen tps over turns, there's even a specific graph in the report.
Didn't have time yet to test it with more than 10 turns, context per-agent is small at n=10 and the tps increase over turns due to that (you need really a massive context to see the tps decrease and opencoti-llamafile is slower with small context sizes).
If you test it let me know how it goes and if the documentation is clear enough!

The ratio grows with the number of agents since the shared prefix is stored once instead of N times: 6.9× is the measured KV-cell reduction at N=8 agents (with a ~875-token shared prefix and short private suffixes).

Projected onto a fixed KV buffer, that geometry fits about 306 agents pooled vs about 9 naive before exhaustion. The more sub-agents attached to a pool, the closer you get to the prefix/suffix ceiling.
The suffixes always diverge, so there's no memory saving there — the batched-decode gains (~+14% aggregate at N=8) come from the smaller cache footprint, not from suffix sharing.

llama.cpp's built-in cache reuse (--cache-reuse) is a different axis: a slot reusing its own previous prompt across turns. It doesn't share cells across concurrent sequences — that's exactly what PolyKV adds, and the two compose.

I would wait before wiring it in, the mechanics are there but they are raw.
I'm adding a proper API control and Langchain/Langgraph extension, actually a very sophisticated one, which will make it easier to use it and way more powerful.

About your question: the prefix is immutable.

If you want to change the prefix you will have to spawn a new pool and new sub-agents.
There's no problem adding a new mcp server or tool def in the suffix but it will live inside that sub-agent session only.
(Which anyway makes sense as if it was something needed by all sub-agents it would be part of the prefix)

PolyKV is exactly this, sharing an immutable prefix and run a pool of sub-agents that shares it.
This allows parallelizing tasks for a single agent/session very efficiently.

Really depends on the specific use case; the headline is that with DCA you can extend any context and overcome the model limitation.
It's not only useful to get to 1M context, which can get damn slow even with a MoE on an RTX6000, but also to get up to an acceptable range with a model trained for a small context.

There's also the KV-window which it's truly novel, not to be under-estimated considering the crazy GPU and memory prices. A modern system with full PCIe 5.0 x16 and DDR5 can hide behind computation up to 3-4GB, depending on the model attention mechanism, and keep almost full gen tps speed.
Offloading model layers to RAM, the classic llama.cpp method, tanks the decoding speed immediately, no matter if is by a small amount or not.

If you are working with agentic framework coding SharedKVPool is the headline.
The auto-tier is a boot selection algorithm, doesn't have much to do with them; it's an helper (pretty sophisticated with probing and micro-bench at boot) to automatically enable the KV-window in conservative mode and/or select a K/V type and strategy that fits the available VRAM.

The prefix with PolyKV is what is shared between the sessions; it doesn't change. Same quantization for everyone, if it changes it's done only once.
The sessions in the same pool share the same prefix and only the suffix changes, that's what diverges and always.
The win is by definition in prefill and memory usage but also on gen tps (less overhead) and multi-turn re-query.

Made a benchmark and updated the docs:
https://huggingface.co/ManniX-ITA/opencoti-llamafile/blob/main/USAGE.md#34-polykv-sharedkvpool-multi-agent-shared-prefix

It's really a game-changer and a killer-feature for agentic frameworks that can drive it, both in capacity and performances.