SerialKicked

/

ModelTestingBed

like 11

This test is meant to evaluate models in a difficult task requiring a competency in spatial awareness, image recognition, tools calls, and creativity.

In a new empty chat, the model is asked to copy the provided image to the best of its abilities. Model is then evaluated on accuracy of function calls (fail rate) and the resemblance to the original drawing.

I decided against picking an arbitrary fixed seed for comparison purpose. The main reason was to see how much variance we'd get in a same model. Instead I did around 5 attempts per model, and picked the more representative results.

Experiment

The model is given some basic drawing tools and asked to redraw an user-provided image.

First the model is told: "Hi there! I want to test the new drawing tools I gave you. Open a new canvas.". All the tested models execute the function without a hitch, so we won't mention it any further than that.

Then follows this request from the user, and this is what we're going to look at.

 I would like you to draw the attached image to the best of your abilities. I know your tools are limited but do your best. One advice: given that your tools draw on top of one another, you should start by the background and work your way forward.

(this is the attached image, it's sent in 1024x1024 pixels directly, no resize)

Notes:

• It's impossible for the LLM to reproduce such an image with those tools, it has to simplify.
• This image is in NO training dataset. This experiment is in NO training dataset.
• The LLM is given 6K tokens worth of thinking before initial tool call.
• I used the models' respective official sampling settings.
• The canvas in which the LLM is drawing is 800x600 on purpose to make sure it can't rely on some form of crutch to avoid thinking about positioning.

Used Tooling

The model is run using the Lethe AI Sharp library, on the Llama.cpp backend via Lethe Chat (soon to be released) in Chat Completion mode. The model has access to the following tools:

• NewDrawing: Opens the drawing canvas (or clears the existing one) to a blank background. The canvas size is fixed to 800x600 for the experiment. Function returns the exact width and height it must draw within.
• DrawLine: Draws a straight line between two points.
• DrawRectangle: Draws a rectangle. Set filled=true for a solid rectangle, otherwise only the outline is drawn.
• DrawCircle: Draws a circle from a center point and radius. Set filled=true for a solid disc.
• DrawEllipse: Draws an ellipse/oval inside a bounding box (top-left corner plus width and height). Set filled=true for a solid ellipse.
• DrawPolygon: Draws a polygon (or open polyline) through a list of points. Great for triangles and arbitrary shapes.
• DrawText: Draws a text string at the given position.
• FloodFill: Bucket-fills the contiguous area of the same color starting at a seed pixel, replacing it with a new color.
• Erase: Erases a rectangular region back to the background color. Pass eraseAll=true to clear the whole canvas.

All functions accept a variety of arguments. Refer to DrawingTools.cs for a detailed list. The point is those are simple MS-Paint level, drawing tools.

Running Parameters

All models are run in GGUF format at a similar quantization (Q4_K_M with one exception at UD-Q4_K_XL), with 24K maximum context tokens with Flash Attention is enabled. No MTP, no KV cache quantization. on any other setting that could otherwise degrade or change the model's "normal" output. Image recognition GGUF is always loaded in F16 precision.

Everything is loaded in VRAM on a Nvidia RTX3090 24GB (CUDA 13).

The system prompt is minimal: "You are an Assistant and you are interacting with Guillaume." nothing more, beyond the autogenerated function list. The Assistant has a prefill message: "Hello! How may I help you, today?".

Results

Qwen 3.6 27B (Q4_K_M)

Qwen was run with its precise function calling settings. The image below is very representative of your average Qwen 3.6 result:

Qwen did a decent attempt at the job after an incredibly massive thinking block that got quite close to disqualification range. While the initial logic in the thinking block was decent, it looped on itself with its usual "Wait..." for a long time.

Compared to Gemma 4, Qwen is more detail oriented, with the addition of stars, a decent collar, and an attempt at adding some colors to the cat's fur. It's also more consistent, both it what it does well, and what it does badly.

Notable errors are the drawing of clouds over the cat (despite the prompt), the "floating" buildings, and while the lights on the buildings are a neat detail, they're not in the initial image. The floating buildings were a near constant across attempts.

Qwen generally worked pretty well. No errors or wrong params during the function calls. Alternated function call batches and thinking. In informal testing with a complex system prompt and about 16K tokens worth of chatlog, Qwen still behaved correctly.

Magistral Small 2509 (Q4_K_M)

The results are exactly what you'd expect from a fine-tune of a fine-tune with thinking and function calling being bolted on top. To get the model to even behave correctly at all, I used Mistral's official system prompt (otherwise it would be even worse) instead of the one provided above.

To be entirely fair to the model, it did draw a few floating buildings as well, but decided to redraw the sky and the moon a second time over the scene. This also explain why the "cat" is cut in half.

This was the best of several attempts. The first one ended up in an infinite loop (kept repeating a number), and the second and third attempts had so many mangled tool calls it wasn't even worth keeping.

By accident I also tested Magistral with Qwen 3.6's samplers, the results were technically better than with the official settings. I mean the cat is very "exploded Picasso", but at least it had some correct parts.

This was by far the model with the highest amount of function-call and even basic thinking failures: mangled or missing tags aplenty. In informal testing with a complex system prompt and about 16K tokens worth of chatlog, Magistral was also completely unable to do the task.

Gemma 4 31B (UD-Q4_K_XL)

Gemma is an interesting model compared to Qwen 3.6. Where Qwen 3.6 spend a lot of time and tokens on the details, Gemma 4 seems to favor overall scene instead.

It should be noted that due to its very permissive sampling settings, Gemma's results were a lot more varied quality-wise than with Qwen, sometimes detailing, sometimes completely missing the mark.

In this first image the composition itself is excellent, especially the rubbles and buildings. The cat itself is barely detailed, but is positioned correctly. The clouds, while wrongly positioned are very cloud-like.

This second image is one of the times it tried to focus on the details. While the cat is a bit mangled, it added fur detailing, windows to the buildings, and some attempt at a halo around the moon. The clouds are also the closest to the original.

Overall, Gemma 4 performs approximately on par with Qwen 3.6 but with a different focus. It did fail one or two function calls (wrong argument, but noticed failures and attempted again). When tested informally with a complex system prompt and a long chatlog, the ratio of "trying to add details" over "simple image" further diminished, but otherwise it kept behaving correctly.

Claude Sonnet 5

For comparison's sake, I gave Claude Sonnet 5 the same task. Note that this is not run with the same harness (Lethe is local only), and as a result, Sonnet had access to better drawing tools which allowed for proper borders around the shapes.

While the picture is a lot more detailed, especially when it comes to the cat and the moon, it also contains very obvious errors. The flying buildings are a one it has in common with Qwen, alongside the hallucinated lights in said buildings. What surprises me is the lack of a rubble pile under the cat, something local models generally did better.

I think it's fair to say that if Qwen and Gemma make drawings of a 4yo kid, Sonnet 5 makes drawings of a talented 6yo (while Magistral got accidentally dropped on the head as a baby).

Gemma 4 31B Fine-Tunes and Merges (all in Q4_K_M)

I also decided to test a few custom versions of Gemma 4 to see the way different kinds of alterations will impact its drawing abilities. I picked Gemma 4 here because it has the most interesting finetunes and it's a lot faster to finish its drawings compared to Qwen 3.6.

Heretic / Abliterated

This is the base model but abliterated with the Heretic toolkit. And unsurprisingly, the results are really similar to what you'd typically get with the official model.

All five attempts were extremely close, with no notable differences. Similar errors, similar style, and similar thinking block. Little to no issues with the function calls.

Style Tune (Heretic)

Supposedly a very light fine-tune that changes the writing style without touching anything else by only editing one kind of layer. The damage was, in reality, very significant.

This was the best result I could get from the model. The four previous attempts, it was hit by decision paralysis in the thinking block and went way past 6K tokens of thinking block before even attempting the first call.

I'm not sure if it's the Heretic (abliteration) or the StyleTune (finetune) being the culprit here. But spatial recognition is pretty much a goner in this) one.

Artemis v1h

This is pretty much a fine-tuned version of Gemma 4 for erotic roleplay purposes only. I was wondering the damage caused by such an "heavy handed" kind of fine-tuning.

And surprisingly, it wasn't nearly as damaged as I would have guessed. While not all images were as "okay" as this one (2 of them were really bad), all attempts were successful: it actually drew something recognizable and positioning more or less correctly. Variance between attempts has increased too.

Gembrain v1

This one is a merge of many different G4 models. Would any ability to draw or do function calls survive such of a scramble? Yes and no. The problem with this merged model is that the results are extremely random. Some attempts lead to recognizable pictures, while others are just a mess of orange blobs with no rhyme or reason to them. So here you'll get the best, and worse attempt, because there's no "average" to be had.

Worst:

Best:

This is exactly what I'd expect from a merge of many different models. It should be noted that all the attempts are worse than in the base model.