0 / 60

Every Pelican That
Ever Rode a Bicycle

Simon Willison asks every new LLM the same question:
Generate an SVG of a pelican riding a bicycle
What started as a quick test became the internet's favorite AI benchmark.

60 models · 58 pelicans · 2024–2026

Source: simonwillison.net

↓ scroll to begin
2024
It started in late 2024. Simon Willison began asking every new LLM the same deceptively simple prompt: "Generate an SVG of a pelican riding a bicycle." The results were... instructive.
Two circles joined by a triangle. The pelican is a grey oval for the body, a circle for the head and has a peak that looks like a yellow banana smile. A wing is hinted at with an upside down curved li

Various Models (Inaugural Post)

There’s a whole lot of buzz around the new Qwen2.5-Coder Series of open source (Apache 2.0 licensed) LLM releases from Alibaba’s Qwen research team. On first impression it looks like the buzz is well deserved. Pelicans on a bicycle. I decided to roll out my own LLM benchmark: how well can different models render an SVG of a pelican riding a...

Read Simon's post →
A really rather good SVG of a California Brown Pelican riding a bicycle
Pixelmator UI showing the SVG with a sidebar showing each of the individual shapes - I have selected

Recraft v3

Recraft V3. Recraft are a generative AI design tool startup based out of London who released their v3 model a few weeks ago. It's currently sat at the top of the Artificial Analysis Image Arena Leaderboard, beating Midjourney and Flux 1.1 pro. The thing that impressed me is that it can generate both raster and vector graphics... and the vector...

Read Simon's post →
Not great at all, description follows

Gemini exp-1121

My benchmark for LLMs: "Generate an SVG of a pelican riding a bicycle". "User might be a kid playing with words" according to Qwen3-4B-Thinking. Say hello to gemini-exp-1121. Google Gemini's Logan Kilpatrick on Twitter: Say hello to gemini-exp-1121! Our latest experimental gemini model, with:

Read Simon's post →
You can tell which bit is the bicycle and which bit is the pelican. It's quite elegant.

QwQ + o1

QwQ: Reflect Deeply on the Boundaries of the Unknown. Brand new openly licensed (Apache 2) model from Alibaba Cloud's Qwen team, this time clearly inspired by OpenAI's work on reasoning in o1. I love the flowery language they use to introduce the new model: Through deep exploration and countless trials, we discovered something profound: when given...

Read Simon's post →
Blue sky, green grass, bicycle looks good, bird riding it is almost recognizable as a pelican
The pelican is now animated - it is pedaling and its wing moves

Llama 3.3

Meta’s new Llama 3.3 70B is a genuinely GPT-4 class Large Language Model that runs on my laptop. New Gemini model: gemini-exp-1206. Google's Jeff Dean: Today’s the one year anniversary of our first Gemini model releases! And it’s never looked better. Check out our newest release, Gemini-exp-1206, in Google AI Studio and the Gemini API!

Read Simon's post →
2025
2025 brought an explosion of new models — and with them, an avalanche of pelicans. Chinese open-weight models entered the chat. Reasoning models learned to think about bicycle geometry. Coding agents tried iterating. The pelicans got better. The bicycles remained tricky.
Hardly recognizable pelican on a bicycle

Phi-4 + DeepSeek V3

DeepSeek are the Chinese AI lab who dropped the best currently available open weights LLM on Christmas day, DeepSeek v3. That model was trained in part using their unreleased R1 “reasoning” model. Today they’ve released R1 itself, along with a whole family of new models derived from that base. microsoft/phi-4. Here's the official release of...

Read Simon's post →
This is not great. The bicycle is a trapezoid. The pelican is very warped and has a orange diamond beak above its head.
The bicycle is better but the pelican is yellow and looks more like a baby chick. Its beak is squashThis one is pleasingly avant-garde. The bicycle does at least have two wheels joined by a frame. The

Gemini 2.0 Flash

Gemini 2.0 is now available to everyone. Big new Gemini 2.0 releases today: Gemini 2.0 Flash-Lite looks particularly interesting: We’ve gotten a lot of positive feedback on the price and speed of 1.5 Flash. We wanted to keep improving quality, while still maintaining cost and speed. So today, we’re introducing 2.0 Flash-Lite, a new model that has...

Read Simon's post →
Grok 3's pelicans are pretty good, though I still slightly prefer Claude 3.5 Sonnet's.

Grok 3

Andrej Karpathy’s initial impressions of Grok 3. Andrej has the most detailed analysis I've seen so far of xAI's Grok 3 release from last night. He runs through a bunch of interesting test prompts, and concludes: As far as a quick vibe check over ~2 hours this morning, Grok 3 + Thinking feels somewhere around the state of the art territory of...

Read Simon's post →
Visit Notes on Google's Gemma 3
A very good attemptA very good attemptVisit Initial impressions of GPT-4.5

Claude 3.7 Sonnet + Gemma 3

Google’s Gemma team released an impressive new model today (under their not-open-source Gemma license). Gemma 3 comes in four sizes—1B, 4B, 12B, and 27B—and while 1B is text-only the larger three models are all multi-modal for vision: GPT-4.5 is out today as a “research preview”—it’s available to OpenAI Pro ($200/month) customers and to developers...

Read Simon's post →
Blue and black wiggly lines looking more like a circuit diagram than a pelican riding a bicycle

OLMo 2 + GPT-4o

mlx-community/OLMo-2-0325-32B-Instruct-4bit (via) OLMo 2 32B claims to be "the first fully-open model (all data, code, weights, and details are freely available) to outperform GPT3.5-Turbo and GPT-4o mini". Thanks to the MLX project here's a recipe that worked for me to run it on my Mac, via my llm-mlx plugin. To install the model: That downloads...

Read Simon's post →
Clearly a bird riding a bicycle. The pedals are in the wrong place and the bird doesn't quite have the majestic beak of a pelican.
A bit better - no pedals, so they cannot be in the wrong place. The bird has a slightly more majesti

o1-pro + GPT-4o

OpenAI platform: o1-pro. OpenAI have a new most-expensive model: o1-pro can now be accessed through their API at a hefty $150/million tokens for input and $600/million tokens for output. That's 10x the price of their o1 and o1-preview models and a full 1,000x times more expensive than their cheapest model, gpt-4o-mini! Aside from that it has...

Read Simon's post →
There's a pelican, and a bicycle, but both of them look disassembled.
Visit GPT-4.1: Three new million token input models from OpenAI, including their cheapest model yetVisit Putting Gemini 2.5 Pro through its paces

DeepSeek-V3

OpenAI introduced three new models this morning: GPT-4.1, GPT-4.1 mini and GPT-4.1 nano. These are API-only models right now, not available through the ChatGPT interface (though you can try them out in OpenAI’s API playground). All three models can handle 1,047,576 tokens of input and 32,768 tokens of output, and all three have a May 31, 2024...

Read Simon's post →
Visit Qwen 3 offers a case study in how to effectively release a model
Described belowDescribed below, againDescribed below

Gemini 2.5 Flash

Alibaba’s Qwen team released the hotly anticipated Qwen 3 model family today. The Qwen models are already some of the best open weight models—Apache 2.0 licensed and with a variety of different capabilities (including vision and audio input/output). Start building with Gemini 2.5 Flash (via) Google Gemini's latest model is Gemini 2.5 Flash,...

Read Simon's post →
See description below

Gemini 2.5 Pro

Gemini 2.5 Pro Preview: even better coding performance. New Gemini 2.5 Pro "Google I/O edition" model, released a few weeks ahead of that annual developer conference. They claim even better frontend coding performance, highlighting their #1 ranking on the WebDev Arena leaderboard, notable because it knocked Claude 3.7 Sonnet from that top spot....

Read Simon's post →
The bicycle has spokes that look like a spider web. The pelican is goofy but recognizable.
The bicycle has too many bits of frame in the wrong direction. The pelican is yellow and weirdly shaThe wheels and pedals are rotating and the pelican is bobbing up and down. This would be a fantastic

Gemini 2.5 (GA)

I presented an invited keynote at the AI Engineer World’s Fair in San Francisco this week. This is my third time speaking at the event—here are my talks from October 2023 and June 2024. My topic this time was “The last six months in LLMs”—originally planned as the last year, but so much has happened that I had to reduce my scope! I’m at...

Read Simon's post →
Description by o3-pro: The image is a playful, minimalist cartoon showing a white bird riding a bicycle. The bird has a simple oval body, a round head with a small black eye, and a yellow beak. Its or
Visit Trying out the new Gemini 2.5 model family

o3-pro + Gemini 2.5 Pro

My benchmark for LLMs: "Generate an SVG of a pelican riding a bicycle". "User might be a kid playing with words" according to Qwen3-4B-Thinking. After many months of previews, Gemini 2.5 Pro and Flash have reached general availability with new, memorable model IDs: gemini-2.5-pro and gemini-2.5-flash. They are joined by a new preview model with an...

Read Simon's post →
Claude Sonnet 4 described this as Minimalist illustration of a white bird with an orange beak riding on a dark gray motorcycle against a light blue sky with a white sun and gray ground
Blue sky and what looks like an eagle flying towards the viewer.Screenshot of a chat interface showing settings options. At the top is a text input field that says

Magistral (Mistral)

Magistral — the first reasoning model by Mistral AI. Mistral's first reasoning model is out today, in two sizes. There's a 24B Apache 2 licensed open-weights model called Magistral Small (actually Magistral-Small-2506), and a larger API-only model called Magistral Medium. Magistral Small is available as mistralai/Magistral-Small-2506 on Hugging...

Read Simon's post →
See description below
Large blue rectangle image, in the middle is a very small rounded corner orange rectangle "basket" c

Mistral Small 3.2

Mistral-Small 3.2. Released on Hugging Face a couple of hours ago, so far there aren't any quantizations to run it on a Mac but I'm sure those will emerge pretty quickly. This is a minor bump to Mistral Small 3.1, one of my favorite local models. I've been running Small 3.1 via Ollama where it's a 15GB download - these 24 billion parameter models...

Read Simon's post →
The bicycle is a sin wave, but you can make out which bits are the frame and the wheels. The pelican is white with a black triangular bill. It's a much better attempt than the Ollama one.

Gemma 3n

Introducing Gemma 3n: The developer guide. Extremely consequential new open weights model release from Google today: Multimodal by design: Gemma 3n natively supports image, audio, video, and text inputs and text outputs. Optimized for on-device: Engineered with a focus on efficiency, Gemma 3n models are available in two sizes based on effective...

Read Simon's post →
Description below.
Screenshot of subscription pricing page showing two plans: SuperGrok at $30.00/month (marked as Popu

Grok 4

Grok 4. Released last night, Grok 4 is now available via both API and a paid subscription for end-users. Update: If you ask it about controversial topics it will sometimes search X for tweets "from:elonmusk"! Key characteristics: image and text input, text output. 256,000 context length (twice that of Grok 3). It's a reasoning model where you...

Read Simon's post →
Description by Claude Sonnet 4: Simple line drawing of a white rubber duck with orange beak sitting on a red bicycle with spoked wheels

Kimi K2

moonshotai/Kimi-K2-Instruct (via) Colossal new open weights model release today from Moonshot AI, a two year old Chinese AI lab with a name inspired by Pink Floyd’s album The Dark Side of the Moon. My HuggingFace storage calculator says the repository is 958.52 GB. It's a mixture-of-experts model with "32 billion activated parameters and 1...

Read Simon's post →
The bicycle has no spokes. The pelican is light yellow and is overlapping the middle of the bicycle, not perching on it - it has a large yellow beak and a weird red lower beak or wattle.
Pricing table with three columns showing Input token count (0-32K, 32K-128K, 128K-256K, 256K-1M), In

Qwen3-Coder

Qwen3-Coder: Agentic Coding in the World (via) It turns out that as I was typing up my notes on Qwen3-235B-A22B-Instruct-2507 the Qwen team were unleashing something much bigger: Today, we’re announcing Qwen3-Coder, our most agentic code model to date. Qwen3-Coder is available in multiple sizes, but we’re excited to introduce its most powerful...

Read Simon's post →
Description by Claude Sonnet 4: Cartoon illustration of a white duck sitting on a black bicycle against a blue sky with a white cloud, yellow sun, and green grass below

Qwen3-Coder

Qwen/Qwen3-235B-A22B-Instruct-2507. Significant new model release from Qwen, published yesterday without much fanfare. (Update: probably because they were cooking the much larger Qwen3-Coder-480B-A35B-Instruct which they released just now.) This is a follow-up to their April release of the full Qwen 3 model family, which included a Qwen3-235B-A22B...

Read Simon's post →
It's a pelican riding a bicycle in front of the Golden Gate Bridge, wearing a blue hat. Overlaid text says Specify the environment or setting where your scene takes place.

Instagram Reel: Veo 3 paid preview

Instagram Reel: Veo 3 paid preview. @googlefordevs on Instagram published this reel featuring Christina Warren with prompting tips for the new Veo 3 paid preview (mp4 copy here). (Christine checked first if I minded them using that concept. I did not!)

Read Simon's post →
Description by Claude Sonnet 4: Minimalist flat illustration featuring a white bird character with orange beak, a purple rectangular tablet or device, gray cloud-like shapes, two black "T" letters, co

Qwen3-Coder

Qwen3-235B-A22B-Thinking-2507 (via) The third Qwen model release week, following Qwen3-235B-A22B-Instruct-2507 on Monday 21st and Qwen3-Coder-480B-A35B-Instruct on Tuesday 22nd. Those two were both non-reasoning models - a change from the previous models in the Qwen 3 family which combined reasoning and non-reasoning in the same model, controlled...

Read Simon's post →
Description by Claude Sonnet 4: This is a whimsical illustration of a white duck or goose riding a red bicycle. The bird has an orange beak and is positioned on the bike seat, with its orange webbed f
Description by Claude Sonnet 4: This image shows a cute, minimalist illustration of a snowman riding

GLM-4.5

I wrote about the new GLM-4.5 model family yesterday—new open weight (MIT licensed) models from Z.ai in China which their benchmarks claim score highly in coding even against models such as Claude Sonnet 4. GLM-4.5: Reasoning, Coding, and Agentic Abililties. Another day, another significant new open weight model release from a Chinese frontier AI...

Read Simon's post →
Visit Trying out Qwen3 Coder Flash using LM Studio and Open WebUI and LLM
A line with two dots, over a rhombus, with two circles and a pelican that looks like a grey snowman.

Qwen3-Coder

Qwen just released their sixth model(!) of this July called Qwen3-Coder-30B-A3B-Instruct—listed as Qwen3-Coder-Flash in their chat.qwen.ai interface. Qwen3-30B-A3B-Thinking-2507 (via) Yesterday was Qwen3-30B-A3B-Instruct-2507. Qwen are clearly committed to their new split between reasoning and non-reasoning models (a reversal from Qwen 3 in...

Read Simon's post →
Red bicycle with the right pieces, Pelican has two pieces to its beak that look about the right shape.

Gemini 3 Deep Think

Deep Think in the Gemini app (via) Google released Gemini 2.5 Deep Think this morning, exclusively to their Ultra ($250/month) subscribers: It is a variation of the model that recently achieved the gold-medal standard at this year's International Mathematical Olympiad (IMO). While that model takes hours to reason about complex math problems,...

Read Simon's post →
The pelican is blue and looks a little like a sad elephant. The bicycle looks more like a flat motorbike. The pelican has a blue tail and orange legs.

XBai o4 + o3-mini

XBai o4 (via) Yet another open source (Apache 2.0) LLM from a Chinese AI lab. This model card claims: XBai o4 excels in complex reasoning capabilities and has now completely surpassed OpenAI-o3-mini in Medium mode. This a 32.8 billion parameter model released by MetaStone AI, a new-to-me lab who released their first model in March -...

Read Simon's post →
Pelican is line art, does have a good beak and feet on the pedals, bicycle is very poorly designed and not the right shape.
This one has shaded colors for the different parts of the pelican. Still a bad bicycle.Visit GPT-5: Key characteristics, pricing and model cardVisit OpenAI's new open weight (Apache 2) models are really good

Claude Opus 4.1 + GPT-5

I’ve had preview access to the new GPT-5 model family for the past two weeks (see related video and my disclosures) and have been using GPT-5 as my daily-driver. It’s my new favorite model. It’s still an LLM—it’s not a dramatic departure from what we’ve had before—but it rarely screws up and generally feels competent or occasionally impressive at...

Read Simon's post →
Visit Qwen3-4B-Thinking: "This is art - pelicans don't ride bikes!"

GPT-5 (Preview)

I’ve fallen a few days behind keeping up with Qwen. They released two new 4B models last week: Qwen3-4B-Instruct-2507 and its thinking equivalent Qwen3-4B-Thinking-2507. A couple of weeks ago I was invited to OpenAI's headquarters for a "preview event", for which I had to sign both an NDA and a video release waiver. I suspected it might relate to...

Read Simon's post →
📝

David Ho on BlueSky: A pelican tried to eat my bike

David Ho on BlueSky: A pelican tried to eat my bike. David Ho caught video footage of one of the pelicans in St James's Park expressing deep curiosity in his bicycle. I think it wants to ride it.

Read Simon's post →
Cartoon illustration of a white bird with an orange beak riding a bicycle against a blue sky background with bright green grass below

DeepSeek 3.1

DeepSeek 3.1. The latest model from DeepSeek, a 685B monster (like DeepSeek v3 before it) but this time it's a hybrid reasoning model. DeepSeek-V3.1-Think achieves comparable answer quality to DeepSeek-R1-0528, while responding more quickly. Drew Breunig points out that their benchmarks show "the same scores with 25-50% fewer tokens" - at least...

Read Simon's post →
This one is pretty good. Bicycle is mostly the right shape, the pelican is perched quite awkwardly on the saddle, it has quite a good attempt at the chain mechanism.

Kimi K2 (Sept Update)

Kimi-K2-Instruct-0905. New not-quite-MIT licensed model from Chinese Moonshot AI, a follow-up to the highly regarded Kimi-K2 model they released in July. This one is an incremental improvement - I've seen it referred to online as "Kimi K-2.1". It scores a little higher on a bunch of popular coding benchmarks, reflecting Moonshot's claim that it...

Read Simon's post →
The bicycle is too simple and way too wide. The pelican is two circles, two orange triangular feed and a big triangle for the beak.
Blue background, brown ground, bicycle looks more like a wheelchair, pelican is actually quite good

Qwen3-Next-80B

Qwen3-Next-80B-A3B. Qwen announced two new models via their Twitter account (and here's their blog): Qwen3-Next-80B-A3B-Instruct and Qwen3-Next-80B-A3B-Thinking. They make some big claims on performance: The name "80B-A3B" indicates 80 billion parameters of which only 3 billion are active at a time. You still need to have enough GPU-accessible RAM...

Read Simon's post →
Described by Grok 4 Fast: A simple line drawing of a white pelican with a yellow beak holding a yellow object, riding a black bicycle on green grass under a blue sky with white clouds.
Described by Grok 4 Fast: Simple line drawing of a white bird with a long yellow beak riding a bicyc

Grok 4 Fast

Grok 4 Fast. New hosted vision-enabled reasoning model from xAI that's designed to be fast and extremely competitive on price. It has a 2 million token context window and "was trained end-to-end with tool-use reinforcement learning". It's priced at $0.20/million input tokens and $0.50/million output tokens - 15x less than Grok 4 (which is...

Read Simon's post →
See description below

GPT-5-Codex

GPT-5-Codex. OpenAI half-released this model earlier this month, adding it to their Codex CLI tool but not their API. Today they've fixed that - the new model can now be accessed as gpt-5-codex. It's priced the same as regular GPT-5: $1.25/million input tokens, $10/million output tokens, and the same hefty 90% discount for previously cached input...

Read Simon's post →
Visit Claude Sonnet 4.5 is probably the "best coding model in the world" (at least for now)

Gemini 2.5 Flash + Sonnet 4.5

My benchmark for LLMs: "Generate an SVG of a pelican riding a bicycle". "User might be a kid playing with words" according to Qwen3-4B-Thinking. Anthropic released Claude Sonnet 4.5 today, with a very bold set of claims: Improved Gemini 2.5 Flash and Flash-Lite (via) Two new preview models from Google - updates to their fast and inexpensive Flash...

Read Simon's post →
Claude Sonnet 4.5 says: Minimalist line drawing illustration of a stylized bird riding a bicycle, with clock faces as wheels showing approximately 10:10, orange beak and pedal accents, on a light gray
Claude Sonnet 4.5 says: Illustration of a white seagull with an orange beak and yellow feet riding a

DeepSeek V3.2-Exp + GLM-4.6

Two new models from Chinese AI labs in the past few days. I tried them both out using llm-openrouter : DeepSeek-V3.2-Exp from DeepSeek. Announcement , Tech Report , Hugging Face (690GB, MIT license). This one felt very slow when I accessed it via OpenRouter - I probably got routed to one of the slower providers . Here's the pelican : GLM-4.6 from Z.ai . Announcement , Hugging Face (714GB, MIT license). Here's the pelican for that: Tags: llm , pelican-riding-a-bicycle , deepseek , ai-in-china ,..

Read Simon's post →
A nice illustration of a pelican riding a bicycle, both pelican and bicycle are exactly as you would hope. Looks sketched, maybe colored pencils? The pelican's two legs are on the pedals but it also h

gpt-image-1-mini

gpt-image-1-mini OpenAI released a new image model today: gpt-image-1-mini , which they describe as "A smaller image generation model that’s 80% less expensive than the large model." They released it very quietly - I didn't hear about this in the DevDay keynote but I later spotted it on the DevDay 2025 announcements page . It wasn't instantly obvious to me how to use this via their API. I ended up vibe coding a Python CLI tool for it so I could try it out. I dumped the plain text diff version...

Read Simon's post →
It's obviously a pelican riding a bicycle. Half the spokes are missing on each wheel and the pelican is a bit squat looking.

GPT-5 Pro

GPT-5 pro Here's OpenAI's model documentation for their GPT-5 pro model, released to their API today at their DevDay event. It has similar base characteristics to GPT-5 : both share a September 30, 2024 knowledge cutoff and 400,000 context limit. GPT-5 pro has maximum output tokens 272,000 max, an increase from 128,000 for GPT-5. It's only available via OpenAI's Responses API. My LLM tool doesn't support that in core yet, but the llm-openai-plugin plugin does. I released llm-openai-plugin 0.7...

Read Simon's post →
Described by Haiku 4.5: A whimsical illustration of a bird with a round tan body, pink beak, and orange legs riding a bicycle against a blue sky and green grass background.

Claude Sonnet 4.5

Introducing Claude Haiku 4.5 Anthropic released Claude Haiku 4.5 today, the cheapest member of the Claude 4.5 family that started with Sonnet 4.5 a couple of weeks ago . It's priced at $1/million input tokens and $5/million output tokens, slightly more expensive than Haiku 3.5 ($0.80/$4) and a lot more expensive than the original Claude 3 Haiku ($0.25/$1.25), both of which remain available at those prices. It's a third of the price of Sonnet 4 and Sonnet 4.5 (both $3/$15) which is notable...

Read Simon's post →
Diagram showing a pelican riding a bicycle illustration alongside its SVG source code. The left side displays two versions: a completed color illustration at top with a white pelican with yellow beak

SVG Visual Features (Anthropic Research)

Visual Features Across Modalities: SVG and ASCII Art Reveal Cross-Modal Understanding New model interpretability research from Anthropic, this time focused on SVG and ASCII art generation. And really, I can't not link to this one given the bonus they tagged on at the end! Now that they can identify model features associated with visual concepts in SVG images, can they us those for steering? It turns out they can! Starting with a smiley SVG (provided as XML with no indication as to what it was...

Read Simon's post →
Bicycle has a red upside down Y shaped frame, pelican is a bit dumpy, it does at least have a long sharp beak.

SWE-1.5 (Windsurf)

Introducing SWE-1.5: Our Fast Agent Model Here's the second fast coding model released by a coding agent IDE in the same day - the first was Composer-1 by Cursor . This time it's Windsurf releasing SWE-1.5: Like Composer-1 it's only available via their editor, no separate API yet. Also like Composer-1 they don't appear willing to share details of the "leading open-source base model" they based their new model on. I asked it to generate an SVG of a pelican riding a bicycle and got this: This one.

Read Simon's post →
Biycle is good though obscured by the pelican. Pelican has an impressive triple beak and is stretched along the bicycle frame. Not clear if it can pedal or what it is sitting on.

MiniMax M2

MiniMax M2 & Agent: Ingenious in Simplicity MiniMax M2 was released on Monday 27th October by MiniMax, a Chinese AI lab founded in December 2021. It's a very promising model. Their self-reported benchmark scores show it as comparable to Claude Sonnet 4, and Artificial Analysis are ranking it as the best currently available open weight model according to their intelligence score: The size is particularly significant: the model weights are 230GB on Hugging Face , significantly smaller than other..

Read Simon's post →
The bicycle is levitating against a blue sky. The pelican looks a little bit more like a baby chicken but does at least have a long beak.

Composer-1 (Cursor)

Composer: Building a fast frontier model with RL Cursor released Cursor 2.0 today , with a refreshed UI focused on agentic coding (and running agents in parallel) and a new model that's unique to Cursor called Composer 1 . As far as I can tell there's no way to call the model directly via an API, so I fired up "Ask" mode in Cursor's chat side panel and asked it to "Generate an SVG of a pelican riding a bicycle": Here's the result : The notable thing about Composer-1 is that it is designed to be.

Read Simon's post →
Sonnet 4.5 described this as: Cartoon illustration of a white duck or goose with an orange beak and gray wings riding a bicycle with a red frame and light blue wheels against a light blue background.
Sonnet 4.5: Minimalist cartoon illustration of a white bird with an orange beak and feet standing on

Kimi K2 Thinking

Kimi K2 Thinking Chinese AI lab Moonshot's Kimi K2 established itself as one of the largest open weight models - 1 trillion parameters - back in July . They've now released the Thinking version, also a trillion parameters (MoE, 32B active) and also under their custom modified (so not quite open source ) MIT license. This one is only 594GB on Hugging Face - Kimi K2 was 1.03TB - which I think is due to the new INT4 quantization. This makes the model both cheaper and faster to host. So far the...

Read Simon's post →
3D scene. The bicycle has a sort of square frame in the wrong place, but good wheels. The pelican is stood on top - a large white blob, a smaller white blob head, a cylinder neck and a conical beak in
The bike is a bit mis-shapen but has most of the right pieces. The pelican has legs that reach the pTwo wheels (tire only) sit overlapping half embedded in the ground. The frame is a half-buried red t

POV-Ray Raytracer Edition

Pelican on a Bike - Raytracer Edition beetle_b ran this prompt against a bunch of recent LLMs: This turns out to be a harder challenge than SVG, presumably because there are less examples of POV-Ray in the training data: The results are really fun though! A lot of them end up accompanied by a weird floating egg for some reason - here's Claude Opus 4 : I think the best result came from GPT-5 - again with the floating egg though! I decided to try this on the new gpt-5-codex-mini , using the trick.

Read Simon's post →
Visit What happens if AI labs train for pelicans riding bicycles?
Left is a simple incorrectly shaped bicycle and a not great pelican. On the right the bicycle has mo

Agentic Pelican (Multi-iteration)

Agentic Pelican on a Bicycle Robert Glaser took my pelican riding a bicycle benchmark and applied an agentic loop to it, seeing if vision models could draw a better pelican if they got the chance to render their SVG to an image and then try again until they were happy with the end result. Here's what Claude Opus 4.1 got to after four iterations - I think the most interesting result of the models Robert tried: I tried a similar experiment to this a few months ago in preparation for the GPT-5...

Read Simon's post →
The bicycle wheels have no spokes at all, the pelican is laying quite flat on it
This bicycle has four spokes per wheel, and the pelican is sitting more uprightVisit Trying out Gemini 3 Pro with audio transcription and a new pelican benchmark

GPT-5.1

Introducing GPT-5.1 for developers OpenAI announced GPT-5.1 yesterday, calling it a smarter, more conversational ChatGPT . Today they've added it to their API. We actually got four new models today: gpt-5.1 gpt-5.1-chat-latest gpt-5.1-codex gpt-5.1-codex-mini There are a lot of details to absorb here. GPT-5.1 introduces a new reasoning effort called "none" (previous were minimal, low, medium, and high) - and none is the new default. When you DO enable thinking you get to benefit from a new...

Read Simon's post →
Visit Claude Opus 4.5, and why evaluating new LLMs is increasingly difficult
A flat-style illustration shows a white, round-bodied bird with an orange beak pedaling a red-framedA plump white bird with an orange beak and small black eyes crouches low on a blue bicycle with overA stylized dark gray bird with layered wings, a yellow head crest, and a long brown beak leans forwa

GPT-5.1-Codex-Max

Building more with GPT-5.1-Codex-Max Hot on the heels of yesterday's Gemini 3 Pro release comes a new model from OpenAI called GPT-5.1-Codex-Max. (Remember when GPT-5 was meant to bring in a new era of less confusing model names? That didn't last!) It's currently only available through their Codex CLI coding agent , where it's the new default model: It's not available via the API yet but should be shortly. The timing of this release is interesting given that Gemini 3 Pro appears to have aced...

Read Simon's post →
📝

Tom Gally's SVG Benchmark

LLM SVG Generation Benchmark Here's a delightful project by Tom Gally, inspired by my pelican SVG benchmark . He asked Claude to help create more prompts of the form Generate an SVG of [A] [doing] [B] and then ran 30 creative prompts against 9 frontier models - prompts like "an octopus operating a pipe organ" or "a starfish driving a bulldozer". Here are some for "butterfly inspecting a steam engine": And for "sloth steering an excavator": It's worth browsing the whole collection , which gives..

Read Simon's post →
Pleasing gradents for the sky and ground and sun. Neat three-circle clouds. A Pelican on a Bicycle title printed on the image. The pelican is cute but stlightly detached from the bicycle. The bicycle
It's not great. The bicycle is distorted, the pelican is a white oval, an orange almost-oval beak, a

DeepSeek V3.2 + Speciale

DeepSeek-V3.2 Two new open weight (MIT licensed) models from DeepSeek today: DeepSeek-V3.2 and DeepSeek-V3.2-Speciale , both 690GB, 685B parameters. Here's the PDF tech report . DeepSeek-V3.2 is DeepSeek's new flagship model, now running on chat.deepseek.com . The difference between the two new models is best explained by this paragraph from the technical report: I covered DeepSeek-Math-V2 last week . Like that model, DeepSeek-V3.2-Speciale also scores gold on the 2025 International...

Read Simon's post →
Visit GPT-5.2
A small white pelican on what looks more like a child's cart.Bicycle looks a bit like a cybertruck

Devstral 2 + Devstral Small 2

Devstral 2 Two new models from Mistral today: Devstral 2 and Devstral Small 2 - both focused on powering coding agents such as Mistral's newly released Mistral Vibe which I wrote about earlier today . Devstral 2 is a 123B model released under a janky license - it's "modified MIT" where the modification is: Mistral Small 2 is under a proper Apache 2 license with no weird strings attached. It's a 24B model which is 51.6GB on Hugging Face and should quantize to significantly less. I tried out the..

Read Simon's post →
Alt text by GPT-5.2-Codex: A minimalist illustration of a white pelican with a large orange beak riding a teal bicycle across a sandy strip of ground. The pelican leans forward as if pedaling, its win

GPT-5.2-Codex

Introducing GPT-5.2-Codex The latest in OpenAI's Codex family of models (not the same thing as their Codex CLI or Codex Cloud coding agent tools). As with some previous Codex models this one is available via their Codex coding agents now and will be coming to the API "in the coming weeks". Unlike previous models there's a new invite-only preview process for vetted cybersecurity professionals for "more permissive models". I've been very impressed recently with GPT 5.2's ability to tackle...

Read Simon's post →
2026
By 2026, the benchmark had become a cultural institution. New models were barely announced before someone asked: "but how's the pelican?"
Cartoon illustration of a white pelican with a large orange beak and yellow throat pouch riding a green bicycle with yellow feet on the pedals, set against a light blue sky with soft bokeh circles and

Kimi K2.5

Kimi K2.5: Visual Agentic Intelligence Kimi K2 landed in July as a 1 trillion parameter open weight LLM. It was joined by Kimi K2 Thinking in November which added reasoning capabilities. Now they've made it multi-modal: the K2 models were text-only, but the new 2.5 can handle image inputs as well: The "self-directed agent swarm paradigm" claim there means improved long-sequence tool calling and training on how to break down tasks for multiple agents to work on at once: I used the OpenRouter...

Read Simon's post →
Slightly wonky bicycle frame but an excellent pelican, very clear beak and pouch, nice feathers.
Not nearly as good - the bicycle is a bit mangled, the pelican not nearly as well rendered - it's mo

Opus 4.6 + GPT-5.3-Codex

Two major new model releases today, within about 15 minutes of each other. Anthropic released Opus 4.6 . Here's its pelican : OpenAI release GPT-5.3-Codex , albeit only via their Codex app, not yet in their API. Here's its pelican : I've had a bit of preview access to both of these models and to be honest I'm finding it hard to find a good angle to write about them - they're both really good , but so were their predecessors Codex 5.2 and Opus 4.5. I've been having trouble finding tasks that...

Read Simon's post →
The pelican is good and has a well defined beak. The bicycle frame is a wonky red triangle. Nice sun and motion lines.

GLM-5

GLM-5: From Vibe Coding to Agentic Engineering This is a huge new MIT-licensed model: 754B parameters and 1.51TB on Hugging Face twice the size of GLM-4.7 which was 368B and 717GB (4.5 and 4.6 were around that size too). It's interesting to see Z.ai take a position on what we should call professional software engineers building with LLMs - I've seen "Agentic Engineering" show up in a few other places recently. most notable from Andrej Karpathy and Addy Osmani . I ran my "Generate an SVG of a...

Read Simon's post →
Whimsical flat illustration of an orange duck merged with a bicycle, where the duck's body forms the seat and frame area while its head extends forward over the handlebars, set against a simple light
Whimsical flat illustration of a white pelican riding a dark blue bicycle at speed, with motion line

GPT-5.3-Codex-Spark

Introducing GPT‑5.3‑Codex‑Spark OpenAI announced a partnership with Cerebras on January 14th . Four weeks later they're already launching the first integration, "an ultra-fast model for real-time coding in Codex". Despite being named GPT-5.3-Codex-Spark it's not purely an accelerated alternative to GPT-5.3-Codex - the blog post calls it "a smaller version of GPT‑5.3-Codex" and clarifies that "at launch, Codex-Spark has a 128k context window and is text-only." I had some preview access to this...

Read Simon's post →
This alt text also generated by Gemini 3 Deep Think: A highly detailed, colorful, flat vector illustration with thick dark blue outlines depicting a stylized white pelican riding a bright cyan blue bi
Also described by Gemini 3 Deep Think: A highly detailed, vibrant, and stylized vector illustration

Gemini 3 Deep Think

Gemini 3 Deep Think New from Google. They say it's "built to push the frontier of intelligence and solve modern challenges across science, research, and engineering". It drew me a really good SVG of a pelican riding a bicycle ! I think this is the best one I've seen so far - here's my previous collection . (And since it's an FAQ, here's my answer to What happens if AI labs train for pelicans riding bicycles? ) Since it did so well on my basic Generate an SVG of a pelican riding a bicycle I...

Read Simon's post →
The benchmark continues. Each new model gets the same prompt. The pelicans keep getting better — but the bicycles remain surprisingly tricky.