How do we define creativity in LLMs?

How do we define creativity in LLMs?

In the recent ACL 2026 paper called “Automated Creativity Evaluation of Language Models Across Open-Ended Tasks”, researchers tried to address that problem.

We often say that one model seems to be more (or less) „creative”, while the definition itself is blurry. Is it measurable?

They came up with two metrics:

Divergent creativity: ability to explore ideas

Imagine you give a model a task: „Write me ten ideas to continue this story.”

One model gives you 10 ideas that would result in completely different plot directions. The other gives you ten ideas that are similar, and in the end the plot would end up in roughly the same place.

The first one would have higher divergent creativity. But generating many divergent ideas doesn't necessarily mean they are good.

This is where convergent creativity comes in.

Convergent creativity

Convergent creativity means whether the model can take an idea and execute it well (follow requirements, keep the writing coherent, etc.).

So:

• Divergent creativity = How many interesting directions can you think of?
• Convergent creativity = Can you actually make one of them work?

Experiment

They compared multiple models including GPT-4o, Llama, Qwen3, DeepSeek R1 and others, on 300 tasks per domain (MacGyver, HypoGen, BookMIA).

To measure divergent creativity, they generated multiple possible responses, grouped semantically similar ones into clusters and calculated semantic entropy — basically checking how many genuinely different ideas the model explores.

As the authors put it, „a less creative model yields surface-level rephrasings and low semantic entropy”, while „a more creative model yields genuinely distinct ideas and high semantic entropy”.

To measure convergent creativity, they used a multi-agent judging framework. Three specialised LLM agents (Problem, Solution and Criterion) analyze the response from different perspectives and share relevant observations. In the end, the most confident agent gives a verdict on whether the response satisfies each criterion.

Results

Larger and newer models generally became better at convergent creativity (they followed instructions better and produced more coherent outputs).

However, divergent creativity wasn’t related to the model size. Moreover, a newer model can become much better at executing an idea without necessarily becoming better at coming up with genuinely different ideas in the first place.

Paper: Automated Creativity Evaluation of Language Models Across Open-Ended Tasks