The best Large Language Models of December 2024

LLM Benchmarks | December 2024

The benchmarks evaluate the models in terms of their suitability for digital product development. The higher the score, the better.

☁️ - Cloud models with proprietary license
✅ - Open source models that can be run locally without restrictions
🦙 - Local models with Llama license

Benchmarking Llama 3.3, Amazon Nova, Gemini 1206

We’ll cover these models in one go.

Meta Llama 3.3 70B Instruct - 45th place.

Llama 3.3 70B Instruct held 40th at the moment of the release, but since then a few other better models showed up. This is the common pattern - if a company doesn’t release better models, it will get pushed down by the competitors pretty fast.

Llama 3.3 70B has a decent Reason, just below Llama 405B and older Llama 3.1 70B, however it doesn’t follow that well instructions in business tasks. This is a typical problem for Llama models. It would normally be fixed by good fine tunes, except that the market started realising that ROI of fine-tuning models in practice is lower than it seemed. So we don’t expect any changes in its position anytime soon.

Amazon Nova - bad

Amazon has released their own versions of LLMs: Amazon Nova Micro, Lite and Pro. They are very cheap to run and quite useless, taking 36, 55 and 79 places.

Do you know what the silver lining here is? These bad models achieve the quality of GPT-3.5 which was ground-breaking back in the day. So the models aren’t that bad, the progress is just making us move the goalposts really fast without noticing it.

Google Gemini Experimental 1206 and 2.0 Flash Experimental

Google Gemini Experimental 1206 - not so good

Google Gemini Experimental took 28th place, which is much worse than Google Gemini 1.5 Pro v2. The latter is really good, if you manage to get over all the Google quirks. However, that’s acceptable because 1206 is only an experimental model, not an official release.

Yet, this matches the quality level of some GPT-4 Turbo versions!

Google Gemini 2.0 Flash Experimental is a more interesting model

It is still experimental, but it made its way into the TOP-10 of our benchmark!

Compared to the previous version of Flash (Gemini 1.5 Flash), this experimental model improved its reasoning capability from 44 to 62, while increasing the overall score from 75 to 84.

Google Gemini 2.0 Flash also pays a lot of attention to instructions (which is really important for Structured Output / Custom Chain of Thought patterns) and has achieved a perfect 100 score in Docs and Integrate. It is the first model to do so.

Google Deepmind writes that the model was created for automatisation and "agentic experiences" —whatever that means — it has 1M input context.

This model also potentially has the lowest usage cost within the top 19 models. 20th model is another cost-wise contender DeepSeek v3 671B.

“Potentially”, because the price for Google Gemini 2.0 Flash is not known at this moment, yet. So we assume it’s the same as Flash 1.5.

Google continues to surprise us in a good way, continuously releasing new models that make it into the TOP-10. This has a side effect of pushing old favourites (Mistral and Anthropic) down from the spotlight of the fame. This doesn’t make these models worse, on the contrary, it means that we are getting more options to choose from!

Deep Seek v3

DeepSeek v3 is a recently released Mixture-of-Experts (MoE) language model with 671B total parameters. It tries to be efficient inference-wise, so only 37B parameters are activated for each token. This reflects in the cost of running this model. This model is locally-capable (you can download it and run on your servers, provided, there are enough GPUs to host the weights).

DeepSeek v3 has improved over the scores of its predecessor DeepSeek v2.5 (now at TOP 30). It can solve business automation tasks in CRM category at 97 score (from 80). Ability to solve software engineering tasks improved from 57 to 62 (although the model still has a long way to go to catch up with good old Sonnet 3.5 Claude v2 at 82).

Even though DeepSeek v3 activates only 37B parameters per token, this doesn’t make it easier to launch the model locally. Mixture of Experts (MoE) means “faster inference”, not “lower VRAM requirements”. We would need something like 8xH200 GPUs to run model inference locally. This makes the model not so suitable for the local use.

What is peculiar about DeepSeek v3 - it is the first model to use FP8 mixed-precision training framework. This approach enables training new LLM models faster, cheaper and with lower VRAM requirements. This should potentially also lead to better out-of-the-box quantisation at inference. Let’s see if that technique will help to create more of small and powerful local models.

Manual benchmark of OpenAI o1 pro - Gold Standard

Let’s move towards the hero of this LLM Benchmark - o1 pro from the OpenAI. But before we proceed, there is an important caveat. There are 6 different flavours of OpenAI o1 model, don’t mix them up!

• o1-mini - the smallest and cheapest of all reasoning models. It is available both in ChatGPT UI and over the API.

• o1-preview - really capable reasoning version that was previously available in ChatGPT UI. It is no longer available there, o1 base replaced it. o1-preview is still available directly through the API.

• o1 - this is the model that replaces o1-preview in ChatGPT UI. This version has by default more limited reasoning capability in the UI, making it less capable than o1-preview. This model isn’t widely accessible via the API, yet (only for Tier-5 accounts). o1 has three possible reasoning effort configurations in the API: high, medium and low. The higher the effort, the more expensive and capable the model becomes.

• o1 - pro - this is the most powerful model of them all. It is available in ChatGPT UI for the monthly cost of 200$. It isn’t available via the API, yet.

There you have it - 4 flavours of o1 model, plus 2 additional configurations (high and low) for the o1 model.

This section focuses solely on the o1 pro. This model, as an exception, was not tested via the API (because it isn’t available, yet), but manually through the ChatGPT UI. Here is how it was done:

• We took the results from the o1-mini benchmark and selected only the tasks where o1-mini made mistakes. Since o1 pro is way more capable, we assumed that if o1-mini got something right, o1 pro would also answer correctly. This way we didn’t have to run manually a few hundred tasks from the entire benchmark, only a few dozen.

• We made sure to disable custom instructions in the ChatGPT UI. Local memory was also disabled.

• We converted benchmark requests from API format to a textual format and launched them manually by copy-pasting.

This is where we’ve encountered the first gotchas.

First of all, o1 pro is embedded deeply in the ChatGPT UI, which tries to be convenient. For example, if task has to return YAML, it will get formatted as markdown, breaking the response completely. We had to fix answers like that manually.

Second, we have historically formatted few-shot samples like this:

System: Task explanation
User: sample request 1
Assistant: sample response 1
User: sample request 2
Assistant: sample response 2 
User: real request

We can’t do role-based prompting in the ChatGPT UI. Besides, System prompt isn’t even accessible in o1 lines of models to prevent reasoning tokens from leaking to the end-users (they are generated by the models without alignment and guardrails). The model isn’t only designed to protect its System prompt (it is also called as Platform prompt in the latest documentation), it also tries to work with the user via the dialogue.

This lead to an interesting outcome: the model gave lower priority to the System instructions and tried to find patterns in past conversations with the user. It could occasionally find them and arrive to the incorrect conclusions, leading in low integrate scores.

So we had to start formatting o1 pro tasks like this:

# Task
Task explanation
## Example
User: sample request 1
Assistant: sample response 1
## Example
User: sample request 2
Assistant: sample response 2
# Request
real request

Having that said, what were the results?

o1 pro reached the very top of our benchmark, achieving almost perfect 97 (remaining 3 points could be attributed to ambiguous tasks in our benchmark).

Within our benchmark that measures capabilities of LLM models in business automation tasks this model is like a gold ingot. It is perfect and expensive. It is an overkill.

As always, these are the good news for two reasons:

• We have clearly arrived to the point that LLMs can easily solve all tasks in our business automation challenges (from 18 months ago). Now we just need to wait for comparable models that are cheaper to run.

• While developing the second version of LLM benchmark, we can keep current o1 pro capabilities in sights and formulate new ones tasks that will challenge even o1 pro. This will make the evaluation complexity curve more smooth, helping the entire benchmark to be more representative in tasks of business automation.

Benchmark of o1 (base) - 🥉TOP-3

Do you remember the disclaimer about different flavours of o1 models above?

This benchmark focuses on o1 (base) model that was tested through the API with reasoning_effort of medium. This is not necessarily the same model configuration as what is available via the ChatGPT UI.

The difference is not only in the different compute limitations, there also is a new chain of command (rules of robotics, as implemented by OpenAI for the reasoning models): Platform > Developer > User > Tool.

O1 base model was tested automatically via the API, just like all the other models (except o1 pro). It scored the 3rd place - slightly better than o1-mini, slightly worse than o1-preview.

reasoning_effort was set to medium (default value) and max_tokens were set to 25000(as per OpenAI recommendation).

What is peculiar, o1 base takes the 3rd place both in capability and in cost. This makes for a very interesting curve: at the very top, reasoning capability is a function of a cost.

o1-preview works better than o1 base, because it generates more tokens, but the result is also better. o1 pro just things deeper and more throughly in general.

This trend also backs up recent research from the Hugging Face on Scaling Test-time compute. It is about improving quality of 3B model to the level of 70B via spending more time on reasoning (and generating possible answers). So we can probably expect more LLM providers to start offering smarter models for an extra price (you pay for the reasoning tokens).

Perhaps afterwards there will also be new ways to launch intensive reasoning locally as well (similarly how it happened with local structured outputs).

What about recently announced o3?

Recently OpenAI has announced its new model o3, which solves tasks from ARC-AGI dataset really well.

Why is there o1 and o3 but no o2? Naming conflict with O2 telecom company could be the root cause.

What is ARC-AGI? It is a set of challenges that attempt to compare human intellect with machine. The website claims that solving ARC-AGI is even more outstanding than the invention of a transformer architecture.

Below is an example of one benchmark. To solve it, machine needs to figure out the rules and produce pixel-perfect response.

As the story goes, o3 was able to solve almost all tasks from this benchmark. This is something that wasn’t considered to be possible before.

This makes o3 theoretically the best LLM model. However, we believe that it wouldn’t have a noticeable impact on business automation tasks in companies any time soon. There is a catch - costs.

Take a look at the chart below from the ARC-AGI announcement. It maps performance of different models vs the cost of solving a single task.

Cost scale is logarithmic. So the cost of solving a single task with O3 HIGH (Tuned) was around 3200 USD per a single pixel-perfect response.

We mentioned earlier that o1 is the gold standard—it’s perfect in business automation and already too expensive to be practical for the most of the cases. o3 pushes the envelope even further.

Yet, adoption for LLM models works out well in cases where we gain a lot of value from automation. This business value is currently achievable in the mundane tasks where LLMs are cheaper, more patient and accurate than humans. These are simple and easily verifiable tasks like data extraction from the documents, request classification, code generation, review of standard contracts etc.

The real issue here is cost-efficiency. o3 from the OpenAI is not cheap at all, so it will not have a big impact. However, it might pave the way for improving quality of the other models, e.g. via the generation of high quality synthetic data that could be used for training.

Our Predictions for 2025

These are the opinionated predictions, based on the patterns observed among our AI Cases.

Hype of fine-tuning of LLMs will die out.

Fine-tuning of LLMs was frequently mentioned as a way “to train LLM on your company data” or “teach LLM new tricks”. Even OpenAI offers fine-tuning as a service.

In theory everything looks simple - just feed the LLM with a lot of documents, and it will learn from them. What happens in practice: instead of getting better accuracy, teams suddenly end up with models that hallucinate a lot more. Most of the times they underestimate the complexity of preparing proper data and following the training regimen.

Among our AI Cases, only one project has successfully fine-tuned LLM (we are not counting embedding models, of course). They had a lot of carefully prepared data for the task, still it took them quite a few iterations.

We believe that in the year 2025 both the businesses and software consultants/vendors will start to realise real complexity and cost of fine-tuning LLMs. They will also understand the power that a good foundational LLM can already provide out-of-the-box, especially if one leverages powerful inference patterns like Structured Outputs and Custom Chain-of-Thought.

Hype of Autonomous Agents will start fading away

We aren’t claiming that autonomous agents are impossible. If enough effort is invested, it is possible to deliver something like that.

However, the concept of an autonomous agent is not very practical. It is a complex product to design, build and integrate, while ensuring predictable quality.

Please let us stress one fact: agents are not technically complex. In essence, it’s just a series of prompts that pass control and context to each other, while using external tools. Yet, due to the shape of the product, it is really hard for humans to setup cost-efficient process of delivering trustworthy agent-driven products. In practice things just start falling apart, budgets run out before the systems stop hallucinating.

Vendors will continue talking about agents in 2025 and selling “enterprise-ready agent frameworks” (they have investments to recoup), but we believe that the hype will start fading.

Will there be an AGI in 2025? What about LLM trends?

There will be no AGI in 2025. Generic intelligence is even a harder task to solve, especially since we are getting particularly adept in moving the goalposts of “what is an AGI?”. As the creators of ARC-AGI have written: “You'll know AGI is here when the exercise of creating tasks that are easy for regular humans but hard for AI becomes simply impossible.” And they are just working on v2 of their benchmark.

Still, a lot of companies will continue to try compete with OpenAI for the spot of the most intelligent model. There is even a chance that Google will eventually dethrone OpenAI.

Just look at the trends of quality increases among models in 2024 (by different providers and cost categories):

As a new shortcut for improving model reasoning, we believe, more AI vendors will start providing reasoning capabilities, similar to o1 models. This will be a temporary workaround to boost model accuracy quick and without heavy investments: just throw more compute, let the model think longer before answering and charge more for the API.

However, we also believe, that the upcoming hype of providing smart reasoning models that are outrageously expensive will also start fading. It is just not very practical.

We also believe that AI vendors will start exposing more advanced features in their LLMs. Everybody already has large context and Prompt Caching (which already makes dedicated RAG impractical in many cases). But there still are powerful features that are not rolled out widely:

• Structured Outputs (constrained decoding) - as a powerful way to increase quality of LLM answers in complex scenarios, especially when coupled with custom chain-of-thoughts. At the moment only OpenAI has a usable implementation. Google is still catching up with its mildly unusable controlled generation that uses VertexAI API format under the hood.

• Document reasoning with VLMs. Latest LLMs are no longer text-only, they can also accept images or audio. This allows to handle complex documents with tables and charts. Anthropic has already a flavour of this capability - it internally sends documents both as text and image to its Sonnet 3.5 model, which works as Vision-Language Model (VLM).

• Native integration of LLMs with the other tools - similar to how OpenAI has Assistants APIs that allow its LLMs to use local RAG and code execution sandbox. Anthropic is also trying to enter the playing field by introducing Model Context Protocol (a standard for connecting LLMs to data sources and external tools, inspired by Language Server Protocol).

We also expect that AI vendors will try to include unique features into their LLM APIs in order to attract users. There will be some standardisation (e.g. Google is testing VertexAI access via OpenAI libraries) and non-conformity at the same (e.g. compare how prompt caching works differently at Google, OpenAI and Anthropic).

The whole situation is going to resemble browser wars. Ultimately standards will start emerging, but until then we can expect a lot of quirks, frequent migration pains and evolving features.

Fortunately, if we look beyond a single provider at the market situation in general - bigger patterns start emerging. By optimising for the generic trends of the AI market, we can reduce the risk of making costly decisions and heading towards the dead-ends.

Transform Your Digital Projects with the Best AI Language Models!

Discover the transformative power of the best LLMs and revolutionize your digital products with AI! Stay future-focused, boost efficiency, and gain a clear competitive edge. We help you elevate your business value to the next level.