The best large language models for digital products in july 2024

Codestral Mamba 7B

Mistral AI has made quite a few releases this month, but Codestral Mamba is our favorite. It's not extremely powerful, comparable to models like Llama 3.1 8B or Claude 3 Sonnet. But there are a few nuances:

• This model is not designed for product or business tasks, it is a coding model. Nevertheless, it competes well with general purpose models.

• The model doesn’t implement the well-studied transformer architecture, but a Mamba (also known as Linear-Time Sequence Modeling with Selective State Spaces). This architecture is considered to be more resource-efficient and have less constraints on working with large contexts. There were multiple attempts to train a good Mamba model, but this is a first one to achieve good results on our leaderboard.

• The new model is available for local use and can be obtained directly from HuggingFace. Nvidia TensorRT-LLM already supports this model.

GPT-4o Mini

GPT-4o Mini is a new multimodal model from OpenAI. It is similar in class to the GPT-3.5 models, but has better overall results. Its Reason capability is quite large for such a small model. GPT-4o Mini is also the first model to score a perfect 100 in our Marketing category (tests working with language and writing styles).

Given the extremely low cost and good results, the GPT-4o Mini seems perfect for small, focused tasks such as routers and classifiers in LLM-driven products. Large scale data extraction tasks also look good.

Mistral Nemo 12B

Mistral AI has been pushing a lot of effort into bleeding edge R&D, it seems. The Mistral Nemo 12B is another example of this.

On one hand, this model is a bit larger than previous 7B models from Mistral AI. On the other hand it has a few interesting nuances that make up for that.

First of all, the model has better tokeniser under the hood, leading to more efficient token use (fewer tokens needed per input and output).

Secondly, the model was trained together with Nvidia using quantization-aware training. This means that the model is designed from the start to run in a resource-efficient mode. In this case, the model is designed to work well in FP8 mode, which means that the model weights take up a quarter of the usual size in memory (compared to FP32 format). Here is the announcement from Nvidia.

It's a nice coincidence that NVidia GPUs with CUDA Compute 9.0 generation are designed to run FP8 natively (e.g. H100 GPUs for data centers)

If you have the latest GPUs, this Mistral Nemo model can be a good replacement for the earlier 7B models from Mistral AI. Since the model also achieves a high Reason score, there is a chance that fine-tuning will push the model even higher.

You can download this model from Hugging Face or use it via the MistralAI API.

Mistral Large 123B v2

Mistral Large v2 is currently the best model of Mistral in our benchmarks. It is available for download, which means you can run it on your local machines (although a license is required for commercial use).

This model also has a large context of 128 tokens. It claims to support multiple languages, both human and programming languages.

In our benchmark, this model has really good results and an unusually high Reason capability. It is comparable with GPT-4 Turbo v3, Gemini Pro 1.5 and Claude 3 Opus.

The unusual size of this Mistral model could indicate that it was also trained with FP8 Awareness to replace the 70B modes in their lineup (12:7 ~~ 123:80). If that's the case, we could see a general trend where new models will appear in these odd sizes. However, they will only run well on the latest GPUs. This may fragment the LLM landscape and slow down progress.

The lineup of the best Mistral models currently looks like this:

Llama 3.1 Models from Meta

Meta has released an update to its Llama 3.1 series that includes 3 model sizes: 8B, 70B and 405B. You can download all models from HuggingFace and use them locally. Most AI providers also offer support via API.

We tested smaller models locally and used Google Vertex AI for 405B. Google almost didn’t mess up the integration (you may need to fix the line breaks and truncate extra tokens at the beginning of the prompt).

The 8B model is not that interesting - it scores lower than the previous 3.0 version, so we’ll skip it. The other two models are way more interesting.

Meta Llama 3.1 70B has made a massive jump in quality when compared to the previous version. It has reached Gemini Pro 1.5, surpassed GPT-3.5 and reached Mistral Large 123B v2. This is great news because we can achieve the quality of the 123B model with a smaller one.

Note, by the way, that Llama 3.1 models can be quite sensitive to quantization (compression). For example, if we run a 70B model with an 8bit quantization (over bitsandbytes), the performance and quality will drop drastically:

This does not mean that all quantization strategies are equally bad (you can find a good article on this topic here). Just make sure you compare your model on your hardware with your specific data.

Meta Llama 3.1 405B Instruct

Meta Llama 3.1 405B Instruct is the last hero of the month. This is the first model that managed to beat the GPT-4 Turbo (its weakest version Turbo v3/1106). You can find it in the TOP 10 of our benchmark:

It is a large model. You need 640GB of VRAM (8xH100/A100) just to run it in FP8 with a small batch and context window. The resource requirements alone mean that very few will use this model when compared to 70B/8B variants. There will be less interesting fine-tuning and solutions.

But that's not all that important. The important points are:

• This is a model that you can download and use locally.

• It outperforms one of the GPT-4 models

• It beats Mistral Large 2 in quality while having a more permissive license

• It reaches the quality of Claude 3 Opus.

This is a small breakthrough. We are sure that smaller models will also reach this level at some point.

Update: Google Gemini 1.5 Pro Experimental v0801

Normally we don’t do benchmark updates after the publication, but this news deserved it. Waiting for one month to report on the new Google Gemini model would be a waste.

This model was released as a public experiment on the first of August (you can find it in the Google AI Studio ). At this point it was also revealed that the model has been running for some time on LMSYS Chatbot Arena, scoring on the top with more 12k votes.

We ran our own benchmark using the Google AI Studio API (the model is not yet available on Vertex AI). The results are really impressive. We are talking about a substantial jump in model capabilities from the first version of Gemini Pro 1.5 in April.

This Google model managed to suddenly overtake almost all GPT-4 models and catch up with the TOP, taking the third place. The scores are quite solid.

Scores could’ve been even better, if Gemini Pro 1.5 paid more attention to following instructions precisely. While extreme attention to the detail isn’t always needed in human interactions, it is essential in products and LLM pipelines deployed at our customers. Top two models from OpenAI still excel in that capability.

Still the news is outstanding, worth the celebration. First of all, we have a new source of innovation that managed to catch up with OpenAI (and we thought that Google was out of the race). Second, companies deeply vested into the Google Cloud will finally get access to top quality large language model within the ecosystem.

And who knows whether Google Gemini 2.0 will manage to increase its modeling capabilities even further. The pace of progress so far has been quite impressive. Just see for yourself: