The best language models for digital products

Based on real benchmark data from our own software products, we evaluated the performance of different LLM models in addressing specific challenges. We examined specific categories such as document processing, CRM integration, external integration, marketing support, and code generation.  

The highlights of the month:

• The elephant in the room - Claude 3.5 Sonnet and artifacts feature

• Confidential computing - how it can make AI more secure and cost-effective for companies

• The trend towards small and powerful LLMs that can be operated locally

Our benchmarks evaluate the models in terms of their suitability for digital product development. The higher the score, the better.

☁️ - Cloud models with proprietary license
✅ - Open source models that can be run locally without restrictions
🦙 - Local models with Llama2 license

A more detailed explanation of the respective categories can be found below the table.

Claude 3.5 Sonnet - Anthropic did it again

Remember how Anthropic made a big quality improvement in their models in March?

They have just done it again by releasing Claude 3.5 Sonnet. This mid-range model is not only more powerful than the top-of-the-range Opus model, but also about five times cheaper.

Improved performance with Claude 3.5 Sonnet

Claude 3.5 Sonnet better follows instructions and has same reasoning capabilities as their top model - Haiku, so this is a huge improvement.

NEW: ARTIFACTS FOR A BETTER USER EXPERIENCE

There is one more big improvement in the product line of Anthropic, though. It is called Artifacts, and it isn’t even about LLM capability, but rather about user experience and LLM integration.

ARTIFACTS: WORKING EFFICIENTLY WITH DOCUMENTS AND CODE

The idea of Artifacts is: when you are working on some document or a piece of code, Claude web chat, will pull this document into a convenient separate window. This document will now become an entity of its own, not just a snippet that is repeated in the web chat. Artifacts are versioned, and you can properly iterate on them.

This may seem like a small feature, but together with Claude 3.5 Sonnet, it becomes a huge productivity boost that makes it worthwhile to use Claude Chat instead of ChatGPT when working with documents and code snippets.

Small, efficient models are getting better and better

Last month we tested several local LLMs. There were some pleasant surprises:

First of all, it was about Google Gemma 7B Instruct. This Google model is often criticized for being too restricted and limited.

However, the OpenChat 3.5 fine-tuning of this model reveals its true capabilities and places this 7B model above the first version of GPT-3.5.

It is rumored that GPT-3.5 had about 20-175B parameters, and this small 7B model (which can run on a laptop) manages to outperform it! The rate of progress is impressive.

In fact, the only local LLM that performs better than this model (in our benchmarks) is AliBaba's Qwen1.5-32B model. However, this model has a non-standard license and requires more than four times as many resources to run.

As you can see from the picture, there are already many 7B models with performance comparable to early versions of GPT-3.5. Based on the trends, the progress will not just end there.

Poorer performing models

Not all local models performed so well in our benchmark. Here are some that performed poorly (mostly because they couldn't follow even basic instructions accurately):

- Yi 1.5 34B Chat

- Google Recurrent Gemma 9B IT

- Microsoft Phi 3 Mini/Medium

- Google Gemma 2B/7B

Apple Privacy Model and Confidential Computing

In its latest announcement, Apple has started to introduce more AI features to its ecosystem. One of the most interesting aspects was the concept of Private Cloud Compute.

Essentially, the iPhone will use a small and efficient LLM model to process all incoming requests. This LLM is not very powerful and comparable to modern 7B models. However, it is fast and will process all requests in a secure way - locally.

It becomes particularly interesting when the LLM-controlled system recognizes that it needs more computing power to process the request.

In this case, it has two options:

• It can ask the user for permission to send the specific request to OpenAI GPT.

• It can securely forward the request to a private cloud compute managed by Apple.

What is private cloud compute?

It is a protected Apple datacenter that uses their own chips to host powerful Large Language Models. The setup gives strong guarantees that your personal requests will be handled securely and nobody, not even Apple, will even see questions and answers.

This is done through a combination of special hardware, encryption, secured VM images and mutual attestation between the software and hardware. Ultimately, they do their best to make it very hard and expensive to break this setup even by Apple or governments.

Apple is all about consumer electronics, is there anything comparable for companies?

Yes, it does exist. It's called confidential computing. The concept has been around for some time (see the Confidential Computing Consortium), but has only recently been properly applied to GPUs by Nvidia. Nvidia introduced it in the Hopper architecture (H100 GPUs) and almost completely eliminated the performance penalty in the Blackwell architecture.

The concept is the same as Apple's PCC:

• data is encrypted in transit and at rest

• data is decrypted during the computation time

• hardware and software are designed to make it impossible (really hard and expensive) to take a look at the data while it is decrypted.

Major cloud providers are already testing VMs with confidential GPU calculation (e.g. Microsoft Azure with H100 since 2023, Google Cloud with H100 since 2024).

This approach is interesting because it offers a third option to companies that need to build a secure LLM-driven system:

Just like with hybrid clouds (they were a big thing in the past, but are a norm these days), we can mix-and-match these options for a cost-effective and secure solution, just like Apple does with PCC. For example:

• Have a small local deployment that runs cost-effective 7B models on our own hardware. It will handle all requests locally.

• If a user request needs more powerful AI/LLM and doesn’t involve critical information - route requests to Azure OpenAI

• If a user request is both sensitive and requires a lot of GPU compute, then - route it to a confidential compute in the cloud.

Ultimately, if the powerful-and-confidential workload is steady enough, it might make sense to add a few local and powerful GPUs to handle it. During the peaks we can still rent confidential compute in the cloud.