The best Large Language Models of September 2024

LLM Benchmarks | September 2024

The benchmarks evaluate the models in terms of their suitability for digital product development. The higher the score, the better.

☁️ - Cloud models with proprietary license
✅ - Open source models that can be run locally without restrictions
🦙 - Local models with Llama license

ChatGPT o1 models are the best

OpenAI has released a radically new type of the model called o1-preview that is followed by o1-mini. These unique models differ from all the other LLM models out there - they run their own chain of thought routine for each request. This allow the model to decompose complex problems in smaller tasks and really think the answers through.

That approach, for example, shines in complex full-stack software engineering challenges. o1, if compared to the “ordinary” GPT-4 feels like an experienced Middle Level Software Engineer that requires surprisingly little hand-holding.

There is one downside in this “chain of thought under the hood” process. O1 produces high quality results, but these results take time and cost a lot more. Just look at the comparative pricing within the Cost column.

We are looking forward to see other LLM vendors take a note of this trick and release their own versions of LLMs with tuned chain-of-thought routine.

Google Gemini 1.5 Pro v 002 - TOP 3

While speaking of the top results and cloud vendors, there is another new model in the TOP-3. Google has somehow managed to catch up with the rate of the progress and release highly competitive model - Gemini 1.5 Pro v 002.

This model systematically improves over the previous version in multiple categories: Code, CRM, Docs, and Marketing texts. It is also the cheapest model in the TOP-6 of our benchmark.

Practitioners already praise this model for great multi-lingual skills, while users of Google Cloud are happy to have top-tier LLM available in their cloud.

For a long time, it felt like OpenAI and Anthropic are the only companies that can really push the state of the art in top-tier LLM models. It also felt like large mammoth companies are just too slow and old-school to release something worthy. Google was eventually able to prove this wrong.

This is how the progress of Google models looks over the time:

Now it doesn’t feel out of the ordinary to expect models of similar quality from Amazon or Microsoft. Perhaps, this will spur forward a round of competition with further price drops and further quality improvements.
Enough with the cloud vendors. Let’s talk about local models now.

(Local models are the models that you can download and run on your own hardware.)

Qwen 2.5 and DeepSeek 2.5

Recently released Qwen 2.5 Instruct is surprisingly good. This is the first local model that beats Claude 3.5 Sonnet on our business tasks. It also costs less than the other LLM models in the top.

Starting from this benchmark we’ll use OpenRouter pricing as the base price for locally-capable LLM models. This allows to estimate workload costs based on the real-world market. It also factors in any meaningful performance optimisations that LLM vendors are willing to use to improve their margins.

Qwen 2.5 72B diligently follows instructions (if compared to Sonnet 3.5 or older GPT-4 versions) and has a decent Reason capability. This Chinese model has gaps in Code and Marketing capabilities.

DeepSeek 2.5 didn’t perform nearly as well in our product benchmarks, despite having a huge size of 236B parameters. It runs roughly on the level of older versions of GPT-4 and Gemini 1.5 Pro.

These actually are outstanding news: more and more local models reach the level of GPT-4 Turbo intelligence. And the fact that a smaller Qwen 72B model has beaten it by a big margin - is worth a separate celebration 🚀

We think, this is not the last celebration of this kind for this year.

Llama 3.2 - Mediocre results, but there is a small nuance

Meta has just released their new versions of Llama - 3.2 model range.

Larger models are now multi-modal. This happened at the cost of the cognitive capabilities in text-driven business tasks, if compared to the previous model versions. Llama 3.2 is still far from the top.

If we look at the table:

• Llama 3.2 90B Vision works on the level of Llama 3/3.1 70B but with worse Reason.

• Llama 3.2 11B Vision works on the level of previous 8B, but with worse reason.

This doesn’t make the new models worse - they have more capabilities now. Our benchmark currently tests only text-based business tasks. Vision tasks will be added in v2.

Having said that, there is a small nuance that really makes this Llama 3.2 release outstanding. Size of that nuance is 1B and 3B. These are the sizes of new tiny Llama 3.2 models that are designed to run in resource-constrained environments and on the edge (optimised for ARM processors, Qualcomm and MediaTek hardware). Despite resource constraints, these models feature 128k token context and surprisingly high response quality in business tasks.

For example, do you remember a huge DBRX 132B Instruct model that claimed to be “a new state-of-the-art for established open LLMs”? Well, Llama 3.2 1B model catches up with it in our benchmark and 3B beats it by a big margin. Just look at the neighbours of these models on this table:

Keep in mind that these benchmarks results are for the base Llama versions. Customised fine-tunes tend to improve overall scores even further.

As you can see, the progress doesn’t stand still. We’ll be waiting for the continuation of the trend where more and more companies manage to package better cognitive capability in smaller models.

To visualise such a trend, we’ve plotted all releases of locally-capable models over the timeline. Then we’ve grouped them together based on the rough hardware requirements for running them. For each group we’ve computed current trend (linregress)

Note: this grouping is very rough. We are using for most frequent hardware combinations that we’ve seen among our customers and within the AI Research. We are also assuming that we are running inference under fp16 without any further quantisations and with enough spare VRAM to keep some context around.

Here are a few observations.

• All models are getting better over the time - both small and big ones.
• Interesting large models started showing up on the radar only this year.
• Large models currently have the fastest rage of improvement.

These observations are obvious. You don’t need a chart to figure them out. However visualisations make rate of the progress more comprehensible. It could then be translated to customers and accounted for in long-term plans.

Transform Your Digital Projects with the Best AI Language Models!

Discover the transformative power of the best LLMs and revolutionize your digital products with AI! Stay future-focused, boost efficiency, and gain a clear competitive edge. We help you elevate your business value to the next level.