In short 1 breakthrough and two tactical decisions. Let’s start with the breakthrough which was to match the best OpenAI model available (o3 is not yet released) only 4 months after its announcement and 1 month after its release and before Google, Meta or Anthropic 😮 How did they do that?

Instead of using humans to annotate reasoning data which takes time and money, they relied on the fact that reasoning problems have a know right or wrong answer so they can use their model to generate answers and use the answers to improve their model in a loop 🔁 Making that work requires some tricks which was the breakthrough 🔥

Now for the tactical decisions. The first was that they decided to make the details and model open which closes any gap OpenAI had with their reasoning models in a short period of time. As a reminder it took almost a year to match GPT4 from a proprietary provider and another six months to have an open source equivalent.

The other was to use a couple of the latest innovations like multi token prediction, 8bits, more experts with less active parameters to reduce the cost of serving the model by 100x 🚀 Note this is not the cost of training but the cost of using the model. The cost of training, unlike what’s been reported, is not that dissimilar to investments from other companies.

Put all of those things together and you have the best available model right now being open source, 100x cheaper to use and from a Chinese company 🇨🇳