Promised weights are not open weights

Three model launches crossed this site’s desk in the past week with the phrase “open-weight” attached. I went looking for the weights.

JetBrains’ Mellum2.1 has them. There is a Hugging Face repository, an Apache 2.0 license, and files you can pull right now.

Mistral Large 4 has an API preview and a promise of weights by the end of the month. The license has not been named.

Reflection’s Beam has a waitlist and a promise of Apache 2.0 weights later this month.

One out of three. I do not think the other two labs are being dishonest, and I expect both to ship. But I would like us to stop using the present tense for things that have not happened.

What can change before release day

A promise of open weights leaves three things undetermined, and all three matter more than the announcement.

The license. “Open-weight” describes a file you can download. It says nothing about what you may do with it. Apache 2.0, a research-only license, and a custom license with a revenue cap are very different offers. Mistral has not said which one this will be. Until it does, nobody can plan a product around the model.

The model. Mistral says Large 4 is still in reinforcement-learning training and its numbers may change. Reflection revised Beam’s results three days after publishing them. That is normal for a preview. It also means the benchmark tables being passed around describe models that are not the ones you will eventually download.

The details. Mistral’s card lists a 1M-token context window; the preview is being served at about half that. Its active parameter count went from 49 billion in launch-day coverage to 52 billion now. Small things, each with a boring explanation, I am sure. They are the kind of small things you can only check for yourself once you have the files.

Why I care

The value of open weights is not that they are free. It is that they are checkable and permanent. You can run the evaluation yourself and see whether you get the vendor’s number. You can keep using the model after the company changes its pricing, or its mind. You can look inside.

A hosted preview offers none of that. For now, Large 4 and Beam are in the same category as any other model you rent by the token. They may be very good ones. Beam’s own table is admirably frank about where it trails. But “open” is a claim about what you can verify, and a promise is exactly the thing you cannot.

There is also a quieter question about the big ones. A trillion parameters is roughly half a terabyte of weights even when squeezed to four bits each. When those files appear they will be open in the sense that matters for audits and for companies with a rack of GPUs. They will not be open in the sense that you, reading this, can run them. Mellum2.1 is nearly a hundred times smaller, and I suspect it will be opened, in the literal sense of somebody loading it up and poking at it, by far more people.

What this site will do about it

The Open Weights page has a tracker with a status column. A model is marked downloadable only when there are files to download. Right now that is one row out of three.

I would be glad to see the other two change. I will believe it when there is a link.