Open-Weight Models

Every local-model writeup asks which model is best. That is the wrong question. Most of what I hand a model is either a judgment between a few options, which does not need text generation at all, or plumbing that does not need a model. The interesting question is the split.

Three buckets. Generative work (drafting, coding, explaining) still goes to a big hosted model, or a mid-size open-weight one when it can stay on the machine. Judgment work (classify this, pick one of these, is this a yes) goes to a decision model that returns an answer and a calibrated confidence in one forward pass. Everything else goes to code. Getting the buckets right should cut cost and latency more than picking a better chat model ever would.

Ollama runs the open-weight generative models locally: Qwen and Gemma. Kimi's weights are open, but at roughly a trillion parameters it is not a laptop model, so it runs through the vendor's API, which is the honest middle category of open weights with hosted inference. The decision side uses Jev-style models, which Ollama now runs alongside chat models: Laya, a small encoder that answers choice, score and yes/no questions with a probability attached. The hosted half started as the AI Gateway probe in this repo's experiments folder and has since moved to T3 Code, which drives several subscriptions from one place.

Running. Two findings so far. The hosted wall was entitlement, not authentication: on Vercel AI Gateway, OIDC alone authenticated cleanly, the smallest model answered at once, and the bigger ones came back 403 (not allowed) or 429 (allowed, throttled), which are opposite answers that most code treats as one. And the decision models are the sleeper: a few hundred million parameters that return a number are a better fit for the judgment step in an automation than a chat model asked to say yes or no.