A live, local experiment
sarvam-1 runs on your own machine, in this tab. It can either read probabilities for the options you allow, or write the same answers token by token as JSON. Load it once, run both, and measure the difference yourself.
Nothing to install and nothing to sign up for. Your ticket never leaves this page — there is no server to send it to. ↓
sarvam-1 Q4_K_M is selected by default and reads a five-criterion ticket in about six seconds on a desktop GPU. A phone is roughly ten times slower — measured at about fifty seconds for the same ticket — so on mobile pick Qwen3 0.6B below, keep the number of criteria small, and expect to wait. The lane reports which criterion it is on and how long is left.
00 / setup
Checking what this browser supports…
Weights come from Hugging Face and stay in your browser cache, so a second visit is instant. First load can take several minutes depending on your network. The quality figures in the repository are for native BF16 weights; these builds are quantized and have not been separately measured.
01 / decision
State 0 characters
Prefilled once. Every criterion below reuses it.
Criteria 0 rows
02A / direct readout
Each criterion is one constrained readout: a softmax over the logits of its own answer letters. Nothing is sampled, so no answer can be malformed or borrow another criterion's options.
02B / generation
The same decisions asked for as a JSON object, decoded greedily. Every token is one forward pass, and the object is only usable once the last brace lands.
waiting for a run
Measured wall-time ratio
Load a model, then run both methods. They run one after the other on the same loaded weights so they never contend for your GPU.
02 / how it works
The answer-letter ids are listed in models.json, not discovered
here: this build of wllama exposes no tokenizer to probe with. They were derived beforehand
by probing both Hugging Face's tokenizer and llama.cpp's and checking they agree, and they
are not contiguous in either vocabulary, so the usual shortcut of assuming consecutive ids
would be wrong. The check that survives in the page is downstream — a readout that comes
back without a score for every declared option is refused rather than shown.
Not the server's parallel path. The Python engine prefills the state once and evaluates every criterion in a single batched pass, which is where its 8.8× at twenty criteria comes from. wllama exposes no batched branch evaluation, so criteria here are scored one at a time with the shared prefix cached between them. Expect a smaller ratio than the repository reports.
Quantized weights. Every published accuracy figure is for native BF16. These are 3-, 4- and 8-bit builds; quantization can move both quality and speed, and no separate measurement has been made.
Conditional probabilities. The readout is a softmax over only the option letters you supplied. It is not calibrated confidence, and a high number does not establish that the decision is correct.
A 2B base model. sarvam-1 was never instruction-tuned. On openjev's authored fixture it scores 0.516 mean family balanced accuracy against a 0.333 chance floor — well above chance, well below instruction-tuned models of the same size. Expect visible mistakes.