Browser-native inference · WebGPU
A language model, running on your own graphics card.
Enargeia loads a 0.5-billion-parameter model into your GPU and generates text there. The compute kernels are hand-written WGSL in this repository — no ONNX Runtime, no transformers.js, no WebGPU wrapper. Nothing you type is sent anywhere, because there is no server to send it to.
Run it
The machinery is visible while it runs.
Loading fetches 334.9 MiB of int4 weights, once, and caches them. The inspector reads
the same telemetry the engine publishes to itself: GPU time per kernel from
timestamp-query, cache occupancy, the memory ledger, and attention weights
as they are computed.
The throughput you see here will read lower than the 45.5 tok/s in the table below, and should. That figure is greedy decode in a headless browser with nothing else on the GPU; this is sampled decode in a foreground tab sharing the GPU with the compositor, and the page itself costs about 4 ms per token. Measured on an M2: 34 tok/s here against 45.5 there.
It reads every weight in the model to write one word.
494.0M parameters · 412 dispatches per token · 24 layers · one GPU→CPU readback, carrying a single token id
Four bits per weight, unpacked in registers and never written down.
334.9 MiB of weights against 1884.6 MiB for the same weights in fp32 — 5.63× smaller. Live residency is 458.0 MiB: those weights, a 24.0 MiB KV cache, and 99.1 MiB of activation scratch. Blocks of 64 along the reduction axis, each with its own scale and zero point. Dequantizing into a buffer would move the same bytes the quantization exists to avoid
A long conversation costs no more per word than a short one.
43.1 tok/s at 2048 against 45.5 at 512. The position-dependent term measured −0.15 µs/position, indistinguishable from zero; before the attention kernel stopped walking history one position per thread it was +3.89
The output is identical every time you run it.
Greedy decode reproduces HuggingFace's tokens exactly, and prefill-then-decode
reproduces the no-cache path 24 of 24. Non-determinism is treated as a
correctness bug, not as noise — it is almost always a missing
workgroupBarrier()
And it is a small model, so it is not very good.
Perplexity 36.21 against 31.90 for the same weights in fp32, over 1,256 held-out positions. Quantization costs about 13.5%; being a 0.5B model costs far more than that
One decode step
412 compute passes, in the order they are encoded.
This is the dispatch list, not a diagram of a transformer — it is what
graph_decode.ts builds once at load and re-encodes for every token. On a
device with shader-f16 there are 460, because caching K and V in half
precision adds a pack dispatch per layer.
Measured, not asserted
Every number here was measured, including the ones that went the wrong way.
Full method and history in BENCH.md. Device: Apple M2, headless Chromium over CDP, so kernels are compared like for like. A foreground tab shares the GPU with the compositor and measures about 15% lower — the demo above is the honest number for what you will see.
| Measurement | int4 | fp32 * |
|---|---|---|
| decode, 512-token context | 45.5 tok/s | 26.9 |
| decode, 2048-token context | 43.1 tok/s | 21.6 |
| prefill, 128 tokens | 1105 tok/s | 225 |
| prefill, 2048 tokens | 797 tok/s | 212 |
| time to first token, 32-token prompt | 219 ms | — |
| cold first load, R2 over CDN | 14.6 s | — |
| second visit, from cache | 1.4 s | — |
| weights resident | 334.9 MiB | 1884.6 MiB |
| KV cache at 2048, f16 | 24.0 MiB | 48.0 MiB |
| activation scratch | 99.1 MiB | — |
| live residency, everything | 458.0 MiB | — |
| perplexity, 1,256 held-out positions | 36.21 | 31.90 |
* The fp32 decode figures are the M5 baseline and were not re-measured after the M6 kernel work. That work — right-sizing dispatch and parallelising the attention history reduction — is independent of weight precision and would lift the fp32 column too, so the decode comparison overstates int4's advantage. The prefill figures are both post-fix and comparable. int4's genuine decode advantage, measured in the same era on both sides, is 1.34× to 1.54×.
The two biggest wins were not on the list of planned optimizations.
Right-sizing prefill dispatch: 11.6× at 128 tokens, from a graph that launched 5.5M workgroups whatever the prompt length. Parallelising the attention history reduction: +50.4% decode at 2048. Of the five items actually planned, two measured null, one had its premise refuted, and one was bounded below the noise floor and skipped
A tiled matmul is worth five naive ones, and the fifth rung is where it stops.
219.4 → 518.2 → 838.9 → 870.8 → 1141.7 GFLOP/s at 1024³. The 8×1 coarsened kernel bought 4% over the 4×1 one; going 2D instead — a 4×2 register block, 0.75 shared loads per MAC — bought 36%
Honestly
What this is not.
Is the model any good?
No. Qwen2.5-0.5B-Instruct is roughly a thousandth the size of a frontier model and behaves like it: it loses the thread over a few paragraphs, invents facts with total confidence, and cannot do arithmetic reliably. If you want a good answer to something, ask a frontier model. This page is about the engine underneath, which would run a better model the same way if a better model fit in a few hundred megabytes.
Should I use this in production?
No — use WebLLM. It supports many more models, is far better tested across devices, has real batching and grammar-constrained output, and is maintained by people who do this full time. Enargeia exists because writing the kernels is the interesting part, and because an engine you can read end to end teaches you things a compiled runtime does not.
What does int4 actually cost in quality?
Perplexity goes from 31.90 to 36.21 on 1,256 held-out positions — about 13.5% worse. An earlier, smaller evaluation over 95 positions measured 30.25 against 35.22; the larger sample is the one to trust, and it also corrected a sign the smaller one had backwards.
An ablation on the tied embedding says where the damage is: keeping that one tensor at int8 costs 66 MiB and recovers 99.95% of what a full fp32 exemption recovers, so that is what ships. The remaining gap is spread across the other 169 quantized tensors, and no amount of embedding precision touches it.
Why is the first load so large?
Because the weights are the model. 334.9 MiB is 494 million parameters at four bits each, plus an int8 embedding table and the per-block scales. It is cached after the first visit — a second visit takes 1.4 seconds and two requests, against 14.6 seconds cold at 28.7 MB/s.
That figure is the weights alone. Everything the engine holds on the GPU — weights, a 24.0 MiB KV cache at a 2048-token context, and 99.1 MiB of activation scratch — comes to 458.0 MiB, which the inspector shows while it runs.
Does my text leave the browser?
No. The only network requests this page makes are for the model file, the tokenizer, and its own JavaScript. There is no inference backend, no analytics on your input, and no request carrying anything you type. You can verify that in the network tab, which is a better assurance than a privacy policy.
Why does it need WebGPU specifically?
Because the engine is compute shaders. WebGL has no compute stage, and WASM on the CPU is roughly two orders of magnitude off for this shape of work. Chrome and Edge 113+, Safari 26+, and Firefox 141+ on Windows have it; where it is missing the page says so and explains what it would have done.
Why "enargeia"?
A term from ancient rhetoric for description vivid enough that the audience sees the thing rather than hears about it. The whole point of the project is that the machinery is visible while it runs.