Running my models in the browser

Two of the demos on this site run entirely on your device: the Portuguese fake-news classifier and the acacia segmentation model. There is no inference server behind them. The page downloads the weights and runs the model in JavaScript and WebAssembly.

I went this way for three reasons. A server would cost money and go to sleep, and a demo that takes thirty seconds to wake up is a demo nobody waits for. Static files are free to host and never cold-start. And nothing you paste or upload leaves your machine, which matters when the input might be your own photos.

The two models needed very different approaches.

The MLP: no runtime at all

The fake-news model is a scikit-learn pipeline: a TfidfVectorizer over 5,000 terms feeding an MLPClassifier with one hidden layer of 256 ReLU units. Run forward, that is one sparse matrix product, a ReLU, a dot product and a sigmoid. No ML runtime is needed for that. It fits in about thirty lines of TypeScript.

The network was the easy part. The preprocessing is where a port goes quietly wrong. If the browser tokenises even slightly differently from Python, every downstream number shifts, and nothing errors. What I had to match:

• Accents. strip_accents="unicode" decomposes each character (NFKD) and drops the combining marks, so “notícia” becomes “noticia”. In JavaScript that’s normalize("NFKD") followed by removing \p{Mn}.
• Tokens. scikit-learn’s default token pattern is \b\w\w+\b, two or more word characters. In Python 3, \w matches any Unicode letter. In JavaScript it only matches ASCII, so a straight copy would split “coração” into pieces. The port uses [\p{L}\p{N}_]{2,} with the u flag.
• Stop words. Tokens are checked against the stop list after their accents are stripped, so the list has to be stored without accents too. Otherwise “não” never matches “nao”, and one of the most common words in Portuguese slips through.
• Normalisation. TF-IDF vectors are scaled to unit length (L2) before they reach the network. Forget that and long articles look more confident than short ones.

The weights are the other problem. The first layer is 5,000 × 256, about 1.28 million numbers, or 5.1 MB as float32. I exported it as float16 instead: 2.56 MB, decoded back to float32 once in the browser with a few bit operations. Half precision keeps about three significant digits, which is plenty for weights that end up in a sum of a few hundred terms. On the reference articles I checked, the browser’s probabilities match Python’s to four decimal places.

The highlighting in the demo comes from occlusion. Remove one word, run the model again, and see how far the probability moves. It sounds expensive, but the input is sparse. A forward pass only touches the rows of the weight matrix for words that actually appear, so an article with 80 known words costs 80 × 256 multiply-adds per pass. Doing that once per word is still instant.

YOLOv8: a runtime, and Ultralytics’ post-processing by hand

The acacia model is a fine-tuned YOLOv8n-seg, a real convolutional network, so here I did need a runtime. I exported the checkpoint to ONNX (13 MB) and run it with ONNX Runtime Web on its WebAssembly backend.

The surprise was that the runtime is bigger than the model. The WASM binary is 14 MB, so the first run downloads about 27 MB, or roughly 15 MB compressed. Two things keep that from hurting:

• Nothing loads until you pick an image.
• The runtime is served from this site rather than a CDN, so it is cached with everything else and the bundler never tries to process it.

The ONNX graph stops at the raw network outputs. Everything Ultralytics does after that, I had to rewrite:

1. Letterbox. Scale the image to fit 640 × 640, keep the aspect ratio, and pad with grey (114, 114, 114), exactly as in training. Different padding means a slightly different input and different scores.
2. Decode. The main output is 37 × 8,400. That’s 8,400 candidate boxes, each with 4 box coordinates, 1 class score (there is only “acacia”) and 32 mask coefficients.
3. Filter and suppress. Drop candidates below the confidence threshold, then run non-maximum suppression at IoU 0.7, as Ultralytics does by default.
4. Masks. The second output is 32 prototype masks at 160 × 160. Each detection’s mask is its 32 coefficients times those prototypes, passed through a sigmoid and cropped to its box. A sigmoid above 0.5 is the same as a logit above 0, so the browser just checks the sign and skips the exponentials.

On the six sample tiles, the browser version finds the same detections as Ultralytics’ own predict. Each tile takes about 330 ms in Chrome on a laptop.

That number is single-threaded. ONNX Runtime can use several threads, but multithreaded WebAssembly needs SharedArrayBuffer, which browsers only allow on cross-origin-isolated pages. That means two extra response headers, and every resource on the page has to opt in. I haven’t done it yet.

What I’d change

• Quantise the detector. An int8 version of YOLOv8n would be roughly a quarter of the size. I’d want to re-check the validation scores first, because a model this small has little capacity to spare.
• Try WebGPU. ONNX Runtime’s WebGPU backend would move the convolutions to the graphics card where it’s available, and keep WASM as the fallback.
• Turn on cross-origin isolation for the demo pages, to get threads.

None of this changes the main lesson. Getting a model into the browser is mostly about making the code around it match the original exactly: tokenisation, padding, thresholds. Once those match, the network itself is usually the easy part.