ai-raccoon is a background server. It runs all day next to an IDE, a build and several agent sessions, and nobody looks at it until it gets in the way. In September it got in the way three times.

A freshly started server sat at about 10% CPU. After a machine restart that lasted more than an hour. With agents building in the repos it watched, it never really stopped. Activity Monitor showed it at 6.8 GB on a 24 GB laptop, with only 537 MB of that actually in RAM and the rest pushed out to swapDisk space the OS uses when RAM runs out. Memory in swap still belongs to the process, and touching it again is slow.. And every embedding it computed ran on the same CPU cores the compiler and the IDE wanted.

Between releases 1.44.3 and 1.51.2 that changed. The same server now sits at 0.3% CPU in ps, holds 1.75 GB, and computes its embeddingsA list of numbers (384 for the current model) that stands for the meaning of a piece of text. Similar texts get vectors that point in similar directions, so search becomes: find the nearest vectors. on the GPU. Search quality did not move. This post walks through what was wrong, what changed, and the vocabulary you need to follow the GPU part: ONNX, execution providers, WebGPU, MLX and why a transformer model is a GPU workload in the first place.

The full dated report, with every measurement and its source, lives in the ai-raccoon repo: Performance report: CPU, memory footprint and the move to the GPUThe dated source report in the ai-raccoon repo: every table, chart and caveat behind this post, with links to the PRs, ADRs and release checklist..

1. CPU: stop paying for files nobody indexes

ai-raccoon watches project directories so it can keep its code index current. Profiling the live server (macOS sample, dotnet-trace, dotnet-counters) showed that the watcher was doing a lot of work for files it would never index.

Every write under .git, .ai-badger, bin or obj ran the full delete cascade for that path. Those directories are excluded, so nothing from them is normally stored, and the cascade almost always deleted nothing. In a three-minute trace there were 110 deletes. 106 of them were for excluded paths, and all 110 together removed 4 rows. Each took 32 ms at the median and up to 863 ms, and together they held the database write lock for 10.5 seconds. Everything else that wanted to write queued behind them.

The delete itself was expensive for a reason SQLite users will recognise. The cascade matched path = @path OR path LIKE @prefix. The LIKE half cannot use an ordinary index, because SQLite's default LIKE ignores case (the query optimizer docsSQLite's own description of how its query planner picks indexes, including when LIKE and OR terms can and cannot use one. list the narrow cases where it can), and there was no (project_id, path) index yet anyway. So SQLite read every row of the project, embeddings included. After a reboot, with a cold page cache, every one of those reads went to disk. That was the "slow for an hour after restart" symptom.

The third cost was connection churn. The watch loop opened the database about 29 times per second, and each open reloaded the vec0 vector-search extension from sqlite-vecAlex Garcia's SQLite extension that adds the vec0 virtual table for vector search. ai-raccoon stores its embeddings in it. and re-ran schema checks.

Release 1.50.1 (PR #741The 1.50.1 pull request: skip excluded-path cascades, index-seek deletes, and a pooled SQLite connection. Includes the review thread.) fixed all three. An excluded path now takes the cascade only if something is actually stored at or under it, which is a single key lookup. Every path cascade became two index seeks on a new (project_id, path) index: one for path = @p, one for the range path >= @p/ AND path < @p0. The range trick works because 0 is the character right after /, so the range holds exactly the paths under the directory and never a sibling like docs-old. And a pooled SQLite connection loads vec0 once and skips the version and digest checks unless the database changed underneath it. The steps documented as running on every open still do; an earlier cut of the patch skipped them and review blocked it.

Side by side, on two copies of the live bank (14 watches, about 50k entries) receiving the same real file events while other agents built and committed, 1.50.0 used 24.8 to 117.1 CPU-secondsHow much processor time a process used. 1 CPU-second over a 60-second window is about 1.7% of one core. per two-minute window and 1.50.1 used 1.2 to 2.8. My own server went from averaging 69% of a core over its first 14 minutes to about 2%.

2. Memory: the 6.8 GB that was not .NET

The first guess for a .NET process holding 6.8 GB is the managed heap. It was 213 MB live (1.45 GB committed). vmmap put 6.0 GB in native malloc, 5.1 GB of it swapped out. The memory belonged to ONNX Runtime, the library that runs the embedding model.

Two things drove it. The model at the time, SFR-Embedding-Code-400M in fp3232-bit floating point: 4 bytes per weight. fp16 uses 2 bytes and int8 uses 1. Smaller means less memory and faster math, and at some point different answers., is 1.75 GB on disk and took 2.6-2.7 GB just to load. And every embedding call sent a batch of 32 chunksOne slice of a document that gets its own vector. Long files are split into chunks before embedding. of 510 tokensThe unit a model reads. For English a token is roughly three quarters of a word, so 512 tokens is about 380 words.. On a CPU a batch is not faster per item than a single row, but its intermediate buffers are 32 times larger:

macOS malloc keeps large freed blocks around, so a single 32-row peak stayed with the process for good. Most of it then went to swap, which is why RSSResident set size: the part of a process's memory that sits in physical RAM right now. It leaves out anything swapped or compressed. looked harmless while the physical footprintWhat macOS charges to a process wherever the memory lives: resident, compressed, swapped, or GPU buffers. It is the Memory column in Activity Monitor. did not.

Three changes brought it down. Release 1.44.3 (#669The 1.44.3 pull request that sends one row per ONNX call instead of a batch of 32.) sends one row per ONNX call; the database still pages 32 rows at a time, only the model call changed, and a test checks the single-row vector is bit-identical to the batched one. Release 1.47.0 replaced the model with IBM's granite-embedding-small-english-r2 in fp16: 97 MB on disk, 252 MB after inferenceRunning a trained model to get an output. ai-raccoon never trains anything, it only runs inference., and it won or tied every retrieval eval against the old one. The third change was for the MLX engine, below.

If you want to see what is inside that small model (12 ModernBERT layers, 384-number vectors, 47M parameters) and how ONNX Runtime executes it step by step, I wrote an illustrated walkthrough: Granite on ONNX RuntimeIllustrated walkthrough: how ONNX Runtime loads and runs granite, the 12-layer model inside it, its attention pattern, and what happens on each embed..

3. How a transformer ends up on a GPU

Before the GPU part, some background, because "move it to the GPU" hides several layers of software.

Why transformers suit GPUs

An embedding model like granite is a transformer encoderThe half of the transformer architecture that reads a whole input at once and produces one vector per token. BERT and its descendants are encoders.. Each token starts as a vector, and each of the 12 layers updates every token's vector in two steps. Attention lets each token look at the other tokens and mix in what is relevant. A small feed-forward network then transforms each token on its own. Both steps are almost entirely matrix multiplications: the same multiply-and-add applied to thousands of numbers at once.

A CPU has a handful of fast cores built for branching, sequential work. A GPU has hundreds to thousands of simpler lanes built for exactly this kind of wide, uniform arithmetic. The same inference that keeps several CPU cores busy is a small job for a GPU, and on a laptop the GPU is idle most of the day anyway.

Attention has one more property worth knowing: its cost grows with the square of the sequence length, because every token is compared with every other token. ModernBERTHugging Face's introduction to ModernBERT, the 2024 encoder architecture granite is built on: alternating global and local attention, RoPE, long context., which granite is built on, softens this by making only every third layer global. The other layers let each token see just 64 tokens on either side. That is why chunk size matters for cost, and I measured how much in Chunk size vs the 128-token attention windowBenchmark of chunk sizes 128 to 1022 on the CPU: recall, embed time, peak RAM and index size, and why the 128-token attention window is not a cliff.: recall held from 128 to 510 tokens, while doubling to 1022 cost 37-46% more CPU and 56-82% more RAM.

ONNX: one file format for many runtimes

Models are usually trained in PyTorch. Shipping PyTorch inside a .NET tool would be heavy and awkward, so the model is exported to ONNX (Open Neural Network Exchange, pronounced "onyx"). An .onnx file is a computation graph of standard operators, such as MatMul, Softmax and LayerNorm, plus the trained weights. Any runtime that understands those operators can run the model, whatever framework trained it.

ONNX Runtime and execution providers

ONNX Runtime (ORT) is Microsoft's engine for running ONNX graphs. It has an official .NET package, which is how ai-raccoon uses it. Creating a session parses the graph, applies generic optimisations, partitions the graph across providers, and then applies provider-specific fusions, such as turning the attention pattern into a single kernel. Running it is a separate call.

Partitioning is where hardware comes in. ORT does not talk to GPUs itself. Hardware backends plug in as execution providersONNX Runtime's documentation of execution providers: the list of hardware backends and how a session assigns graph nodes to them. (EPs): CUDA and TensorRT for NVIDIA, DirectML for Windows GPUs, CoreML for Apple, OpenVINO for Intel, WebGPU as a cross-platform GPU option, and the CPU provider that is always there. When a session starts, ORT asks each provider, in priority order, which parts of the graph it can run. Each takes the pieces it supports, and whatever is left falls back to the CPU provider.

That fallback is convenient and also a trap. A graph that is 95% on the GPU but bounces back to the CPU for one operator in every layer pays for copying data back and forth each time, and can end up slower than running on the CPU alone. How much of the graph each provider actually accepts is the number to check.

WebGPU and MLX

Two providers matter for ai-raccoon on a Mac.

WebGPU is a modern GPU API designed for browsers, with native implementations that sit on top of Metal on macOS, Direct3D 12 on Windows and Vulkan on Linux. ORT's WebGPU providerONNX Runtime's WebGPU provider documentation: runs models through Dawn on Metal, Direct3D 12 or Vulkan. uses it (through Google's Dawn implementation) to run models on the GPU of whatever machine it is on.

MLX is Apple's array framework for Apple silicon. On these chips the CPU and GPU share the same physical memory (unified memoryOn Apple silicon the CPU and GPU share one pool of physical RAM, so GPU buffers count against the same memory as everything else.), so there is no copying across a PCIe bus, and GPU buffers show up in the process's footprint. ai-raccoon can use MLX, as an opt-in, through an ONNX Runtime plugin providerAn ONNX Runtime plugin execution provider that runs models on Apple silicon GPUs through MLX, loaded into a stock ORT at runtime.: a provider shipped as a separate library and loaded into a stock ORT at runtime, a mechanism ORT added in 1.23.

4. Moving embedding to the GPU

With that in place, here is what happened, measured on an Apple M4 with ONNX Runtime 1.30.0, one row per run.

The latency gain is modest. The CPU gain is the point: a 512-token embed went from a quarter of a CPU-second to a few milliseconds, which is CPU the build and the IDE get back.

Four decisions got it there.

fp16, not int8. int8 is smaller still, but an int8 graph on the GPU reproduced its own CPU vectors only at cosine similarityHow closely two vectors point the same way. 1.0 means identical direction. Search ranks results by it, so small drops can reorder results. 0.944-0.966. That is enough to reorder search results without anyone noticing. fp16 on WebGPU matches fp16 on the CPU at 0.9998.

Turn off spinning. After the move, WebGPU still cost 44-47 ms of CPU per embed (measured in a separate session from the table above). Almost all of it was ONNX Runtime's thread pool spinning: threads busy-waiting in a loop for the GPU to finish instead of sleeping. Spinning helps a CPU workload shave microseconds off wake-up time. With the GPU doing the work it only burns cores. Setting session.intra_op.allow_spinning to 0 (#712The pull request that turns off ONNX Runtime's intra-op thread spinning for GPU sessions.) dropped the cost to 3.5-4.1 ms with no change in latency.

Rewrite attention for MLX. The stock MLX plugin rejected all 12 attention nodes, so attention fell back to the CPU across 15 separate islands of the graph. That is the fallback trap from above, and it made MLX 3.4x slower than WebGPU. Rewriting attention into operators the plugin supports put all 547 nodes on the GPU in a single fused subgraph. End to end it roughly ties WebGPU (32.1-34.2 s against 34.1 s for a 1,376-chunk code ingest), because chunking and full-text indexing do not care which device embeds, so MLX stays opt-in.

Do not ship what aborts. Standard ONNX Runtime has no WebGPU provider off macOS. Release 1.51.0 added the WebGPU plugin for Windows and Linux; its first run on a real Linux GPU aborted the whole process with Invalid memory type: -1, the same abort already reported upstream against the CUDA plugin provider (onnxruntime#28329Upstream ONNX Runtime issue: the CUDA plugin provider aborts the process with Invalid memory type: -1 on its first run.). An abort cannot be caught, so 1.51.2 turned the plugin off again. As of 1.51.2, Windows and Linux run on the CPU provider, and CUDA is opt-in and unmeasured.

5. The MLX cache that grew to 17.9 GB

MLX brought its own memory problem, and only the footprint caught it.

MLX keeps a cache of freed GPU buffers so it can reuse them, keyed by buffer size. Every distinct chunk length needs differently sized buffers, and ingest produces hundreds of distinct lengths, so almost nothing was reused and the cache kept growing. At the end of a run MLX held 92 MiB of live memory and about 17 GB of cached buffers. RSS never showed it. On a 24 GB laptop, the footprint did.

The fix had two parts. Every row is padded to the next multiple of 64 tokens, with the padding masked out so the vector does not change (cosine 0.999998 or better). That collapses hundreds of lengths into at most 16 (one per 64 tokens up to 1024). Then mlx_set_cache_limit caps the cache at 512 MiB.

After 1.51.1 shipped, the release checklist measured a fresh MLX server at 866 MB (peak 909 MB) after embedding 200 notes. The detailed MLX measurements, including how footprint tracks the number of distinct lengths rather than chunk size, are in Chunk size on MLXThe same chunk-size benchmark on the MLX GPU path: recall parity with the CPU, and how memory follows the number of distinct lengths..

Did search get worse?

No. Every GPU path scores within bootstrapResampling the same queries thousands of times to see how much a score moves by chance. If the range of the difference crosses zero, the two are tied. noise of the CPU reference on nDCGA search-quality score between 0 and 1. Higher means the right answers show up nearer the top of the results., measured through the product's own hybrid search:

The CPU changes are pinned by tests that can fail: EXPLAIN QUERY PLAN on every cascade statement must show index seeks and no SCAN, path ranges must never match a sibling like docs-old or docs.v2, and the excluded-path skip was red before the change and green after. The 1.51.1 release checklist ran against the nuget.org package: 25 pass, 3 substituted, 0 fail.

Caveats

CPU-provider latency depends on how busy the machine is. The ~12 ms / ~50 ms figures come from the ADR-0108 session on a quiet M4. Under a load average of 25-55 the same provider took 31-244 ms and 135-1012 ms. The GPU numbers were the same in both sessions, which is the real argument for the GPU: its cost barely moves when everything else is busy.

The side-by-side CPU windows had no clean idle baseline, since real file events kept arriving. The controlled A/B holds load equal and gives the 5.84 to 0.64 and 14.17 to 1.13 figures.

The two MLX footprints are different workloads: 1.1-1.2 GB is 400 rows of 300-1022 tokens, 866 MB is 200 short notes.

Windows and Linux GPU latency is unmeasured. I had no machine to measure on, and CI has no real GPU.

If you want to learn this properly

While learning the material behind this post, I worked through three illustrated explainers, and they are the best place to go deeper than a blog post can. Each has diagrams, a glossary, the raw numbers and a short quiz at the end.

Granite on ONNX RuntimeIllustrated walkthrough: how ONNX Runtime loads and runs granite, the 12-layer model inside it, its attention pattern, and what happens on each embed. is the one to start with. It covers what ONNX Runtime does between reading a file and returning a vector, and how execution providers split the graph. It then opens the model up: tokenizer, embedding table, the 12 encoder layers with their attention and feed-forward blocks, and where the 47M parameters actually live.

Chunk size vs the 128-token attention windowBenchmark of chunk sizes 128 to 1022 on the CPU: recall, embed time, peak RAM and index size, and why the 128-token attention window is not a cliff. answers a question that confused me: if most layers only see 64 tokens either side, why does a 510-token chunk still search well? It measures recall, CPU and RAM from 128 to 1022 tokens on the CPU and shows where the cost starts to bend.

Chunk size on MLXThe same chunk-size benchmark on the MLX GPU path: recall parity with the CPU, and how memory follows the number of distinct lengths. runs the same benchmark on the GPU. Recall matches the CPU run arm for arm, time per token stays flat, and memory turns out to follow the number of distinct input lengths. That is where the 17.9 GB cache from section 5 was found.

What I take from it

All three problems had the same shape: work nobody asked for, done over and over. Deletes for files that were never stored. A batch buffer nobody needed, kept forever. Threads spinning while the GPU worked. A cache for shapes that would never come back. None of it looked like a bug, and most of it never showed up in RSS. It showed up in the physical footprint and in CPU-seconds, which is where I measure first now.

ai-raccoon is open source (MIT): github.com/Arasz/ai-raccoonai-raccoon on GitHub (MIT): a local-first memory server for AI agents, with hybrid keyword and vector search over notes and code.. Install it with dotnet tool install -g ai-raccoon.