A WebGPU RAG assistant, from CMS to browser.

WebGPU gives browsers access to GPU computation. Here it handles inference: applying model weights to produce an embedding or generate text. Local execution avoids calling a hosted generation API for each answer. It still requires compatible browser support, drivers and GPU resources, on an HTTPS page.

The presence of navigator.gpu is insufficient. The application checks the hardware adapter and rejects a fallback software adapter. Physical tests included the Samsung Galaxy S25 Ultra’s Adreno 830 in Samsung Internet and an NVIDIA Ampere GPU on a computer. Features and buffer limits vary: browser branding and total phone RAM do not prove that an export will run.

Gemma 4 E2B uses LiteRT-LM, with @litert-lm/core pinned to 0.17.1, the GPU_ARTISAN backend explicitly requested, a 4,096-token context window and thinking disabled. The LiteRT-LM Web API remains an early preview, qualified here on the tested devices. E5 embeddings use a separate path: ONNX Runtime Web with WebGPU. Several ONNX exports, WebLLM/MLC and compact runtimes were evaluated before this choice.

Experimental compact runtimes also required work on WGSL shaders, quantized weights and cached model states. The Nanbeige 4.2 path must respect its architecture’s two passes: dropping a pass to save time would not be an equivalent optimization. A lightweight format or smaller model is insufficient; its operations must run correctly on that GPU and the RAG pipeline must remain reliable.

Integration required two separate Web Workers. The LiteRT SDK uses importScripts and starts in a classic worker; the embedding worker is a JavaScript module. This keeps the response model loaded during E5 retrieval and the interface responsive. Cancellation covers both workers; GPU loss releases the runtimes and reports an actionable error.

The CMS selects published pages, approved knowledge entries and eligible form information. Prepared text is split using the actual embedding tokenizer: 400-token passages with a 40-token overlap, a bounded title and a check of the 512-token limit after adding the prefix. Oversized passages are rejected instead of silently truncating information.

The selected Multilingual E5 Base profile produces 768-dimensional vectors. Inputs distinguish query: for questions and passage: for documents. Outputs use masked mean pooling and L2 normalization. Indexing and retrieval must use identical operations: a correctly sized vector can still be numerically wrong.

On mobile, the MatMulInteger path in some quantized exports was incompatible with the runtime used. A reproducible preparation tool retains INT8 weights while adapting matrix operations to a portable FP32 path. E5 artefacts undergo SHA-256 verification and comparison with four numerical reference fixtures, requiring a cosine of at least 0.999. References are calculated outside the browser; visitor inference remains on WebGPU.

MariaDB stores vectors in a VECTOR(1024) column. Remaining dimensions are padded with zeroes, preserving cosine calculations. Each index is linked to its embedding profile and each passage to its content hash. Outdated batches and unnormalized vectors are rejected.

For each question, the server ranks passages by cosine distance, filters language and still-published sources, applies the configured threshold and limits passages from any one document. The illustrated configuration keeps four passages with a 0.75 threshold. E5 Small, 384 dimensions and about 118 MB versus Base’s 279 MB, is available for further testing; it needs its own index and cannot be derived by shortening Base vectors.

A follow-up such as “what about maintenance?” is hard to search without its earlier topic. A first model pass creates a self-contained query from the request, recent exchanges and any reference. Output cleaning prevents model commentary from becoming a search query. If rewriting fails without a fatal runtime error, search can fall back to the original question.

The search retains the question and its rewrite when they provide distinct queries. The server combines their rankings through reciprocal rank fusion: passage ranks are combined rather than letting a rewrite entirely replace the visitor’s request. This reduces the risk of losing an important detail during rewriting.

Persistent memory is optional and stored in the visitor’s browser through IndexedDB: up to 60 exchanges for 7 days. Entries contain a question, answer, vector, language, embedding profile and timestamp. Expired entries are removed when accessed; users can disable or clear memory. If storage is unavailable, discussion continues without persistence.

Memory includes local semantic retrieval. The current question vector is compared with earlier questions in the same language and embedding profile; the best result above the profile threshold can enrich context. Recent exchanges take priority. An old generated answer is not reintroduced as a factual source: only the relevant earlier question supplies distant background context.

The writing prompt combines the CMS role and instructions, latest request, useful conversation history and retrieved passages. Sources are presented as data separate from system instructions. Drafts remain private: the interface displays progress, followed by an accepted answer. Waiting therefore includes multiple inference passes, not just visible writing.

A second pass of the same model assesses whether the draft addresses the question and whether essential facts are supported. LiteRT structured exchanges use a local emit_result tool with no external action. The application validates JSON objects and required field types. Valid JSON alone does not establish that the assessment is correct.

One real bug involved unequal source context: the writer saw up to 1,250 characters per passage, while the reviewer saw only 650. A price range farther down could be missing from review. Both passes now receive the same sources with a 1,250-character limit, avoiding a request to verify evidence withheld from the reviewer.

A rejected draft can trigger corrective retrieval and a second version. The process is bounded: at most two drafts and one additional search. Repeated old answers and internal commentary are filtered too. Without an acceptable answer, the assistant states the limitation and offers contact.

Native generation counters are retained; empty outputs and outputs reaching their token cap are rejected. No completion signal absent from the SDK is fabricated. Review reduces some defects but uses the same model as writing and can be wrong itself. False drafts and missing-information cases are essential tests; visitors can still inspect sources.

Response inference and persistent memory run on the device. The service remains connected: it downloads models and libraries, sends search vectors to the server and receives source texts. An embedding is not an anonymity guarantee. Uncovered questions can be recorded in the CMS to improve knowledge, for at most 90 days without a new request and up to 5,000 questions per site.

APIs enforce site scope, administrator sessions and operation-appropriate CSRF protections. Incoming vectors are checked for profile, dimensions, finite values and normalization. Search excludes withdrawn pages and unauthorized forms; draft knowledge entries are not exposed in answers.

When web references are enabled, the server can read a public URL supplied by the visitor. Protocols, network addresses and redirects are checked to block private-network access. Downloads have size and time limits. Useful text is extracted without running page scripts and remains external reference data, not a new system instruction.

The assistant can also help prefill an authorized form from information explicitly supplied by the visitor. Field schemas, language, target form and validity duration are checked. Existing values are preserved, consent is never automatically checked, and no form is submitted without user action.

The CMS does more than select a model name. It prepares sources, resumes indexing, tracks pending passages and offers real tests. Mobile/tablet and desktop response profiles are independent. Client detection uses browser and device information; narrowing a desktop window does not switch it to a mobile model.

A test is linked to the model, configuration and retrieval performed. Changing a model, instructions or index invalidates the relevant qualification: another test and reactivation are required. Configuration signatures prevent activation based only on an obsolete test. Purely visual settings preserve answer qualification.

Verification covered real GPU generation, numerical embedding consistency, price and maintenance questions, false promises and unpublished figures. Mobile fixes were also checked with the keyboard open, panel opening and closing, an accessible send button and no overflow. Releases back up the affected scope and verify public pages.

Remaining work has specific targets: broaden qualification questions, measure cold downloads and quotas, check more browser/GPU pairs and compare new Adreno-compatible exports. A native app could access other accelerators; hosted inference could reduce phone workload. These alternatives would change the architecture and are not presented as already delivered.