KRR Chat: A Language Model Built from Eigenvalues

In the post Eigenvalues & AI, we built a chain: from linear regression through kernel methods to neural networks – all connected through eigenvalues. At the end stood a live demo: the KRR Chat, a language model that works without a neural network.

This post is the technical deep-dive. We examine every component of the KRR Chat: the architecture, the pipeline, the complete source code – and the honest limitations. If you have read the main post, you will find the theory turned into running code here. If not – no problem: this post also works on its own.

Chapter 1

The Dual Model: Retrieval + Generation

KRR Chat consists of two independent systems that work together – similar to a person who first looks something up in a book and then answers in their own words.

Why two systems?

A single KRR model on all 805 sentences fails due to the capacity limit: 1820 distinct words on 256 hash buckets means an average of 7 collisions per bucket. The model cannot distinguish “eigenvalue” from “convergence” when both hash to the same bucket.

The solution: division of labor. The retrieval system provides topically relevant context (broad knowledge, 805 sentences). The generation model produces fluent continuations (deep knowledge, 104 sentences, but with only 4 collisions per bucket). In AI research, this pattern is called RAG – Retrieval-Augmented Generation. Large language models like GPT-4 and Claude also use RAG to incorporate up-to-date knowledge.

Chapter 2

The Pipeline: From Question to Answer

When you ask “What are eigenvalues?”, the following happens:

1. Keyword extraction: Stop words (what, are, the, is, ...) are removed. What remains: ["eigenvalues"]
2. Retrieval: All 805 sentences are searched for this keyword. Exact match = 5 points, partial match (e.g. “eigenvalue” in “eigenvalues”) = 2 points. Duplicates are removed. The top 2 sentences are displayed in cyan.
3. Invisible seeding: The last 5 words of the last retrieval sentence are taken as the context for the generation model. This is the crucial transition: retrieval provides the starting point, generation carries on.
4. KRR generation, word by word: From the 5-word context, the model computes:
   • Feature vector \(\mathbf{x}\) via word hashing (128 buckets × 5 positions + 3 bigram hashes = 1024 dimensions)
   • Random Fourier Features: \(\mathbf{z} = \sqrt{2/D}\,\cos(\mathbf{x}\boldsymbol{\omega} + \mathbf{b})\) with \(D = 1536\)
   • Next word: \(\hat{y} = \arg\max_w\, \mathbf{z}^\top \mathbf{W}_{:,w}\)
   This word is appended, the context shifts by one position, and the process repeats.

Conversation memory

KRR Chat has a simple memory: for follow-up questions (e.g. “Tell me more” without a new keyword), no new retrieval is triggered. Instead, the model continues generating from the last word of the previous answer. This creates a natural flow – the answer picks up where it left off.

Chapter 3

The Three Colors: Memorization vs. Generalization

Every word in the answer is color-coded – and this is where things get interesting. The colors honestly show where each word comes from:

What the colors reveal

Typically, 60–80% of generated words are green (verbatim) and 20–40% are orange (generalized). What does that mean?

A KRR model with a 505-word vocabulary primarily memorizes. It has largely learned the 104 training sentences by heart and can reproduce them. But at the seams – where a retrieval sentence ends and generation takes over, or where the model navigates between two learned sentences – orange words appear. These are moments when the kernel space interpolates: the current context lies between two training points, and the model chooses a word that was never trained in this exact combination.

This honesty is intentional. Large language models like GPT-4 generalize elegantly – but you cannot see when they memorize and when they interpolate. KRR Chat makes this distinction visible.

Chapter 4

The Source Code: Three Functions

The entire KRR Chat consists of three functions. Here is the complete inference code – readable, commented, no tricks. The full source code comprises ~120 lines of JavaScript.

1. encode() – Turning words into numbers

The feature map \(\phi(x)\): each word is mapped to an index via a hash function. The position in the context determines the weight – later words count more because they are closer to the word being predicted.

// Word → hash index (0..127). Each word gets a "fingerprint".
function hash(word) {
  var h = 0;
  for (var i = 0; i < word.length; i++)
    h = (h * 31 + word.charCodeAt(i)) >>> 0;
  return h % 128;  // 128 buckets
}

// Context (5 words) → feature vector (1024 dimensions)
// Each word occupies a position × hash bucket.
// Position weighting: later words count more.
function encode(context) {
  var features = new Float32Array(1024);  // 5×128 + 3×128
  for (var pos = 0; pos < 5; pos++) {
    var weight = 0.4 + 0.6 * (pos / 4);  // 0.4 → 1.0
    features[pos * 128 + hash(context[pos])] += weight;
  }
  return features;
}

Mathematics: This is \(\phi(x)\) – a feature map that embeds words into a high-dimensional space. Instead of representing each word as a one-hot vector (which would require 505 dimensions per position with a 505-word vocabulary), we compress via hashing to 128 dimensions. Hash collisions are unavoidable (505 words onto 128 buckets ≈ 4 collisions/bucket), but the model still learns – as long as the collision rate is low enough.

2. rff() – Approximating the kernel

The Random Fourier Features following Rahimi & Recht (2007): a cosine transformation that makes the dot product in the transformed space equal the Gaussian kernel.

// φ(x) → z(x) via Random Fourier Features
// z(x) = √(2/D) · cos(x·ω + b)
// Then: z(x)ᵀz(x') ≈ k(x,x') = exp(-‖x-x'‖²/2σ²)
function rff(features) {
  var z = new Float32Array(1536);   // D = 1536
  var scale = Math.sqrt(2.0 / 1536);
  for (var j = 0; j < 1536; j++) {
    var dot = 0;
    for (var k = 0; k < 1024; k++)
      dot += features[k] * omega[k * 1536 + j]; // ω: random, fixed
    z[j] = Math.cos(dot + bias[j]) * scale;
  }
  return z;  // kernel-space vector
}

Mathematics: The matrix \(\boldsymbol{\omega}\) contains random values drawn from \(\mathcal{N}(0, 1/\sigma^2)\). It is drawn once (with a fixed random seed for reproducibility) and never modified – it is not a learned parameter. The cosine ensures that \(\mathbf{z}(x)^\top\mathbf{z}(x')\) approximates the Gaussian kernel \(k(x,x') = \exp(-\|x-x'\|^2/2\sigma^2)\). The larger \(D\), the better the approximation.

Why \(\sigma = 1.5\) and \(\lambda = 10^{-6}\)? The kernel parameter \(\sigma\) controls the “sharpness” of similarity: small \(\sigma\) means only very close contexts count as similar (memorization), large \(\sigma\) means distant contexts also influence each other (generalization). \(\sigma = 1.5\) is an empirically chosen compromise – found by testing on the training corpus to maximize Top-1 accuracy. The regularization parameter \(\lambda = 10^{-6}\) is deliberately small: our training set is dense enough that strong regularization would hurt prediction quality. In larger models, systematic cross-validation would be standard – with 2000+ hand-curated pairs, empirical tuning suffices.

3. predict() – Determining the next word

The prediction: a single matrix-vector multiplication. No softmax, no attention, no feedforward network.

// Context → next word
// scores = z(x)ᵀ · W   (W was learned offline via KRR)
function predict(contextWords) {
  var z = rff(encode(contextWords));
  var scores = new Float32Array(505);  // one score per word

  // Matrix-vector multiplication: scores = zᵀW
  for (var j = 0; j < 1536; j++)
    for (var v = 0; v < 505; v++)
      scores[v] += z[j] * W[j * 505 + v];

  return argmax(scores);  // word with highest score
}

Mathematics: The weight matrix \(\mathbf{W}\) was computed offline via the ridge regression system: \(\mathbf{W} = (\mathbf{Z}^\top\mathbf{Z} + \lambda\mathbf{I})^{-1}\mathbf{Z}^\top\mathbf{Y}\). Here, \(\mathbf{Z}\) is the matrix of all RFF vectors from the training data, \(\mathbf{Y}\) is the one-hot matrix of target words, and \(\lambda\) is the regularization parameter. In the browser, only the multiplication \(\mathbf{z}^\top\mathbf{W}\) takes place – a single matrix operation per word.

How they work together

The three functions form a pipeline:

The entire chatbot is this pipeline in a loop – plus a keyword search that determines the starting point. Each iteration runs in under a millisecond; with WebGPU acceleration, even in microseconds.

Chapter 5

Why Float64? The Precision Lesson

A natural first approach: train a single KRR model on the entire 805-sentence corpus – directly in the browser. We tried this. And it fails instructively.

The browser training disaster

Three problems pile up:

The solution: train offline, predict online

Training happens offline with Float64 (NumPy) – a full 15 decimal digits of precision. The weight matrix \(\mathbf{W}\) is then compressed as Float16 and loaded in the browser.

Inference runs in the browser with Float32 – for a single matrix-vector multiplication, that is perfectly sufficient. The numerical instability arises only when solving the linear system, not when applying the solution.

Chapter 6

What the Model Can Do – and What It Cannot

Honesty is one of the most important virtues in AI research. Here is a sober assessment:

What KRR Chat can do

• Reproduce learned sentences: Questions about eigenvalues, kernel methods, PageRank, quantum mechanics – all topics covered in the 104 training sentences are answered fluently and correctly.
• Navigate between topics: Through the retrieval system, the chat can find topically relevant starting points and then continue via generation.
• Show transparency: The color coding makes visible what a language model does – a pedagogical quality that no large LLM offers.

What KRR Chat cannot do

• Generate truly new content: The model interpolates in kernel space between learned points. It cannot invent explanations that are not (at least in fragments) contained in the training data.
• Answer out-of-distribution questions: Ask about “Pythagoras” or “Shakespeare” – retrieval finds nothing, and generation has no meaningful starting point.
• Say “I don’t know”: Instead of admitting ignorance, the model keeps generating – even when the context no longer makes sense.

The stop-word problem

At transitions between topics – when the model has finished a learned sentence and cannot find a suitable follow-up – the output sometimes collapses into stop-word gibberish: “the the of and the is a...”. This happens because stop words are the most frequent words in the vocabulary and receive the highest score in uncertain contexts.

The fix: As soon as the model generates three consecutive stop words, generation is halted and a new retrieval is triggered (re-seeding). This interrupts the gibberish and finds a new topical anchor point.

Chapter 7

The Connection Back to Theory

Every component of KRR Chat corresponds to a concept from the Eigenvalues & AI post. Here is the complete mapping:

KRR Chat is therefore not just a demo – it is a living textbook in which every line of code corresponds to a mathematical concept. The feature map is Chapter 6. The regularization is Chapter 5. The numerical instability is Chapter 3. Everything is connected.

And that is the real punchline: whether a model has a 505-word vocabulary or 50,000 – the mathematical structure is the same. Eigenvalues, kernels, regularization. The chain holds.

From Prototype to Chatbot: KRR Chat Becomes Kalle

The KRR Chat above is a prototype: 104 English sentences, pure prose generation, no dialog structure. It proves that KRR can model language. But a prototype is not yet a chatbot.

So we asked: What happens when you apply the same mathematical principle to a real dialog? The result is Kalle – a German-language chatbot with the same three equations (RFF, KRR, BoW+IDF), but a completely different corpus: 2113 curated dialog pairs instead of 104 technical sentences. The differences reveal which parameters actually matter for language models:

Key insight: Kalle has undergone a radical architectural transformation – and the lessons are instructive for anyone thinking about AI systems.

Frequently Asked Questions

What is the difference between KRR Chat and ChatGPT?

ChatGPT uses a neural network with billions of parameters and attention mechanisms. KRR Chat uses Kernel Ridge Regression with a single weight matrix (1536 × 505 = ~780,000 parameters). Both solve the same task – “given context, predict the next word” – but with entirely different mathematical tools. KRR Chat makes visible through color coding what the model memorizes and what it generalizes.

Why does KRR Chat respond in English?

The 104 training sentences and the retrieval corpus of 805 sentences are written in English because the mathematical terminology is predominantly English. The model has no understanding of “language” – it has only learned which English word is likely to follow which context. A German version would simply require German training sentences.

Can KRR Chat be trained with more data?

Yes – Kalle proves it: from 57 to 2113 dialogue pairs. But the journey showed: quality beats quantity. With 4301 programmatically generated pairs, accuracy dropped to 35%. Only radical curation (fewer pairs, each with purpose) brought quality back. The switch from hash encoding to Word2Vec embeddings solved the collision problem. The key is not “more data” but “better data” – a lesson that applies to large language models too.