Hi SaugaTech Community,
The World Cup is over. Spain edged out Argentina 1-0 in extra time to lift the trophy, Messi’s last World Cup dream ended agonisingly short, the fan zones have packed up, Celebration Square has gone back to being just a square, and the GTA has exhaled after one of the more electric summers this city has had in a while. If you were part of any of it — the watch parties, the flags, the noise — we hope you’re still riding a bit of that high.
But the last couple of weeks have also been a reminder of something a lot less fun: nature does not care about our schedules.
Over 850 wildfires have been burning across Canada this month, more than 180 of them in Ontario alone. On July 15th, smoke drifting down from the northwest made Toronto the second most polluted major city on the entire planet for a stretch — worse air quality than cities that top that list on an average day. Skies over the GTA turned that unmistakable orange-grey. If you stepped outside and felt your throat catch, or checked an air quality app and did a double take, you weren’t imagining it.
It’s a strange kind of whiplash — the joy of a World Cup summer and the quiet menace of smoke a thousand kilometres away finding its way into our lungs. Two very different reminders, in the same few weeks, of how connected we all are to things well outside our control.
Alright. On to what we’re actually here for this week.
This week, we’re kicking off a new thread in the Compass, AI Under the Hood — one that goes underneath the AI tools most of us already use every day, into how they actually work.
Why We’re Starting This Series
Here’s something we’ve noticed across almost every SaugaTech meetup, every WhatsApp thread, every builder conversation over pizza: most of us are genuinely fluent in what LLMs can do. We know how to prompt them, chain them, wrap them in an app, ship a demo in a weekend. That fluency is real, and it’s valuable — it’s exactly what’s let this community build the things it’s built.
But if you ask the room how a large language model actually works — not the API call, not the prompt, but the mechanism underneath — the room tends to go quiet.
And that’s not a criticism. Most of us came into this AI wave through the product layer, not the research layer. There was never really a reason to go deeper. The tools worked. You didn’t need to know how an engine combusts fuel to drive a car.
Except increasingly, that’s not quite true anymore.
The gap between “I can use an LLM” and “I understand why an LLM behaves the way it does” is becoming the gap between a good AI feature and a great one.
It’s the difference between a chatbot that hallucinates unpredictably and one you’ve engineered to fail gracefully. Between a RAG pipeline that returns garbage half the time and one that’s actually reliable. Between prompting by vibes and prompting with a real model of what’s happening inside the black box.
As more of us build serious products on top of this technology that underlying understanding stops being optional.
So we’re starting a new thread in the Compass: a slow walk through the Foundational research papers of modern AI, explained in plain English, with an eye specifically toward what each one means for something you might actually build. No paper is off limits because it looks intimidating. That’s the point of doing this together.
🚀 First Things First
We are already planning for the August meetup, you’ll hear the details in next week’s Compass. And if you’re not in the WhatsApp community yet, that’s where the follow-up conversations to editions like this one tend to happen: Click here to request to join the group
⚓ The Paper: “Attention Is All You Need” (2017)
Every large language model you’ve ever used — GPT, Claude, Gemini, Llama — is a descendant of one 2017 research paper out of Google Brain, co-authored by Aidan Gomez, a 21-year-old University of Toronto student at the time. We told his story in our Canada Day edition. This week, we’re telling the story of the paper itself. If you want to read the detailed paper, here is the link to it
The Problem It Solved
Before this paper, AI language models — RNNs, LSTMs — read sentences one word at a time, left to right, like a finger tracing under each word. The model carried a “memory” forward as it read, but that memory was limited. By the time it reached the end of a long sentence, it had often half-forgotten the beginning.
Take the sentence: “The trophy didn’t fit in the suitcase because it was too big.”
What does “it” refer to — the trophy or the suitcase? A human resolves this instantly. An RNN, reading word by word, often struggled — by the time it reached “it,” the earlier context had faded.
There was a second, more practical problem: speed. Reading word by word means you can’t parallelise. You can’t throw a thousand GPUs at a sentence and process it faster, because word 5 depends on having already processed word 4. You’re stuck in a queue.
The Core Idea: Attention
The paper’s proposal: stop reading sequentially. Look at every word in the sentence simultaneously, and let the model figure out which words matter most to each other. They called this mechanism Attention — and the paper’s confident, almost cheeky title was a direct claim: you don’t need the old sequential machinery at all. Attention alone is sufficient.
Think of it like being at a loud party. You don’t process every sound in the room equally — your brain attends to the conversation that matters and tunes out the rest. When the model processes the word “it” in our trophy sentence, Attention lets it look at every other word at once and ask: which of you am I most connected to? “Trophy” scores high. “Suitcase” scores a bit lower. “Because” barely registers. The model uses those scores to build a representation of “it” that’s genuinely informed by the right context.
How It Works: The Library Analogy
The mechanics use three concepts — Query, Key, Value — that sound abstract until you picture a library.
Query = what you’re searching for
Key = the label on each book on the shelf
Value = the actual content inside the book
You compare your Query against every Key in the library, score how well each matches, then pull the Value from the best matches and combine them into your answer. Every word in a Transformer generates its own Query, Key, and Value. Processing a word means scanning every other word’s Key with your Query, scoring the matches, and blending in their Values — weighted by how relevant they are.
The paper also introduced Multi-Head Attention — doing this eight times in parallel, each “head” learning to notice something different. In the paper’s own visualisations, you can see individual heads specialising: one resolves what a pronoun refers to, another tracks grammatical structure, another tracks semantic similarity. Eight specialists reading the same sentence, each noticing something different, pooling their observations into one richer understanding.
One elegant fix solves a side effect of processing everything at once: if you’re not reading in order, how does the model know “dog bites man” differs from “man bites dog”? The paper adds a mathematical position signal — sine and cosine waves — to each word before processing, so the model always knows where a word sits in the sentence, even while processing it out of order. Like numbering theatre seats before the lights go down.
Encoder and Decoder: The Two Rooms
The full Transformer has two halves. The paper was originally built for translation, so this framing helps:
The Encoder reads the input sentence. It runs the full attention mechanism — every word looking at every other word — and produces a rich, context-aware representation of the whole sentence. Think of it as a very deep understanding of what was said.
The Decoder generates the output — the translation — one word at a time. But crucially, when generating each output word, it can attend to the full encoder output. So when it’s generating the French translation of word 5, it can look back at the full English sentence and ask: which English words are most relevant right now?
The decoder also has a constraint: when generating word 5, it can only look at words 1-4 that it has already generated. It can’t peek ahead. This is called masking — and it’s what makes the model useful in the real world where you’re generating text you don’t know yet.
The Result That Made Everyone Sit Up
The paper tested on English-to-German and English-to-French translation benchmarks. The results:
Outperformed every previous model including ensembles of multiple models
Achieved state-of-the-art translation quality
Did it in 3.5 days of training on 8 GPUs — a fraction of the compute cost of previous best models
That last point is as important as the quality improvement. Not only was it better — it was dramatically cheaper and faster to train. That combination is what made the research world stop and take notice immediately.
Why It Changed Everything
The Transformer architecture the paper is the direct ancestor of:
GPT-1, 2, 3, 4 — OpenAI took the Decoder half and scaled it massively. That’s ChatGPT.
BERT — Google took the Encoder half and used it for understanding and classification tasks. It powers Google Search to this day.
Claude, Gemini, Llama, Mistral — every major LLM running today is a Transformer.
AlphaFold — DeepMind used Transformers to solve protein folding, one of biology’s hardest problems.
⚓ Why This Actually Matters — Real-World Mechanics You’re Already Using
Here’s where the theory turns into something you can use in a product decision next week.
Every “chat with your documents” feature is a Query/Key/Value problem in disguise. When you build a RAG system — retrieval augmented generation, where a model searches your documents before answering — you are literally implementing a version of the library analogy above. The user’s question is the Query. Your document chunks are the Keys. The actual chunk content is the Value. If your RAG pipeline returns bad results, nine times out of ten it’s because your Keys are badly constructed — poor chunking, weak embeddings, unclear document structure — not because the underlying model is “not smart enough.” Understanding the mechanism tells you exactly where to go fix it.
Context window limits exist because Attention is computationally expensive. In the paper’s own complexity table, self-attention scales as the square of sequence length. Double the input length, quadruple the compute. That’s not an arbitrary product limitation from Anthropic or OpenAI — it’s a direct mathematical consequence of the mechanism we just walked through. When you’re deciding whether to dump your entire codebase into a prompt or be selective about what context you feed a model, you’re making a decision with real cost implications rooted in this exact table.
Multi-head attention is why the same model can write code and write poetry. Different heads specialise in different relationships. This is part of why a single Transformer-based model can flexibly handle wildly different tasks without being explicitly programmed for each one — the architecture itself is built to notice many kinds of patterns simultaneously, and which patterns matter shifts depending on what you’re asking it to do.
Hallucination is partly an Attention problem, not just a “the model doesn’t know” problem. When a model attends weakly or ambiguously to the right context — because your prompt is vague, your retrieved documents are poor, or the relevant information is buried too far back in a long context — it fills gaps with its next most confident guess. That guess is often wrong but fluent. If you’ve ever wondered why a model confidently states something false, part of the answer is sitting in the attention scores you never see.
Encoder-only and decoder-only aren’t just jargon — they’re two different extremely powerful products. That’s clearly demonstrated from the fact that Google and OpenAI used just one of them to build massive platforms. If you’re building a classification tool — sentiment analysis, intent detection, content moderation — an encoder-style model is often the right, cheaper, faster choice. If you’re building something generative, you need the decoder machinery. Knowing this distinction can save you from reaching for a giant generative model when a small classifier would do the job better and cheaper.
The founders who build the most efficient AI products aren’t necessarily the ones with the cleverest prompts. Increasingly, they’re the ones who understand enough of what’s actually happening under the hood to make the right architectural call before they write a single line of code.
✨ SaugaTech Epilogue — Understanding the Engine
As builders, each of us don’t need to be able to implement a Transformer from scratch to create great AI products. Most of the best builders in this community never will, and that’s fine.
But there’s a difference between driving a car and understanding, even loosely, why it moves. The founders who’ll build the next generation of great AI products in this region are the ones willing to spend a little time under the hood — not to become researchers, but to make better decisions the next time something breaks, scales badly, or hallucinates in front of a customer.
That’s what this series is for. We’ll keep going — next up, we’ll look at how models went from predicting the next word to actually following instructions, and what changed to make that possible.
Let’s keep building, Let’s keep learning, Together.
Team SaugaTech
CONNECT | COLLABORATE | INNOVATE