Hi SaugaTech community,
It’s Friday. The long weekend is almost here — Ontario’s civic holiday Monday means three full days away from Slack, standups, and whatever your manager last pinged you about. Most of the western GTA is already mentally checked out, and honestly, fair enough.
Before you close the laptop though — this one’s worth five minutes.
Last week we kicked off a new thread in the Compass — a slow walk through the foundational research papers of modern AI, explained in plain English, with an eye toward what each one means for something you might actually build. We started with the 2017 paper that introduced the Transformer architecture. If you missed it, Part 1 is here. Worth a read before this one, though you don’t strictly need it.
This week, Part 2.
If you’ve opened ChatGPT or Claude this week and typed something like “write this in the style of my last three emails” or “here are two examples, now do a third” — congratulations, you were running a five-year-old research technique without knowing its name.
That’s actually where this edition starts. Last time in the Compass, we went back to 2017 and unpacked Attention Is All You Need — the paper that let machines read a whole sentence at once instead of word by word. Cool mechanism. But an engine on its own doesn’t tell you what car you’re driving. This week’s paper is the one that took that engine and built GPT out of it — and in the process, changed what “using AI” actually means for the rest of us.
Grab a coffee. Let’s get into it.
🚀 First Things First
Save the date — our next meetup is August 22nd at IDEA Mississauga, Square One and this one’s been requested a lot: Tech Careers in 2026: The New Rules of Getting Hired.
We’re bringing in a tech recruiter and a couple of hiring managers to share what’s actually changed in tech hiring right now — a follow-up of sorts to last year’s “Navigating Tech Careers in the Age of AI,” this time straight from the people doing the hiring. If you’re job hunting, thinking about a switch, or just curious what’s landing on the other side of the table these days, this one’s for you.
Details and RSVP will drop in the WhatsApp group early next week https://chat.whatsapp.com/HgtfWvkWZpXE2Qvis2Yjrm
⚓ The Paper: “Language Models Are Few-Shot Learners” (2020)
In May 2020, a team at OpenAI published a paper introducing GPT-3. At the time it was the largest neural network ever built — 175 billion parameters. That number got all the headlines. But honestly, the number wasn’t really the interesting part.
The interesting part was this: you could teach this thing a brand new task just by showing it a few examples, and it would pick up the pattern without anyone touching a single one of its 175 billion parameters.
That’s the exact trick that makes ChatGPT, Claude, and pretty much every AI assistant usable by a normal person. Before this paper, that’s not how any of this worked.
The problem the researchers tried to solve
Before GPT-3, if you wanted a language model to do something specific, say sort customer support tickets by urgency, you had to fine-tune it. That meant collecting thousands, sometimes hundreds of thousands, of labeled examples for that exact task, then retraining the model on them. Want it to do something else too? New dataset. Fine-tune again. Every task started the whole expensive process over from scratch.
The OpenAI team basically pointed out something kind of obvious once you say it out loud: that’s not how people learn things. Nobody hands a new employee ten thousand labeled examples to teach them what an urgent email looks like. You show them two or three, maybe give a quick explanation, and they’re off. Why should a model need it any other way?
The core idea — in-context learning
The bet the paper made: what if a model, if it’s big enough and has seen enough text, can learn a brand new task just by reading a handful of examples right there in the prompt — no retraining, no gradient updates, nothing?
They called this in-context learning and tested it three ways:
Zero-shot — just an instruction, no examples. “Translate this to French.”
One-shot — exactly one example of the task, then you ask it to do a new one.
Few-shot — somewhere between ten and a hundred examples stuffed into the prompt before your actual question.
None of these touch the model’s weights. It reads your examples, figures out what you’re going for, and does it — all in one pass, all within the length of a single conversation.
If that sounds like exactly how you already prompt ChatGPT or Claude — throw in a couple examples of the format you want before asking the real thing — that’s not a coincidence. You’ve been doing in-context learning this whole time. This paper is the reason it works at all.
The evidence — and why bigger turned out to actually be different
The paper didn’t just claim this worked. It showed something specific: this ability barely existed in small models, and got dramatically stronger the bigger the model got.
They trained eight versions of the same architecture — from 125 million parameters all the way up to the full 175 billion — and ran all of them through the same tests, in zero-shot, one-shot, and few-shot settings. The gap wasn’t subtle. Take a simple task like unscrambling a word with its letters shuffled around. Small models barely improved even with a hundred examples in front of them. The full-size GPT-3 got dramatically better with more examples, and beat what any smaller model managed even with a hundred shots.
That gap between “no examples” and “a few examples” widened as the models got bigger. Small models basically ignored the extra examples. The 175B model used them well. Which told the researchers something important — this isn’t a trick you can just find in any model. It’s something that seems to emerge only once you’re big enough.
Some of the actual numbers, all from prompting alone, no retraining involved:
Trivia questions: 71.2% accuracy few-shot, which actually beat systems that had been specifically fine-tuned for that exact task
Arithmetic: near-perfect on 2-digit addition, and genuinely decent even on larger numbers it had never explicitly been trained to add
Unscrambling words: putting a shuffled word back together, something the model was never directly trained to do
Inventing new words: give it a made-up word and its definition once, and it could use that word correctly in a brand new sentence
Writing news articles: people could only correctly guess GPT-3 had written a short article 52% of the time. Basically a coin flip.
All of it from a big model reading a few examples and figuring out the pattern on its own.
Where it fell short
The paper is refreshingly honest about this. GPT-3 was bad at tasks that required comparing two pieces of text directly — telling whether one sentence implies another, or whether the same word is being used the same way in two different sentences. On those, it barely beat random guessing even with lots of examples. It also sometimes contradicted itself over longer passages, and struggled with basic common-sense physics questions despite doing fine on benchmarks that were supposedly testing the same thing. That kind of candor is honestly part of why this paper still holds up as a reference point five years later.
⚓ Why this actually matters for what we build
This paper is basically the reason building with AI feels so different from building with any earlier generation of software.
Good prompt vs bad prompt — this paper explains the difference. Every time you drop two or three examples into a prompt before asking your real question — here’s the tone I want, here’s the format, now do it for my actual case — you’re literally running the few-shot procedure this paper described. A bad prompt is usually just an instruction on its own. “Write a professional email.” The model has to guess what “professional” means to you specifically, and it’ll guess based on whatever’s most average across everything it’s seen — which is rarely exactly what you wanted. A good prompt looks more like a small classroom. A couple of examples of the output you’re after, plus a couple of clear traits you want the model to lock onto — the tone, the structure, the things to avoid. You’re not explaining the task in more words. You’re showing it. If a model isn’t getting your format right, a longer explanation almost never fixes it. Two or three good examples almost always do.
Why the same API key can power ten completely different products. Before this paper, building an AI feature for classifying tickets and a separate one for summarizing documents meant training two different specialized models. After GPT-3 showed in-context learning scales with size, one large model could plausibly do both — same weights, no retraining, just a different prompt. That’s the whole reason a small team in Mississauga can build a support triage tool on Monday and a document summarizer on Tuesday using the exact same underlying model.
Context window size is not just a spec — it’s a product decision. Every example you give a model in a prompt takes up context. The paper showed performance improves as you give more examples — but there’s a hard ceiling on how many you can fit before the window runs out. That ceiling varies by model: GPT-3 had a 4,000 token window, modern models go much higher, but the constraint is always there. This is also why long conversations start going sideways. As the context fills up, earlier information gets less attention weight — the model’s focus narrows toward what’s most recent and the beginning of your conversation effectively fades. So “how many examples is enough” isn’t just a prompting question — it’s a function of which model you’re using and how much window you’ve already burned through. If you’re building something that relies on long multi-turn conversations or big document context, the model’s context window should be one of the first things you check, not an afterthought.
Fine-tuning still has a job — but it’s a narrower one. If your task is well-defined and you have real labeled data, a fine-tuned smaller model can still beat a big general one. But if you’re still figuring out what the task even is, few-shot prompting gets you there faster and cheaper every time.
The founders who build the most efficient AI products aren’t necessarily the ones with the cleverest prompts. Increasingly, they’re the ones who understand enough of what’s actually happening under the hood to make the right call before they write a single line of code.
✨ SaugaTech Epilogue — Bigger wasn’t just more. It was different.
There’s a temptation to hear “175 billion parameters” and think the whole story is just about scale for its own sake. Bigger number, bigger flex. This paper is basically proof that’s not what actually happened. Scale didn’t just make the model a bit better at things smaller models could already sort of do. It unlocked something smaller models genuinely couldn’t do at all — learning a brand new skill on the spot, from a handful of examples, with zero retraining.
That’s a different kind of progress than “faster” or “cheaper.” It’s the kind that changes what a technology can actually do, period. And it’s exactly the kind of shift worth paying attention to as models keep getting bigger. The next real jump probably won’t come with a press release attached. It’ll show up quietly in a benchmark table somewhere, same as this one did back in 2020 — and the builders who notice it early will already have something shipped by the time everyone else catches up.
We’ll keep going with this series. Next time, we’re looking at how models went from just predicting the next plausible word to actually following instructions like a helpful assistant would — and the training trick that made that leap possible.
See you on August 22nd for the careers panel.
Let’s keep building, Let’s keep learning, Together.
Team SaugaTech
CONNECT | COLLABORATE | INNOVATE