Hi SaugaTech community,
Meetup #11 happened last week, and if you were in the room at IDEA Mississauga, you already know it ran long. “Tech Careers in 2026” was supposed to wrap on schedule. It didn’t. The panel kept going, the questions kept coming, and by the time we finally had to call it, most of the room just moved to pizza and kept the conversation going anyway. Good problem to have, and it’s got us thinking about how to scale this thing up — more on that below.
For now, we’re picking the series back up. Part 4 of AI Under the Hood is the paper that finally taught AI to behave like something you’d actually want to talk to.
🚀 First Things First - Volunteering with SaugaTech
Meetups like #11 don’t run themselves, and honestly, they’re starting to outpace what our small core team can carry alone. So we’re trying to do something we haven’t really done before: putting together a formal volunteering program as a part of our newly incorporated non-profit entity.
Three areas, specifically to start off with:
Event Curation — helping shape what future meetups actually look like. Picking themes, lining up speakers and panelists, thinking through format.
Content Management — this newsletter, recap posts, social content, the website. If writing or editing is your thing, this is where we’d love to have your help.
Mentoring & career guidance— working with members of the community, individually or as a group, share your own experiences, help them shape their career direction
None of these need to be a huge time commitment. A few hours a month, pointed at something you’re actually good at, goes a long way for a community this size.
Here is the link to the Volunteering Interest form. We look forward to hearing from you. Feel free to share it in your networks to those who might be interested in helping us out
Anyway, coming back to today’s topic.
The Practical Problem Statement
Picture the most well-read new hire you’ve ever had on a team. They’ve absorbed the entire internet, they can write in any style, hold a conversation on nearly any topic, and finish a sentence in exactly the way it statistically tends to get finished. Ask them a direct question, though, and something odd happens. Half the time they just keep talking, adding more and more plausible-sounding text, because that’s the only thing they were ever actually trained to do. Nobody taught them what it means to help.
That’s roughly the state language models were in before this week’s paper. Technically brilliant, occasionally useless, and this is the paper that fixed it.
If you’ve been following along, this one closes a loop we’ve been building since Part 1. Attention gave models the ability to read context. GPT-3’s few-shot learning gave them the ability to adapt on the fly. RAG gave them a way to look things up instead of guessing. This paper is the one that finally taught them to behave like something you’d actually want to talk to.
Grab a coffee. Let’s get into it.
⚓ The Paper: “Training Language Models to Follow Instructions with Human Feedback” (2022)
In early 2022, a team at OpenAI published the paper behind InstructGPT, the direct predecessor to ChatGPT. The core discovery is almost embarrassingly simple to say out loud: making a model bigger does not make it better at doing what you actually asked. Size and helpfulness turned out to be two completely different problems, and this paper is the one that finally tackled the second.
The problem the researchers identified
A raw language model like GPT-3 is trained on one objective: predict the next word on a page pulled from the internet. That’s a genuinely useful skill, but it’s not the same skill as “follow this instruction helpfully and safely.” Ask a raw model to write a professional email and it might just as easily continue your prompt with a list of other emails you could write, or a joke about email, or nothing useful at all. The researchers call this a misaligned objective. The model is optimizing for “sounds like the internet,” when what the user actually wants is “does what I asked.”
This gap shows up as three separate failure modes, and it’s worth naming them individually because you’ll recognize all three from any raw model you’ve used.
Models make things up.
Models sometimes ignore what you actually asked and answer a nearby question instead.
And Models can drift toward toxic, biased, or simply inappropriate output, because that’s present somewhere in a training set the size of the internet.
The core idea — teach the model with a manager, not just a textbook
The fix here has three stages, and the analogy that holds up best is a new hire’s first few months on the job.
Stage one is supervised fine-tuning. Human contractors write out example responses to a wide range of prompts, showing the model what a genuinely helpful answer actually looks like. Think of this as the onboarding binder. Here’s what a good customer email sounds like. Here’s how you handle an ambiguous request. The model gets fine-tuned directly on these examples.
Stage two is reward modelling. Contractors are shown several different model responses to the same prompt and asked to rank them from best to worst. That ranking data trains a second, separate model, called the reward model, whose entire job is to look at a response and predict how a human would rate it. This is the manager who’s read enough of your work to guess, fairly reliably, whether you’re about to get positive feedback or a note back asking you to redo it.
Stage three is reinforcement learning. The original model gets fine-tuned again, this time using the reward model as a coach, adjusting its behaviour over and over to chase a higher predicted score. The technique used here is called PPO, Proximal Policy Optimization, which is really just a careful way of nudging the model’s behaviour without letting it swing wildly or forget what it already knew how to do.
Put together, this is RLHF. Reinforcement Learning from Human Feedback.
And the researchers found something that should give every builder in this community real pause about the “just make it bigger” instinct: a 1.3 billion parameter InstructGPT model, fine-tuned this way, was preferred by human raters over the raw 175 billion parameter GPT-3, a model over a hundred times its size.
The evidence that it worked
The headline number is that stark preference gap. In direct comparisons, InstructGPT’s output was preferred over raw GPT-3 output 85% of the time, and preferred 71% of the time even when GPT-3 was given a carefully hand-crafted prompt meant to coax better behaviour out of it.
The hallucination numbers are the ones worth sitting with a little longer, especially after we spent all of Part 3 talking about RAG. On closed-domain tasks, where the correct answer should come entirely from what’s already in the prompt, InstructGPT made things up about half as often as GPT-3 did, a 21% hallucination rate compared to 41%. On the TruthfulQA benchmark specifically, it produced truthful and informative answers roughly twice as often.
There were smaller wins too. When explicitly asked to respond respectfully, InstructGPT generated about 25% fewer toxic outputs than GPT-3. And the model showed a genuinely interesting kind of generalization: even though the fine-tuning data was almost entirely English, InstructGPT got noticeably better at following instructions in other languages and at answering questions about code, tasks it was never specifically trained to improve on.
Where it fell short
InstructGPT still hallucinated, just less often, and it developed a couple of new bad habits along the way. It would sometimes hedge on simple questions that had a clear answer, giving a wishy-washy “there are many perspectives” response instead of just answering. It could also be led astray by a false premise buried in a question, going along with something untrue instead of catching it.
And there’s a bigger, more uncomfortable question sitting underneath all of it. Whose preferences was this model actually trained on? The paper is upfront that the labelers were a small, specific group, mostly English-speaking, hired through a couple of platforms, following instructions written by OpenAI researchers. “Aligned” here means aligned to that particular group’s judgment calls, not some universal standard of correctness. That distinction matters more the more these models get deployed into contexts none of those labelers ever saw.
The side effect nobody saw coming
Worth a quick detour here, because this paper didn’t just introduce a training technique. It quietly created an entire labour category. RLHF runs on a constant supply of humans ranking model outputs and writing example answers, and the paper’s own appendix is candid about it: about 40 contractors, hired through Upwork and a company called Scale AI, screened for how well their judgment matched the researchers’.
That demand didn’t stay small. Scale AI’s revenue went from roughly $250 million in early 2022, right as RLHF took off, to around $750 million by late 2023, and Meta later paid $14.3 billion for a 49% stake in the company in 2025. The broader data annotation and labeling market is projected to grow from about $1.2 billion in 2024 to $10.2 billion by 2034. Newer entrants like Surge AI, Labelbox, and Mercor have built entire businesses competing for the same kind of work: paying people to rank AI outputs so a reward model can learn what “better” looks like.
It’s a strange thing to sit with. The paper that taught machines to sound more human ended up creating a lot of very human jobs to make that possible.
⚓ Why this actually matters for what we build
This paper is the reason a raw model and a genuinely usable assistant feel like completely different products, even when the underlying architecture is identical.
It explains why “the base model” and “the chat model” behave so differently for the exact same task. If you’ve ever pulled a raw, non-instruction-tuned model off a hub and been confused by how oddly it responds compared to something like ChatGPT or Claude, this is why. Same architecture underneath, wildly different training on top of it. When you’re choosing between a base model and an instruction-tuned one for a product, you’re choosing between raw capability and actual usability, and for almost anything customer-facing, usability wins.
It’s also a genuine argument for a smaller, well-trained model over a bigger, raw one. A 1.3B model beating a 175B model on real user preference is a big deal if you’re building anything cost-sensitive. Before you reach for the biggest model on the market, it’s worth asking whether a smaller, well-aligned one might actually serve your users better, and cost a fraction as much to run.
It also gives a name to something every builder eventually notices the hard way: models will over-hedge or misread a false premise, and it’s not really a bug so much as a side effect of how they were trained to be careful. If your product needs a direct, confident answer to a well-defined question, it’s worth testing specifically for these two failure modes, because they show up more than people expect.
And this is where the thread to Constitutional AI actually starts. Anthropic’s Constitutional AI, which we’ll cover properly in a future edition, took this same basic RLHF framework and asked a pointed question: what if the model itself could help judge its own responses against a written set of principles, instead of leaning entirely on humans to hand-label every example? It’s not a replacement for the ideas in this paper so much as an evolution of them, aimed at reducing how much this process depends on human labelers seeing every harmful case in advance, and at making that alignment process more consistent and more scalable as these models get deployed more broadly.
✨ SaugaTech Epilogue — four papers, one continuous story of evolution
Four parts into this series now, and it’s worth stepping back and seeing the shape of the whole thing.
Attention taught a model how to actually read what’s in front of it, understanding context instead of processing words in isolation.
Few-shot learning, once that model got big enough, taught it to adapt to a brand new task on the spot, just from a couple of examples.
RAG gave it a way to reach outside its own training and check a real source instead of guessing.
And this week’s paper, RLHF, taught it to actually behave, to take everything it’s capable of and point it at what the person in front of it actually wants.
Understand, adapt, look things up, behave helpfully. None of those four ideas replaced the one before it. Each one is sitting underneath the model you used this morning, doing its job quietly, all at once.
There’s a decent lesson for builders buried in the order these ideas arrived in, too. Raw capability came first, and it wasn’t enough on its own. It took a genuinely separate effort, actually listening to what people wanted and training toward that, to turn a very well-read machine into something worth talking to. Worth remembering the next time a product feels technically impressive but somehow still misses the point.
See you soon.
Let’s keep building, Let’s keep learning, Together.
Team SaugaTech
CONNECT | COLLABORATE | INNOVATE