Act one
Building the machine
An AI model isn’t programmed with rules about books. It’s grown, in five steps. Watch for the amber note at each one: that’s where a person’s choice slips in and stops being visible.
-
Pour in the text
A model starts as nothing. You build it by feeding it an enormous pile of human writing scraped off the internet: websites, forums, Reddit, Wikipedia, some books. That pile is everything it will ever “know.” Notice what’s usually not in it: the book you’ll later ask it to rate. It probably never read that book. Even for an old classic whose text did leak into the pile, it cannot tell a line it memorized from one it made up. Mostly it knows books secondhand, from whatever the internet said about them.
A person decided thisSomeone chose what goes in the pile and what stays out. Include a lot of one kind of writing, leave out another, and the machine’s whole sense of “normal” shifts before it has learned a single thing.
-
Make it guess the next word, a billion times
Training is one boring trick at enormous scale: hide the next word, make the machine guess it, tell it whether it was right, nudge it, repeat. Do that across the whole pile and it becomes a very good autocomplete. It is not building a card catalog of facts to look up. It is learning which words tend to follow which words, including our stereotypes, our moral panics, and our associations, because those are in the writing too. The whole loop, in one picture:
the pile of text …the cat sat on the mat once upon a time… …poured the tea into a cup + billions more linesthe modelit fills in the blankThe dog sat on the mat
Run that loop billions of times and the guesses get sharper. Nothing here is a fact being looked up: the model just learns which word the pile made most likely to come next, then picks from those odds. Absorbed from the pileWhatever slant is in the pile becomes the machine’s reflex. If the text it ate treats a topic as shameful, it learns to treat that topic as shameful. Nobody typed that rule. It was absorbed.
-
Now it’s a dice machine, not a filing cabinet
Here is the part people miss. The finished machine does not store answers and look them up. Ask it something and it weighs the possible next words, then rolls the dice, and whoever runs it sets how loaded those dice are. Most leave a little randomness switched on, so the same question asked twice can give two different answers. A rating isn’t retrieved from a record. It’s drawn from a spread of odds.
See it yourselfThe live glass-box demo shows this directly: the same book, same machine, and the score lands on whatever number the dice came up. Sometimes a 3, sometimes a 4. That spread is the whole point.
-
Teach it manners, and values
A raw autocomplete will say anything. So the company runs a second pass: people rate its answers “good” or “bad,” and it learns to produce more of the “good” ones. That sounds neutral. It isn’t. Someone wrote the instructions those raters follow, the document that defines what “good” means. That document is a statement of values, and it is now baked into the machine. (This second pass is what people mean by a model’s “safety training,” or its “guardrails.”)
Someone defined “good”“Good” was defined by people you will never see, working toward a goal you were never told. From here on, the machine has a disposition, and it wears it as if it were common sense.
-
Whisper instructions before every question
Finally, whoever runs the machine gets to slip hidden instructions in front of every request: You are a strict content rater. Flag anything a parent might object to. The person reading the final rating never sees this line. It’s the cheapest lever of all: no training, no code, instant, and invisible.
Someone can change this, invisiblyTwo libraries can run the identical model and get opposite ratings, because one of them changed a sentence you’re not allowed to read.
So the number on the cover is not a fact about the book. It’s the output of a stack of human choices, dressed up as a measurement. Every mechanical step above has a thumb on it.
Interlude · made-up example, not real data
There isn’t one machine. There are thousands, each raised on a different diet.
Everything above describes one machine. But every model ever built had a different upbringing: not just a different pile of text (step 1), but different values and instructions layered on afterward (steps 2 through 5). Two machines can read the identical book and see two different things, because they were raised differently, end to end. Here is the same book handed to four of them.
The same book — a coming-of-age memoir with a few frank passages — handed to four machines that were each fed a different diet.
▲ The four numbers below are invented to show the idea. They are NOT real test results. (Real results start further down.)
-
Fed on medical and science texts1out of 5
Frank body words look ordinary to it. It has read thousands of them in clinical papers, so they don’t register as alarming.
-
Fed on published novels and library records2out of 5
It treats the book as literature and rates the whole story, not the two passages a challenge would pull out of context.
-
Fed on the open internet, arguments and all4out of 5
It absorbed years of people online calling this exact book obscene, and echoes that back as if it were its own judgment.
-
Fed on children’s books and school worksheets5out of 5
Its entire world is picture books and spelling lists, so anything more adult than that reads as off the scale.
Same book. Four upbringings. Four numbers, and no shared yardstick between any of them. The machine has no context and no way to get any; it can only grade the book against the upbringing it happened to have. Pick the machine and you have all but picked the number.
To be clear, the four diets above are invented to show the mechanic, not measured. For the real version, seven actual models rating the same three books, see the seven-model comparison, where real numbers disagree for exactly this reason.
Act two
Aiming it at a book
Now flip it around. Say you want a specific title flagged. You don’t need to touch the code, or understand any of it. Here are the levers, cheapest first.
-
Write the prompt like an advocate, not a referee
“Rate this as a protective parent.” “When in doubt, round up.” “Treat any LGBTQ content as sexual.” Same book, same machine; the number moves. It moves the instant you add one line, and that line is invisible to everyone who reads the rating.
-
Write the rubric so the answer is decided before the machine runs
Often the banning has already happened here, in the definition of the scale. If your rubric (the scoring rules) says “any depiction of a same-sex relationship = 5,” then the machine isn’t judging anything. It’s confirming a rule someone already wrote. The number just makes that rule look measured and objective.
-
Shop for a machine that already agrees with you
Different models hand back different numbers for the identical book. So pick the one that scores harshest by default. In this demo, a model with no built-in safety limits rated Gender Queer anywhere from 3 to 5 across identical runs, while a tiny, low-end model called Goodnight Moon a 4 for sexual content and recommended it for ages 3–7 in the same breath. Choose your machine, choose your outcome.
-
Run it until you like the answer
Because it’s a dice machine (step 3 of building it), you can feed the same book ten times and keep only the highest score. Or turn up the “temperature,” the dial that controls how much randomness the machine is allowed (how loaded the dice are, from step 3), so it swings harder. Nobody ever sees the nine runs you threw away.
-
Let reputation do the work
Remember, it never read the book. It pattern-matches the title and whatever’s been said about it online. A book that’s been loudly called “controversial” gets flagged for having been called controversial. Then the ban makes news, the news becomes training text, and the next machine learns the book is dangerous. The loop feeds itself.
Not a thought experiment
This is already a product
Services that sell automated “content ratings” of library books to schools already exist, and they run on the same logic as everything above: a prebuilt set of judgments, matched against a school’s catalog by title, and sold as if it measured the books. The school sends a spreadsheet, the service sends back a list of “violations” for a flat fee. No one reads the books.
You can watch the laundering happen in public record. A 2025 Utah state audit reported that 95% of a set of books “contained sexual conduct” — a scary number that, on a second look, measures almost nothing about libraries. The audit drew its whole sample from one such service’s own flag-list, so the 95% only says how often that one company’s flags matched a statute, not what’s actually on school shelves. A private product grading its own homework, then printed as a fact about the shelves.
None of this takes malice, or a conspiracy, or even much skill. It takes a rubric, a prompt, and a model, each chosen by someone with a goal, producing a number that hides the choosing. That’s the trick. It isn’t cross-referencing a spreadsheet. There is no fact at the bottom. It’s a person’s judgment, run through a machine that makes judgment look like arithmetic. Nobody has to intend a cover-up for the number to work like one.
The machine isn’t the authority. It’s the cover story.
If someone hands you a rating
Five questions that catch every trick
You don’t need to know how any of this works to test it. Every lever above has a plain question that exposes it. Ask these out loud, at the desk or the board meeting:
“Did a person read this book, or did a program read a blurb about it?”
Almost always the second. A number generated from a one-paragraph description isn’t a judgment about the pages; it’s a guess about the summary.
“Which AI gave you this, and would a different one agree?”
The score is a property of the machine, not the book. A different program, or the same one run again, routinely lands on a different number. One number shown by itself is a number that was chosen.
“Can I see the exact instructions it was given?”
A single hidden sentence, like “flag anything a parent might object to,” can move the score before anyone opens the book. If they won’t show you the prompt, they’re asking you to trust the part they’ve hidden.
“What does the scale say? What makes something a 5?”
If the rule is “any depiction of a same-sex relationship counts as a 5,” the machine didn’t judge anything. It confirmed a rule a person already wrote. The rubric is where the decision actually happened, so read it.
“How many times did you run it, and is this the highest number you got?”
Because the answer changes run to run, anyone can roll it ten times and keep the scariest one. Ask to see every run, not the one on the slide.
None of these are hostile questions. They’re the ones a real measurement would have no trouble answering, and a manufactured one can’t survive.