Build the Panic, Then Sell the Cure

There's a sentence in 404 Media's reporting on AI book-banning tools that I keep coming back to, because it describes a machine I've watched get built one vendor pitch at a time. It's from Laney Hawes, a volunteer with the Texas Freedom to Read Project: "All of this is not real — it's manufactured."

She's right, and the manufacturing has a supply chain. A state passes a vague law about "harmful" or "indecent" material. Districts panic about liability. Companies appear, selling AI scanners that promise a "defensible process." The scanners over-flag, that's what they're built to do, and the flags become paperwork, and the paperwork becomes removals. Then next year the same companies sell the upgrade. None of it requires a single actual instance of the harm the law names. The market runs on the fear of the harm, which is a renewable resource.

I want to walk through how that machine works, because the people running it are getting good at making it look like neutral compliance, like a librarian doing due diligence with a better tool. It isn't that. It's the censorship side learning the same trick the vendors learned years ago: turn a public institution's obligations into a recurring bill.

1. The tools don't do what they claim, and that's not the point

Start with what these things actually are. 404 Media documented a tool called BLOCKADE (it stands for "Blocking Lustful Overzealous Content, Keeping Away Depravity and Extremism," which tells you most of what you need to know) that takes a book file, runs it against a list of roughly 300 flagged words with severity scores, and asks a commercial chatbot (xAI's or OpenAI's) to rate the book's "appropriateness." The script defines "educational inappropriateness" outright as "content offensive to conservative values," and instructs the model not to explain its decisions.

Sit with that for a second from the desk side. A keyword tally is not a reading. Asking a model to score a book against "conservative values" doesn't define those values: it outsources the definition to whatever the model guesses they are. Jeremy Blackburn, who directs the Institute for AI and Society at Binghamton University, put the problem plainly to 404 Media: there's "a lot of responsibility being abdicated" when you hand a chatbot the job of deciding what counts as pornography with "no real thought into how to evaluate what comes out."

The output is bad in a specific, predictable direction: it over-flags. Even the people using these tools say so. And the bad output is the feature, not the bug, because the goal was never an accurate reading of a book. The goal is volume: enough flagged passages, formatted to look official, to attach to a formal challenge. Strip a sentence from its page, score it, print it on letterhead, and to a review committee that never read the book it looks like an expert finding. PEN America's Kasey Meehan named the through-line: take the shortcut, and "you're going to end up pulling so many books that tend to be the most targeted anyway."

The tool doesn't have to be right. It has to be fast, and it has to look like process.

2. The Utah case: own the format, then sell the scanner

Here's where the market gets sharp. 404 Media reports that Rated Books, the Utah-based operation run by activist Brooke Stephens, which hosts the book reports Moms for Liberty members produced before that group wound down, helped popularize the scored-report format itself: the official-looking "this book is porn" document built to be attached to a challenge. That's the demand side. It trained committees and activists to expect a numeric verdict.

And the method isn't a secret: it's a tutorial. In a how-to video on her own YouTube channel, "How to Challenge a Book in DSD" (Utah's Davis School District), Stephens screen-shares a written instruction sheet and walks viewers through it. The steps don't ask anyone to read the book:

"Open up the report and copy the excerpts. Paste the above excerpts into AI, along with the brightline description… and ask AI to list any brightline violations."

That is the whole method, in her own words: copy the pre-collected excerpts out of a report (the ones I reviewed are stamped "Generated by RatedBooks.org"), paste them into a chatbot, ask it to flag "brightline" violations of the state standard, and attach the output to a challenge form. The book itself is never read by anyone in the chain. It is, by design, a workflow for producing challenges at volume: the human step is copying and pasting, and the reading is handed to a model that, as section 1 showed, is built to over-flag.

And there's a part a review committee can never see: which AI even judged the book, or how. The "AI" in these reports is a black box by design. Stephens has described her own process: in a January post on the Rated Books Facebook page, she said the team runs excerpts through multiple chatbots "to see which AI tool performs best," then overrides the model whenever it isn't severe enough: "AI will think consensual BDSM is not a 5," she wrote, "but our definition of a 5 is just the inclusion of BDSM." She's candid that the AI over-applies, too, reframing it as a feature she calls "Flagged by AI for Local Review." So the score attached to a book isn't a neutral machine output anyone could audit. It's whichever chatbot they chose that day, adjusted by hand to be harsher, with no disclosure of the model, the prompt, or whether it would return the same answer tomorrow. The committee gets a number and a verdict; it never sees the slot machine behind them.

Rated Books is also not one parent with a spreadsheet. It's a node in an organized network, and the public filings show it. Stephens incorporated Rated Books as a Utah nonprofit in May 2025 and runs it as president; she's also the education director of Utah Parents United, the state group that coaches parents through filing challenges. The overlap is on the record: the board of another of her nonprofits, Empowered Parents, has listed the same three people who lead Utah Parents United: Stephens, its president Nichole Mason, and its marketing director Corinne Johnson. And Rated Books' own founding board, per its incorporation paperwork, includes a Kimberly Fletcher of Branson, Missouri, the name and hometown of the founder of Moms for America, a national conservative organization. So the "this book is porn" report that lands on a district's reconsideration table isn't one neighbor's opinion. It's the output of a small, interlocking, nationally-connected operation, and the AI is what lets that operation scale.

This month, per the same reporting, the National Book Rating Index, described as a Rated Books affiliate project, began selling access to NarraTrue, an AI content scanner. Five dollars buys a CSV of specific page numbers and verbatim excerpts: the raw flagging input that feeds the exact challenge format Rated Books popularized.

For a long time the open question was who profits. 404 Media was careful not to say Stephens personally does, and I'll only go as far as the documents; but the documents go far. The National Book Rating Index, which sells the $5 NarraTrue scanner, presents itself as a neutral public index. Per its own site, it's operated by S&S Apps, LLC, a for-profit software company that, by its own description, was co-founded by Brooke Stephens and her husband, Aaron Stephens, who is listed as the LLC's manager at the same Layton address as the Rated Books nonprofit. So one household runs both the nonprofit that popularized the reports and the for-profit that builds and sells the AI tools built on them. That isn't double-billing, and I'm not alleging anyone broke a law. It's something quieter and more documentable: the family that defines the problem also owns the company selling the fix.

And the product's accuracy problem is already on the record, against the people deploying it. Jessica Horton, who runs the watchdog group Let Davis Read, told 404 Media she has won appeals of committee decisions specifically because those decisions leaned on Rated Books reviews that took books out of context. "Committees are basing their decisions off of that biased information," she said, "and so they're going to be more predisposed to remove books because the only thing they're seeing is a red flag saying, 'Hey, this book is porn.'" The tool has a documented failure rate. It's being sold anyway. That gap, known-unreliable, sold regardless, is the tell that you're looking at a market, not a method.

3. The vendor case: sold to the district, by the district's own crisis

The other end of the market sells straight to schools. The Texas company BookmarkED pitches districts an AI "book intelligence" dashboard as the answer to their compliance liability under the state's new book laws. 404 Media obtained emails from the company to New Braunfels ISD promising a "truly defensible process," "a review process you can stand behind." The district fast-tracked a contract of nearly $9,000 while maintaining it was still "exploring." Per the Texas Freedom to Read Project, NBISD has removed more than 1,400 books to comply with the new laws, with new purchasing suspended indefinitely.

What makes this more than one district's bad call is that the staff using the tool could see it wasn't working. In a November 2025 email 404 Media obtained, an NBISD elementary librarian wrote that BookmarkED "seems that if it has any content at all, it flags rather than taking it as a whole." "Taken as a whole" is not a casual phrase: it's close to the actual legal standard these laws are supposed to apply. The librarian was describing, in real time, a tool failing at the one thing it was bought to do.

And the wider reporting backs her up. Texas Standard found BookmarkED had signed at least $130,000 in contracts across the state; librarians said the product "did nothing that it said it was going to do effectively"; and in eight districts with usage data, none had more than 0.04% of parents actually using the parent-facing features the law was sold to enable. A six-figure statewide spend to serve a fraction of one percent of parents, with the actual function being collection removal, not parent choice.

Meehan's warning to districts is the one I'd staple to every procurement file: be wary of the rent-seeking baked in, "if not for the 'grifty' energy these companies give off, then for the local decision-making power that's being abdicated to Silicon Valley":

"Your state passes harmful legislation that removes and censors books, and then you have companies appear that then want to charge districts to review their collections."

That's the whole machine in two sentences.

4. Why this rhymes with everything else here

If you've read the rest of the Investigations, this shape is familiar. The work this site mostly tracks is the lending side: who owns the systems public libraries use to lend digital books, and how that ownership turns a public good into a recurring bill. The censorship side is a different fight, but it's running the identical play: manufacture a scarcity (here, of "compliant" collections), then sell the public institution a subscription to relieve it. Same grammar, different object.

And both come down to the same thing: the binding constraint sits above the app, not inside it. On the lending side, that's publisher licensing and first-sale law. Here, it's the statutes: Utah's HB 29 "bright-line" rule, Texas's SB 13, and now the federal H.R. 7661, the "Stop the Sexualization of Children Act," introduced in February and advanced out of the House education committee, which would tie federal funds to removing "sexually oriented material" and writes gender dysphoria and "transgenderism" directly into that definition. The AI tools don't create the censorship. The laws create it; the tools just industrialize the paperwork and bill you for the privilege. Following the money runs up to the legislature, not just to the vendor.

Which is also why no amount of "but our AI is more accurate" fixes anything. A better scanner pointed at a worse law removes more books, faster. The tool's quality follows from a choice the tool is designed to launder: the choice to treat a whole book as the sum of its most-flaggable sentences.

5. What this looks like from the desk

This is a founding-document site, not an autopsy one, so here's the forward part: what a working library person can actually do when one of these reports lands on the reconsideration table.

Make the whole-book standard non-negotiable. Nearly every one of these laws says material is judged "taken as a whole." An AI flag list is, by construction, the opposite of that. When a challenge arrives backed by a scored report, the first question for the committee isn't "how many flags?" It's "did anyone read the book?" Horton's appeals in Utah won on exactly this ground. Out-of-context is not a vibe; it's a procedural defect you can name.

Treat "won't explain its reasoning" as disqualifying. Blackburn's point cuts both ways: a tool that's told not to explain itself can't be evaluated, and a flag that can't be evaluated isn't evidence. If a submitted report can't say why beyond a severity score, that's not rigor. It's the absence of rigor wearing rigor's clothes.

Read the procurement like a vendor contract, because it is one. If a "book intelligence" product comes through your district, the questions are the same ones we ask of any vendor here: who owns it, what does it do with your collection data, what's the recurring cost, and, the one these companies hate, what's the independent evidence it does what the pitch says? When the answer is "$130,000 statewide and 0.04% parent uptake," you have your answer.

Keep the reconsideration process human, slow, and on the record. The whole value proposition of these tools is speed: flag faster, remove faster, generate more challenges than any committee can carefully hear. The counter is not a faster tool. It's a process that refuses to be rushed, documents its reasoning, and puts a person who read the book in the room. That's not nostalgia. It's the part the machine can't manufacture.

The censors are reaching for AI, in Hawes's words, "to give themselves the illusion of control." The reporting suggests they're mostly buying that illusion from each other, at public expense. We don't have to buy it for them.

What's already reported, and what this adds

Most of the facts above are other people's work, and they deserve the credit. The tools, the BookmarkED contracts, and most of the quotes in sections 1 and 3 come from 404 Media's original reporting, cited throughout. That a family business sits behind Rated Books was flagged before this piece, too, by an independent newsletter, and by a local Michigan reporter who noted a "husband-and-wife team."

What this piece adds is documentation. The conflict of interest isn't an allegation here; it's on the filings. The Rated Books nonprofit and the for-profit S&S Apps that operates the National Book Rating Index share a founder, a married couple, and a home address, all in the Utah corporate record. The interlocking boards, the same handful of activists running Rated Books, Empowered Parents, and Utah Parents United with a seat reaching Moms for America, come straight from the incorporation papers. The point of pulling those records was simple: to move the story from "people are saying" to "here is the document."