AI Evaluation Scorecard for Libraries

Eight criteria every AI feature should pass before going live.

This rubric was developed for the Little Schitt Creek Regional Library's Digital Reference Desk. Before any AI feature goes live, it must pass all eight criteria. No exceptions.

It's published here so other libraries can adapt it. Change the specifics to match your context, but keep the principles. If a tool can't pass all eight, it's not ready.

1 Privacy

Patron data is sacred. Period.

  • Does this feature store, log, or transmit patron data?
  • Is the session wiped when the tab closes?
  • Does any data leave the library network?
  • Could a patron's reading, searching, or browsing history be reconstructed?
Must pass

2 Transparency

If you can't explain it, don't deploy it.

  • Can we explain how this feature works in plain language?
  • Is the "How This Works" disclosure written and visible?
  • Could a curious 10-year-old (or a skeptical board member) understand the explanation?
Must pass

3 Local Control

Your infrastructure, your rules.

  • Does this run on library-owned hardware?
  • Does data stay on the library network?
  • Are we dependent on an external API or cloud service?
  • Can we operate this feature if the vendor disappears?
Must pass

4 Equity

Every patron means every patron.

  • Does this feature work for patrons with disabilities?
  • Does it work on low bandwidth connections?
  • Does it work for patrons with limited tech literacy?
  • Do accessibility modes (like Sensory Quiet Mode) still function?
Must pass

5 Bias Audit

Default to diverse. Test for gaps.

  • Have we tested for demographic bias in outputs?
  • Do recommendations surface diverse voices by default (not as an afterthought)?
  • Have we tested with queries that reflect our actual community demographics?
Must pass

6 Staff Impact

Augment the humans. Don't replace them.

  • Does this augment staff work or replace it?
  • Have staff been trained on how to use and think about this tool?
  • Were staff consulted during design?
  • Does the "Prefer a Human?" option remain visible and easy to find?
Must pass

7 Patron Benefit

The patron is the point. Not efficiency metrics.

  • Is the primary beneficiary the patron (not the library's internal metrics)?
  • Does this solve a problem patrons actually have (not one we imagine for them)?
  • Would patrons choose to use this if they understood how it works?
Must pass

8 Reversibility

If it's not working, we can stop.

  • Can we turn this off without losing data or breaking workflows?
  • Do staff know how to operate without this tool if needed?
  • Is there a documented rollback plan?
Must pass

Quick Reference

# Principle Core question Required
1PrivacyIs patron data protected and ephemeral?Yes
2TransparencyCan we explain this in plain language?Yes
3Local controlDoes data stay on library infrastructure?Yes
4EquityDoes this work for all patrons?Yes
5Bias auditHave we tested for bias in outputs?Yes
6Staff impactDoes this help staff, not replace them?Yes
7Patron benefitIs the patron the primary beneficiary?Yes
8ReversibilityCan we turn this off cleanly?Yes

How to Use This Scorecard

  1. Before development: Review all eight criteria with your team. If any criterion will be impossible to meet, reconsider the feature.
  2. During development: Test each criterion as you build. Don't leave evaluation for the end.
  3. Before launch: Walk through every question with staff present. Document your answers.
  4. After launch: Revisit quarterly. Conditions change. Models update. Community needs shift.
  5. When in doubt: If a feature can't clearly pass all eight, it's not ready. Better to wait than to deploy something that erodes trust.
A note on AI mistakes: AI tools can produce incorrect information ("hallucinations"). Any scorecard should also include a plan for how your library communicates this to patrons. At Little Schitt Creek, every tool displays a visible reminder that AI can make mistakes and patrons should verify facts independently.

Developed by Little Schitt Creek Regional Library with Unhinged Librarian. Free to use, adapt, and share. No attribution required, but appreciated.