How do you build AI that teachers trust?

Most schools aren’t asking whether to use AI anymore. They’re asking which AI to trust.

That’s a much harder question. And the answer isn’t accuracy scores or feature lists. It’s about whether a tool was built with classrooms in mind from the start, before it ever reached a student.

When comparing edtech tools, it’s essential to understand exactly how each company builds its AI. In conversation with MacKenzie McCowan (AI Governance Specialist at Atomi), we explore why accuracy alone isn’t enough to trust an AI tool, what responsible AI governance looks like and what goes into building AI that’s safe for the classroom.

Watch MacKenzie's full interview on Youtube.

Why isn’t accuracy enough to trust an AI tool?

Accuracy is a persistent challenge presented by AI. AI hallucinations (where the tools fabricate ungrounded or false information) remain common. One study from the Columbia Journalism Review found that eight of the most popular generative search tools provided incorrect answers to more than 60% of queries.

But as AI’s capabilities improve, accuracy is no longer the biggest challenge in a school context. As MacKenzie shares, “It’s not that difficult to develop an AI product that is largely accurate. It’s never going to be perfect, but with the tech advancements, we can get pretty close.”

Trust moves beyond technical capabilities. It’s the design, governance and value system that informs the technology, and it requires edtech companies to think very differently.

It is difficult to produce an AI that you can trust, not only because it’s accurate, but because you know your data is safe, the content and tone will be aligned with your students’ age and ability, and you can predict how a tool will behave in a given scenario.
— MacKenzie McCowan

What goes into building AI that’s safe for a classroom?

Off-the-shelf AI models aren’t built with pedagogical guardrails or age-appropriate design in mind. Not only are generic chatbots susceptible to inaccurate outputs and data privacy risks, but they can also pose a threat to cognitive development.

One study of year 9 to 11 maths learners using a basic GPT-4 tutor found that these students performed 17% worse on follow-up exams, indicating the tool functioned as a ‘crutch’ rather than a purpose-built coach or revision tool. Source.

In contrast, AI that is purpose-built for the classroom is responsible, transparent, and accountable. It works within the school curriculum and pedagogy that teachers are already familiar with, and is designed for the age and context of each learner.

At Atomi, specialists across governance, curriculum, education and product design collaborate to build teaching best practice into AI tools from the start.

In practical terms, that means:

• Built on the curriculum, not data scraped from the web, to improve accuracy and ensure material is appropriate for students in the classroom.
• Student data remains secure; every AI vendor is contractually blocked from training their models on Atomi data.
• Pedagogically reviewed, ensuring every AI feature’s output is benchmarked against peer-reviewed research, along with feedback from students and teachers.

It’s the background data, like curriculum information, teacher marking input, education theory and privacy protections, that produces something wildly different from an off-the-shelf model trained by scraping data from the internet.
— MacKenzie McCowan

What does responsible AI governance look like in practice?

Responsible AI governance is part of what makes a tool predictable and trustworthy.

Mac shares that good AI governance should be mostly invisible. By the time something goes wrong in the classroom, the groundwork should already be in place to ensure obligations are known, accountability is assigned and escalation routes are proactively established.

In more specific terms, responsible AI governance involves:

• Clear lines of accountability built months before the tool reaches the classroom, ensuring responsibilities are well defined.
• Student data is contractually protected to ensure models aren’t being trained on students’ information and are held to strict compliance and privacy standards.
• Teachers have full visibility at every stage, ensuring they can see every response it generates and shape, act on and build on it.

As Mac shares, “A lot of features ship on the basis that they’re useful when they work. The right test is the opposite.”

The question I wish more edtech companies asked before shipping is: What happens when this fails? Because it likely will, at some point in some circumstance. Who absorbs the consequence? If the answer is the teacher or the student, your product isn’t ready.
— MacKenzie McCowan

How should teachers evaluate an AI tool before using it in the classroom?

Carefully evaluating AI tools is key to ensuring trustworthy technology in the classroom. The best edtech providers are transparent, committed to supporting diverse learners, and deliberate about how they use AI.

AI designed for the classroom doesn’t undermine learning or enable cheating. In fact, it’s designed to do the opposite: surface student thinking, rather than replace it.

AI can make learning more accessible for students who are falling behind, can provide extensions for students who are ahead of the curve, and overall make learning more personalised without compromising the essential pedagogical values that make learning successful.
— MacKenzie McCowan

Consider asking edtech companies these questions when comparing your options:

• Whose data does this use, and how is it protected?
• Is this tool designed for my students’ age and context, or adapted from something generic?
• Can I predict how this will behave before I use it in class?
• When something goes wrong, who’s accountable?

Building AI teachers can trust begins before the tool reaches the classroom

Trust in an AI tool is earned long before a student interacts with it. It’s built through the governance decisions edtech companies make prior to launch.

The best AI tools are shaped by the curriculum, pedagogy best practices and a contractual commitment to protect student privacy.

While accuracy is important, it’s only the starting point in a classroom. Instead, it’s important to look for AI tools that prioritise what classrooms need: learners gaining immediate, personalised support, and teachers gaining time back to do the work that defines great teaching.

Want to see what AI built for the classroom looks like in practice? Take a look at how Atomi's AI works alongside teachers, grounded in the curriculum and pedagogy you already trust.