Is That AI Answer True? Teaching Students to Test What Chatbots Say About Science
By Ahmed, founder
Students now ask chatbots their science questions, and the answers arrive fast, fluent and confident. Sometimes they’re right. Sometimes they’re subtly wrong, and the tone is identical either way. Teaching students to say, “That’s a claim, let’s test it,” may be one of the most useful science skills we can give them.
Why confidence isn’t evidence
A chatbot answer is generated from patterns in text. It can be accurate, out of date, oversimplified or simply wrong, and it rarely signals which. The same goes for forum posts, adverts and social media videos. The skill isn’t “never trust AI.” It’s knowing what would count as evidence, and checking it when it matters.
This fits what science already teaches: a claim is only as strong as the evidence behind it.
A classroom approach
Here’s a structure that works for Grades 5 to 7.
1. Show an AI answer as a claim. Print a chatbot answer to a science question such as “Does a heavy object fall faster than a light one?” or “Does salt water melt ice faster than fresh water?” Ask students to underline what the answer claims.
2. Turn it into a test. What would we do to check? What would we change, keep the same and measure? Students write a fair-test plan.
3. Run the tests that fit in a lesson. Two easy ones: drop a full and an empty matching bottle together from the same height, and place ice cubes in matching cups of fresh and salty water and compare melting. Both need only everyday materials.
4. Grade the answer. Was it correct, partly correct or incorrect? What did it leave out? Was the confident tone justified?
5. Discuss. Why might a chatbot get this wrong? What should we do before relying on an answer?
Our lesson Is that AI answer true? follows this plan with printed answers, two tests and a grading rubric.
Habits to build
- Ask “How do you know?” of every source, including a chatbot.
- Look for the evidence, not the tone. A confident paragraph without any evidence isn’t more reliable.
- Test what you can. Some claims can be checked in ten minutes with a cup of water.
- Compare sources. Do two independent sources agree? What do they each rely on?
- Check the scale. “Heavier objects fall faster” is nearly true when you compare a feather to a rock, because of air resistance, and false for two compact objects. Details matter.
Compare four sources
A second activity is to compare four sources making the same claim, such as a study summary, an advert, a forum post and an AI answer. Students rate the strength of the evidence behind each and then run a fair test to settle it. That’s the idea behind Which source should we trust?, where the claim is about paper towel absorbency, and the answer comes from students’ own data.
What not to do
Don’t teach students that AI is useless or that tests always beat reading. Good science uses both: reading widely, then checking claims that matter. And don’t let a failed AI answer turn into a gotcha. The goal is a habit, not a laugh.
A note on our own materials
The lessons in the Test the Claim unit use printed answers, so students aren’t using chatbots in class, and no accounts or student data are needed.
Where to go next
Explore the whole Test the Claim unit, including Does salt make water boil faster?, a myth students can settle with a thermometer. Pair it with our guide to claim, evidence and reasoning. Sign up for three lessons free.