The team at Tom’s Guide has been observing and reporting on the recent AI chatbot revolution with a consequential question looming in the back of our minds: Can the likes of ChatGPT replace us when it comes to reviewing tech devices?
Equal parts concerned and curious, we prompted OpenAI’s chatbot to write review introductions of products we’ve tested over the years. ChatGPT is trained with information up to 2021, so we couldn’t ask it to spit out a review of the iPhone 14 Pro Max or PSVR 2, but we did use some stand-out devices from the archives as our guinea pigs.
What’s more, we challenged our staff to call out copy written by AI compared to copy written by Tom’s Guide’s expert product reviewers — without being told which was which. Watch the video below to see how it went:
Humans vs. AI: Who writes the better product review?
Product reviews are important to helping our readers make informed purchasing decisions. That’s especially true when it comes to first-generation devices sporting new technology. Take the Samsung Galaxy Z Flip (2020), for example.
Our Samsung Galaxy Z Flip review, written by managing editor Philip Michaels, demonstrates the day-to-day use of an innovative smartphone from the perspective of someone with many years of mobile testing experience.
Reading aloud Michaels’s review introduction and one crafted by ChatGPT (but not informed of which was which), computing writer Tony Polanco said the AI version seemed “too dry” to have been written by a human. Editor Kate Kozuch correctly recognized that one line from the real review — “a closed-up Z Flip fits easily in my hand and my paws aren't exactly meaty” — sounded like something Michaels would say.
We also experimented with a review of the original Apple Watch. While ChatGPT’s version featured accurate specs and a reasonable overview of Apple’s debut smartwatch, global editor-in-chief Mark Spoonauer said “it seemed too generic to be an actual review.” Spoonauer happened to write the original review for the first-gen Apple Watch in 2017, so he knew which of the two introductions told a real story.
Finally, we compared reviews of the Amazon Echo Spot, the bygone circular smart display. While Polanco found both the reviews “read very well,” Spoonauer compared the AI version to “an Amazon product listing.” Editor-in-chief Mike Prospero easily identified his own writing from the original review of the product, too.
Should we be worried about our jobs?
All four Tom’s Guide folks correctly distinguished the AI-written reviews from the human ones. After reading through all the rounds of this experiment, Kozuch recognized how ChatGPT often used “ultra-generic phrases like unique and sleek” to describe products. She wasn’t impressed.
“AI is not going to replace the human experience of living with and using a product,” Spoonauer said. “I’m not worried about it coming for our jobs at the moment, but it’s clear that the industry is using these tools.”
Prospero said that there’s, “too many problematic things about ChatGPT.” While we were able to spot a real review from a one generated by a chatbot, not all consumers might see the same cues. And it could become murkier as Bing with ChatGPT and Google Bard emerge as AI-powered destinations for review-based buying advice.
