Does Pangram Detect Any AI-Written Reviews on Amazon?

AI-generated content has become the norm and can feature in everything from academic journals to news headlines and academic submissions. Now, it’s become clear that AI may be affecting our shopping habits.

Amazon was the first online retailer to introduce a customer review feature, but the system is potentially being manipulated by large language models (LLMs) like ChatGPT, Claude and Gemini. Leaving AI-generated reviews is an illegal practice under U.S. federal law, and to address this, Amazon is ironically tackling the problem with AI.

The company has stated that sophisticated tools are being used to flag suspicious review histories and remove AI-generated reviews before a customer encounters one.

Our Study

Our research suggests Amazon’s still has work to do.

At Pangram, we wanted to find out how many reviews on Amazon are AI-generated. To do this, we ran nearly 30,000 customer reviews across 500 of Amazon’s best-selling products through our AI Detector. We looked at ten best-selling product categories, including baby, toys and games, laptops, medical devices, wellness and relaxation, beauty, and furniture.

Aren’t ‘Verified Purchase’ reviews supposed to be trustworthy?

Our study found that nearly all AI-generated reviews (93%) have the ‘Verified Purchase’ badge, which is used as a reliable trust signal by customers.

But if a review was written by AI, how trustworthy is it?

Some reviews are likely written by sellers using LLMs to simulate feedback and push their product rating higher. Others may be written by well-meaning customers who don’t have the time to write one themselves or can’t find the words to describe their opinion of a product.

It doesn’t matter, because the end result is a potentially inaccurate depiction of a product.

Which categories are the worst offenders?

Of the ten product categories analyzed, baby products top the list for AI-written reviews. This is especially concerning when you consider that busy parents often rely on quick and trusted recommendations for essentials like formula and diapers:

Baby: 5.2%

Beauty: 5%

Wellness & Relaxation: 4.4%

Gardening: 1.7%

If AI-written reviews are inflating product ratings, it’s not just misleading – it could directly affect purchasing decisions that impact child health and safety.

Positive reviews were more likely to be AI-written, as just 10 percent of AI-generated reviews were 1-star, compared to 21 percent of human-written ones. A staggering 74% AI reviews were five-star.

Why does this matter?

To their credit, Amazon has been trying to address AI-created reviews, but based on what the study found, their efforts aren’t working well enough.

The volume of AI-generated reviews slipping through the cracks suggests Amazon still has work to do.

AI-generated reviews undermine trust, and when online reviews are one of the biggest things we rely on when deciding what to buy a product, we may start to doubt if a purchase is worth it.

How to spot AI-generated reviews

When reading reviews, look for mentions of specific product details. Generic comments or ‘fluffy’ language can be a sign that a review is AI-generated.

It’s also important not to take the ‘Verified Purchase’ badge as a guarantee of authenticity. The study revealed that 93% of AI reviews have this badge, so this is no longer a reliable symbol of trust.

If you intend to leave a review, resist the temptation to use ChatGPT or another LLM. Your honest experience and opinion matter and could help people make an educated decision about purchasing a product.

You can also use Pangram’s AI Detector to verify. Our model detects AI content as short as 75 words and can help you do quick verification if you need it.

Conclusion

Despite AI-generated reviews being illegal, they continue to slip through the cracks and give customers a potentially unrealistic overview of a product. Not even Amazon itself is able to catch all of these reviews!

Businesses have a responsibility to ensure customers are receiving the truth about products, not the opinion of Chat GPT, so more must be done to prevent them from being published on e-commerce websites.