Earlier this year, a merchant taking personal payments asked our AI support assistant how to quickly accept a payment with his card reader. It checked our documentation and confidently replied, “tap ‘Simple Payment.’”
But there was no Easy Payment button. We had renamed it to “Add Custom Amount” months earlier, but the documentation had not been updated. The trader couldn’t find what the AI was describing and eventually gave up.
Support reported the issue, we updated the quick start guide and product page, and the documentation has been resolved. But this incident stuck with me because the assistant did exactly what he was supposed to do. It read the documentation and shared what it found. The problem was that the documentation described a product that had already been changed.
Here’s the key point to remember: when you ask an AI how to do something in WooCommerce (like changing a setting, finding a feature, or fixing an error), it answers based on the documentation, not based on real-time knowledge of the product. The product changes with each release, but the documents are written by humans. Any gap between them can lead to a wrong answer that sounds completely confident.
So we created something to bridge that gap.
It is a WordPress plugin, built as an internal Automattic project, and it works as a documentation writer that is always on the lookout for changes.
It keeps track of WooCommerce releases and all incoming requests to fix documentation. Whenever something changes, such as a renamed button, a moved setting, or a new feature, an AI model prepares an update. The draft must then meet certain standards before it is published:
- We score from 1 to 5 using our quality rubric, which looks at accuracy, voice and tone, structure, grammar and style, discoverability and imagery.
- It must also pass automated security checks. If a draft removes more than about 30% of a page, breaks the formatting of the page, leaves a placeholder, or adds a new image link, it will be flagged for a human to review.
- A writer from our team reviews the draft in a preview and approves it before it goes live on woocommerce.com.
My favorite part of the section is the discoverability part: the first paragraph of each document should work as a standalone answer that an AI can quote. We’re not just writing documentation anymore for people scrolling through a page. We create the source material for every AI answer about WooCommerce, and our column reflects that.
We don’t just score new concepts. We also score the live documentation with the same rubric.
At the end of June, 39 of the 117 WooCommerce documents we reviewed were below our quality standards. That’s why we built this system. WooCommerce is constantly being updated and even the best team of writers can’t re-read every page after every release. But the plugin can. It checks pages like a writer would with unlimited time, and highlights those that need attention.
Part of the pipeline automatically applies our terminology list to keep product names consistent, such as using ‘WooCommerce’ instead of ‘woocommerce’. It does this with an instant search-and-replace function.
Some entries in the list are written as writing advice rather than exact word changes. The system couldn’t tell the difference, so it pasted the advice into live pages. For a short time, a published document told readers “(descriptive link text describing the destination)” where a link should have been.
I noticed this while working on something related. We checked all 258 WooCommerce documents, found 9 that were affected, fixed them all, then removed the problematic entries and added a safeguard to prevent these types of errors from happening again.
I share this because it’s the same lesson from a different angle. Automation follows the rules exactly as they are written, quickly and without judgment. That makes it both useful and risky. Our system is built with this in mind: strict controls that fail safely, a human reviewing everything, and audits for the automated steps.
What this means for troubleshooting
If you ask an AI assistant a WooCommerce question and take the answer at face value – and most people do, because that’s the point – the quality of the answer was determined before you even typed your question. It depends on whether the documentation matches the product you are using.
It is our job to keep documentation accurate, and the system above is how we do that. There’s a smaller part for you, but it’s still important:
- If an AI’s instructions don’t match what you see on your screen, the documentation may be out of date. Before you think you’ve made a mistake, check the document it uses.
- Please let us know if you notice a problem. When using our AI assistant, you can rate an answer with a thumbs down or tap “Contact Support” to tell us what’s wrong. If you are using a document that does not appear to be current, please let our team know via ‘Was this guide helpful?’ thumb at the bottom of the page.
The merchant with the card reader did nothing wrong. Neither does the AI. The responsibility was ours, and building a system that makes improvements quickly – and catches changes before you ever notice them – was the best way we could learn from that experience.
David is an AI engineer at Automattic who specializes in shipping production LLM infrastructure for customer support and documentation systems on WordPress.com and WooCommerce.
