Your Members Ask the AI About You Now. It's Getting the Answer Wrong.

A member wants to know whether your annual conference counts toward their recertification. Two years ago they searched Google, clicked your site, and read the page you wrote. This year a growing share of them never reach your site at all. They type the question into ChatGPT, or they read the AI answer sitting above Google's results, or they ask Perplexity, and they act on whatever comes back. The association was not in the room for that conversation. In a lot of cases, the answer was wrong.

This is not a forecast. OpenAI reported ChatGPT passing 800 million weekly users at the end of 2025, and it crossed a billion monthly users by the middle of 2026.¹ Consumer surveys now put roughly a third of searches starting inside an AI tool rather than a search box, and one large study found more than three quarters of people have used ChatGPT to look something up.² Your members are ordinary internet users. They are doing this about you, whether or not anyone on staff has looked.

Here is the part that should worry whoever owns member communications. When researchers at Columbia's Tow Center tested eight AI search tools on straightforward sourcing questions, the engines answered incorrectly on more than 60 percent of them, and they did it with total confidence.³ Perplexity was the most accurate and still missed better than a third of the time. ChatGPT's search was wrong on about two thirds. The failure mode is not that the AI says it does not know. It is that it produces a clean, plausible, specific answer, and often attributes it to a source that says no such thing.⁴ Now apply that to "how many credits does this session carry" or "what is the deadline to renew without a lapse," and you can see the shape of the problem.

Associations are unusually exposed here. Your facts are precise and consequential (eligibility rules, credit hours, dues deadlines, member categories) and they change on a schedule. They are also scattered: a value that lives in a 2023 PDF, a paragraph on an old microsite, a forum thread where a member guessed once, and a current page you updated last month. An answer engine does not know which of those is authoritative. It blends them, or it grabs the one that reads most cleanly, which is frequently not yours. The parallel question, whether the agents crawling the web can even read your data on your terms, I wrote about earlier this summer. This is its twin. One is about access. This one is about accuracy, and it is already live in front of members.

Run the test your members already ran

You do not need a project to find out where you stand. You need an afternoon and the list of questions your member-services inbox actually answers most. Take the top ten. Ask each one, phrased the way a member would phrase it, to ChatGPT, to the AI answer above Google's results, and to Perplexity. For each answer, write down three things.

The ten-question audit: what to record, and what counts as a miss

What you record	What a pass looks like	What counts as a miss
The answer itself	Factually correct and current	Any fact that is wrong or out of date
Whether it named a source	Cites a specific page	No source named at all
Whose page it cited	Your current, authoritative page	A forum, an old PDF, a directory, a competitor

The scoring is the whole point, so make it concrete. A right answer that cites your current page is a pass. Anything else is a finding, and each kind tells you something different. A wrong fact means the engine is reading a stale version of you that still exists somewhere. No source named means your authoritative page is not clearly the answer to that question, so the model fell back on training-data memory. A third-party source (a member forum, an old event listing, a vendor's directory) means someone else has become the internet's answer about your own rules. If a rep at a booth answered members the way the AI just did, you would retrain them by Friday.

The reason to run this today, rather than after the next platform decision, is that the fix is almost never architectural. You are not buying a warehouse or standing up a governed layer to correct a wrong dues deadline. You are finding the stale PDF and unpublishing it, or writing the one clear page that states the fact plainly and dates it, so that the cleanest and most current version of the answer is finally yours. That is a content and hygiene job an editor can do, not a migration. It is the same instinct behind getting the boring plumbing right that kept the renewal email out of spam: the channel changed under you, and the correction is unglamorous and cheap. The engines will keep changing. A member told the wrong renewal date will keep costing you the renewal. Close the gap between what you say and what the AI says about you, one answer at a time.

Quick takes

The answer is now the destination, not a stop on the way. The AI summary above Google's results, and the chat reply inside ChatGPT, are increasingly where the question ends. The member reads it and moves on without a click. That is convenient for them and invisible to you, which is exactly why a periodic read of what those surfaces say about you has to become someone's actual job, not a thing you notice when a member complains.

Structure helps the machine find your good page. When your canonical answer lives on a clean, well-labeled page with a plain question as the heading and a dated answer beneath it, the engines have a far easier time picking it over the forum thread. You do not need a vendor for this. You need the fact to exist once, clearly, on a page you control, which is the same discipline that separates the 7 percent of nonprofits getting real value from AI from everyone else who skipped the foundation.

Monitoring tools exist. Do not wait for one. A market of "AI visibility" dashboards has appeared to track how the engines describe your brand. Some are useful at scale. None of them are a prerequisite for the first pass, and buying one before you have read ten answers by hand is how a hygiene problem gets turned into a procurement cycle. Do the manual read first. Let what it finds tell you whether you need anything more.

Worth a read

Columbia Journalism Review, on the Tow Center study of eight AI search engines. The primary source for the accuracy numbers, and worth reading for the texture of how the errors happen, not just the headline rate.

Orbit Media's AI-search adoption survey. A practitioner-friendly look at who has actually switched to asking AI, for what kinds of questions, and where they still reach for a search box.

Nieman Lab's summary of the same research. The short version if you want to hand a skeptical colleague one link that explains why "the AI said so" is not a reliable answer about anyone.

The open question is not whether members will keep asking the machine instead of you. They will. It is whether, the next time one of them asks, the cleanest answer on the internet about your own rules is the one you wrote.

---

From the Mind of Ravi Rooprai is a weekly column on association tech, data, and AI. Read the perspectives for the longer arguments behind it.

Researched with AI assistance and fact-checked against primary sources. The analysis, judgment, and writing are mine. How this column is made →