Every estate agency technology vendor is now an AI company. The announcements come weekly. AI valuation tools. AI enquiry handlers. AI lead qualification. AI everything.
Most of it is built on the same foundation: a large language model, a prompt, and an output. The vendor wraps it in a product interface, calls it intelligence, and sells it as a decision-making tool.
I am not writing this to dismiss that foundation. Large language models are genuinely capable. They can read unstructured text, extract context, identify patterns, and produce summaries that would take a human significantly longer to generate. We use one ourselves.
But there is a specific failure mode in AI systems that the property technology market has not yet named honestly, and that failure mode matters enormously when the system's output is being used to make commercial decisions about which sellers to prioritise.
The failure mode is this: these systems can be wrong with complete confidence.
What hallucination means in a property context
Hallucination is the term used when a language model produces output that is plausible-sounding but factually incorrect. It is not a bug in the conventional sense. It is a structural characteristic of how large language models work.
Language models are trained to predict what comes next in a sequence of text. They are extraordinarily good at this. They produce fluent, coherent, contextually appropriate output at a rate and scale that no human writer can match.
But fluency is not accuracy. A model that predicts plausible next tokens does not know whether those tokens represent true statements about the world. It knows what words tend to follow other words. It does not know what is actually the case.
In a property context, this produces a specific and practical problem. An AI system asked to assess a seller enquiry might determine that the enquiry represents high instruction intent and express that determination with confidence, when the underlying reasoning is based on language patterns that superficially resemble high-intent enquiries rather than on the structural signals that actually indicate instruction likelihood. The output looks right. The confidence is real. The reasoning that produced it is not visible. And the agent who acts on it has no way of knowing whether the assessment is sound.
Fluency is not accuracy. A system that sounds certain is not the same as a system that is correct.
The black box problem
A black box system is one that produces outputs without making its reasoning visible. You put something in. You get something out. What happened in between is not accessible to the person using the output.
Most AI systems in property technology are black boxes. The enquiry goes in. A classification or a score or a recommendation comes out. The reasoning that produced that output is either absent or expressed in language so general as to be uninformative.
This creates a practical problem for anyone using the output to make a commercial decision. When an agent calls a seller because the AI flagged them as high priority, what happens when the seller turns out to be a casual enquirer with no intention of moving in the near term? The agent has no way to evaluate the system's error. They cannot identify what was misread, or whether the same error is likely to recur on similar enquiries. The black box offers no surface for learning or correction.
When an agent skips a seller because the AI classified them as low priority, and that seller subsequently instructs a competitor, the error is structurally invisible. There is no output to examine. There is only an instruction that did not arrive.
Black box errors in intake classification are silent. They do not announce themselves. They accumulate invisibly in the gap between instructions won and instructions possible.
Where confidence breaks down
The conditions under which a model produces high-confidence wrong answers are not exotic. They arise regularly in production systems.
The most common is distribution shift: the model encounters an input that differs in some meaningful way from what it was trained on. The model has no mechanism to recognise that it is outside its area of competence. It produces an output at whatever confidence the input pattern suggests. The output can be completely wrong. The confidence can be very high.
A related failure mode is overconfidence on ambiguous inputs. Where a human expert would pause and acknowledge uncertainty, a model often does not. It was trained to produce outputs, and it produces them. Ambiguity in the input does not reliably translate to reduced confidence in the output.
There is also the problem of confident interpolation. Models trained on large datasets develop a kind of fluency that can look like understanding. They produce outputs that are grammatically correct, contextually plausible, and delivered with apparent certainty. The plausibility of the output and its accuracy are different things. The confidence score does not distinguish between them.
A model has no mechanism to recognise when it is outside its area of competence. It produces outputs regardless. The confidence can remain high.
The value of expressed uncertainty
A system that can express genuine uncertainty is more useful than one that cannot, even if its headline accuracy is lower.
Expressed uncertainty gives the user actionable information. When a system indicates it is not confident in a particular output, the user knows to apply more scrutiny, seek additional information, or make the decision themselves rather than deferring. That is valuable. It makes the collaboration between a person and a system more effective, because the person knows when the system is reliable and when it is not.
A system that is always confident gives the user nothing to work with. Every output looks equally authoritative. The user has no signal for when to defer and when to question. They are left either trusting everything, which is dangerous, or trusting nothing, which defeats the purpose.
Building systems that express uncertainty honestly requires treating uncertainty as a first-class output rather than a failure state. That is a design choice, and it is one that most systems do not make, because expressing uncertainty can look like a weakness to people evaluating the product. In practice it is the opposite. A system that knows what it does not know is more trustworthy than one that claims to always know.
A different way to build
The architecture decision that addresses this is not to stop using AI. It is to be precise about what AI is being asked to do.
AI is genuinely good at reading unstructured text and identifying what is present in it. Given a seller enquiry, a capable language model can surface context, identify stated conditions, and recognise patterns across varied and informal language in ways that would take a human significantly longer to work through. That is a real capability and it is worth using.
The problem arises when the same process that reads the enquiry is also the one that makes the final recommendation. Reading and deciding are different tasks. They draw on different capabilities, they fail in different ways, and the thing that makes AI good at one does not make it well suited to the other. A system that combines both into a single step cannot cleanly separate them when something goes wrong.
The question worth asking of any AI system that makes commercial recommendations is not whether it uses AI. It is what the AI is being asked to do, and whether the parts of the process that require consistency, stability, and auditable behaviour over time are being handled by something designed to provide those properties.
When that question is answered well, the output is something an agent can not just act on but evaluate. The classification is consistent. The reasoning is visible. The errors, when they occur, can be identified and corrected rather than absorbed invisibly into the queue.
What to ask any AI vendor
Before trusting any AI system with a commercial intake decision, these are the questions worth asking.
Can you show me the reasoning behind each output? Not a summary. Not a confidence score. The specific basis for the classification and what evidence it rested on. If the answer is no, the system is a black box.
How does the system handle contradictory signals? A seller who writes tentatively but has a hard deadline and a prior agent visit is not a low-intent enquiry. Does the system resolve that contradiction correctly? Can it show you how?
What happens when the model is uncertain? Does the system express uncertainty differently from confidence? If a seller enquiry contains minimal signal, does the output reflect that ambiguity or does it produce a confident classification regardless?
Where does the classification logic live? If the answer is inside the AI model, then the logic is not auditable, not version-controlled, and not yours to inspect or correct. If the answer is in a defined process that the AI informs but does not control, then the system has separated reading from deciding in the way that matters.
Can the system be wrong with confidence? Any honest answer to this question is yes. The follow-up is: how do you know when it is? If there is no mechanism for identifying confident errors, the system's reliability cannot be evaluated, only assumed.
These are not hostile questions. They are the questions that any agency making a commercial decision based on AI output should be able to answer. The vendors who welcome them are the ones worth talking to. The ones who deflect them are telling you something important about what is inside the box.