We Taught Our AI to Say 'I Don't Know'

TL;DR

We made the controversial decision to build "I don't know" as a core feature -- when our system lacks sufficient evidence, it says so rather than generating a confident-sounding guess. During a high-stakes demo, this honesty won over a senior analyst who called it "the first AI tool that's been honest with me." In high-stakes domains, the willingness to withhold builds the credibility that makes confident answers actually trustworthy.

The demo was going poorly, or so I thought.

We were presenting KnowYourCompany.ai to a research team at a large buy-side firm. Six analysts in the room. Their head of research on the Zoom. A trial decision that would affect whether we landed our biggest account to date.

One of the senior analysts was running the system through its paces. She'd been asking increasingly specific questions, the kind of layered, multi-source queries that only someone who's been covering a sector for a decade would think to ask.

Then she asked about the pricing outlook for a mid-cap healthcare company. Specifically, she wanted the system to assess whether recent regulatory pricing changes, combined with the company's hospital mix shift and a new competitor entering the market, supported or undermined the bull case on gross margin expansion.

It was exactly the kind of question our system is designed to handle. Except for one problem: we had limited alternative data coverage on this particular company. We had earnings transcripts, exchange filings, and sell-side estimates. But we didn't have the supplier pricing data, channel check intelligence, or hospital-level reimbursement detail that a reliable answer would require.

So the system did what we'd designed it to do.

It said: "Insufficient context to form a reliable view. Available data covers earnings commentary and filings, but lacks supplier pricing data and channel checks needed for a defensible pricing outlook. Confidence level: low."

My stomach dropped.

What Happened Next

The analyst looked at the screen for what felt like thirty seconds but was probably five. Then she turned to her colleagues and said something I'll never forget:

"This is the first AI tool that's been honest with me."

She went on to explain that she'd tested two other AI research platforms that same week. Both had answered the same question confidently. One had produced a three-paragraph analysis concluding that the margin outlook was "constructive." The other had generated a bullish summary with four supporting data points and a chart.

Neither had flagged that the supporting evidence was incomplete. Neither had noted the absence of supplier-level data. Neither had said anything resembling "I'm not sure."

And here's what struck me: when she'd followed up on those confident answers by doing her own research, she found that the picture was considerably more complicated than either tool had suggested. The regulatory pricing changes had an uneven impact across hospital categories. The new competitor's pricing strategy was still unclear. The supplier data, which neither tool had access to, suggested potential cost headwinds that contradicted the bullish narrative.

The confident answers weren't just wrong in degree. They were wrong in direction. And if she'd trusted them without verification, she would have reinforced a position that the evidence didn't fully support.

"I can work with 'I don't know,'" she told us. "I can't work with confidently wrong."

The Internal Debate

I wish I could say we always planned to build "I don't know" as a feature. The truth is messier.

The decision came out of a debate that nearly split our product team. It started about six months before that demo, when we were designing the system's response architecture: the rules governing what the system does when it receives a query.

The default approach, and the one that most AI products use, is straightforward: always generate a response. If the model has any relevant information, synthesize it into an answer. If the user asks a follow-up, generate another answer. The goal is responsiveness. The metric is coverage, the percentage of queries that return a substantive result.

Half our team advocated for this approach. Their argument was practical and, honestly, compelling:

Users expect answers. When you type a question into a system and it comes back with "I can't help you," that feels like a product failure. It feels broken. And in a competitive market where analysts are evaluating multiple tools side by side, the tool that answers the question wins the demo over the tool that doesn't.

They were right about the expectation. They were right about the demo risk. And if we'd been building a consumer product, a search engine, a writing assistant, a general-purpose chatbot, I think they would have been right about the decision.

But we're not building a consumer product. We're building research infrastructure for people who allocate capital based on the information they receive.

The Cost of a Confident Wrong Answer

The other half of the team, led, I'll admit, by me, made a different argument. It wasn't about what users expect. It was about what happens when the expectation is met with unreliable information.

In equity research, a confident wrong answer has a specific and measurable cost.

Consider the chain of events: An analyst asks an AI tool about a company's margin outlook. The tool returns a bullish assessment with supporting data points. The analyst, working under time pressure and managing a coverage universe of 15-20 companies, incorporates this assessment into their morning note. The portfolio manager reads the note. Capital gets allocated based on the thesis.

If the assessment was wrong, if the tool missed a conflicting data point, used stale information, or generated a plausible-sounding narrative that wasn't supported by the evidence, the damage isn't an inconvenience. It's a financial loss. It's an erosion of the analyst's credibility with their PM. In the worst case, it's a pattern that, repeated enough times, ends careers.

I talked to one analyst during our user research who described it this way: "I'd rather my tool says nothing than says something I have to spend 45 minutes fact-checking. Because if I can't trust it, I'm not saving time. I'm adding a step."

This echoed something we'd heard repeatedly in earlier research. The 45-minute trust gap, the time analysts spend verifying AI-generated outputs, is often longer than the time they would have spent doing the research manually. The AI creates a draft answer, and the analyst then has to validate every component of that draft. If the draft is wrong, they've wasted the time it took to generate it and the time it took to discover it was wrong.

An "I don't know" doesn't create that chain. It tells the analyst: this question requires more evidence than the system currently has. You should investigate directly, consult additional sources, or wait for more data. The analyst can make an informed decision about how to proceed, instead of unknowingly building on a shaky foundation.

How We Actually Built It

Saying "I don't know" sounds simple. Building a system that knows when to say it is anything but.

The core challenge is what we call the confidence assessment problem: for any given query, how does the system determine whether it has enough evidence to generate a reliable response?

We approached this through three mechanisms, each connected to the four-layer stack architecture I've written about before:

Source sufficiency scoring. Before generating a response, the system evaluates whether it has access to the data sources that the query requires. A question about margin trajectory needs earnings data, segment breakdowns, and ideally peer comparisons. A question about pricing outlook needs supplier data, channel intelligence, and hospital-level reimbursement detail. If key source categories are missing, the system flags the gap and quantifies how much the absence affects confidence.

This isn't a simple checklist. Different questions require different evidence bases. The system learns, over time, which data sources are necessary for which types of analytical conclusions. A question about revenue growth has different evidence requirements than a question about competitive positioning or management credibility.

Temporal validation. The system checks whether its available data is current enough to support the query. If an analyst asks about Q3 margin trajectory and the most recent available data is from Q1, the system flags the temporal gap. This sounds obvious, but it's a failure mode that plagues most AI tools: they retrieve relevant-looking content without verifying that it's relevant in time.

Conflict detection. If the available evidence contains contradictory signals, say, management commentary suggesting margin expansion while supplier data suggests cost headwinds, the system doesn't silently resolve the conflict. It surfaces it. "Available evidence is mixed: management commentary supports margin expansion, but supplier pricing data suggests potential cost headwinds. Insufficient evidence to determine which signal is more reliable."

When any of these mechanisms flags an issue that drops confidence below a threshold, the system returns an explicit statement about what it doesn't know and why, rather than generating the best-guess answer it could produce with incomplete evidence.

What the Data Showed Us

After we shipped this feature, we ran a study with our early user cohort. We tracked every instance where the system said "I don't know" and followed up with the analysts to understand what they did next.

The results surprised even me.

In roughly 60% of cases, the analyst told us the "I don't know" response was more useful than a confident answer would have been, because it directed them toward the specific evidence gap they needed to fill. Instead of receiving a plausible-sounding answer and spending time verifying it, they immediately knew where to focus their own research.

In about 25% of cases, the analyst told us the response helped them avoid a potential error. They had been leaning toward a conclusion that the system's confidence assessment revealed was inadequately supported. The "I don't know" wasn't a dead end. It was a guardrail.

In the remaining 15%, the analysts wished the system had attempted an answer even with incomplete data. This is real, and we're working on it. We're building a "low confidence" response mode that generates an answer but explicitly labels it as preliminary and flags the specific evidence gaps. The analyst gets a starting point, but with clear warnings about what's missing.

The net result: our early users report significantly higher trust in the system's outputs when it does provide a confident answer, precisely because they know it's capable of saying "I don't know." The willingness to withhold builds credibility for the times it delivers.

One analyst put it perfectly: "When your system tells me something, I actually believe it. Because I've seen it tell me when it doesn't know. That's how trust works."

The Broader Lesson

I think about this decision a lot, not just as a product choice, but as a principle for building AI in high-stakes domains.

The AI industry has an incentive problem. The tools that demo best are the ones that always have an answer. The tools that win POCs are often the ones with the highest query coverage. The tools that generate the most impressive-looking outputs are the ones with the fewest guardrails.

But the tools that analysts actually rely on, the ones that become embedded in their workflow rather than relegated to the "nice to have" category, are the ones they trust. And trust isn't built by being comprehensive. It's built by being honest.

This extends well beyond equity research. Any domain where decisions have significant consequences, healthcare, legal, engineering, policy, faces the same tension. Speed and comprehensiveness are easy to demo. Trustworthiness only reveals itself over time.

We made a bet that trustworthiness would win in the long run. So far, the data supports that bet.

But I recognize it's a hard sell in an industry that's been conditioned to expect AI that always has the answer. The hardest thing about building "I don't know" as a feature isn't the engineering. It's convincing people, including our own team, that honesty is a competitive advantage.

In high-stakes domains, it is.

Other posts in the Founder Stories series: