Back to Blog
    14 min read30 views

    Why AI Isn’t Replacing Intelligence Analysts Any Time Soon

    Why AI Isn’t Replacing Intelligence Analysts Any Time Soon

    A digital representation of a human analyst examining complex data patterns alongside a translucent AI interface.

    Generated with Gemini Pro Image

    I’ve been watching the artificial intelligence space for well over a year. Ask anyone who knows me well, and they’ll tell you I’m an early adopter of most new technology. I am always looking for ways to experiment with emerging technologies to see how they may improve my life or work. So, for the last year or so, that’s what I have been doing. I have used ChatGPT, Gemini, Claude, and a few others - running prompts, stress-testing outputs, and trying hard to see what these tools are capable of. While looking for opportunities, I am also always on the lookout for places where cracks start to show.

    What I keep finding: When writing research and analysis papers (or even when researching this blog post), the most sophisticated models available right now (Sonnet 4.6, Opus 4.6, GPT-5.4, and Gemini 3.1 Pro) will confidently cite nonexistent sources, mix up details, conflate information, and outright make things up (hallucinations). Not occasionally. Routinely. And the longer a report gets, the worse it gets.

    This is not a bug. It’s not something that can be solved with a better model. It’s simply a limitation of how these systems work, and the research (and my own efforts) are proving it. Our job now is to learn how to work within the guardrails and educate the masses on what they are.

    Architecture Distinctions

    At their core, every AI tool you interact with, whether via a polished front end or directly through an API, runs on a large language model (LLM) (and occasionally some other magic). This is the foundation. But what sits on top of that foundation varies enormously from tool to tool and generation to generation. These differences matter when evaluating the reliability of an AI-powered system and how you use it.

    Standalone LLMs

    Standalone LLMs operate in a closed-world setting. These are models that most people operate locally today; there are hundreds of free ones available for experimentation on platforms like Hugging Face. A standalone LLM's knowledge is entirely encoded in its parameters during training. It is static and bounded by a training cutoff, with no direct access to external or real-time data. When you ask it a question, it generates the most statistically plausible response by predicting the next most likely token based on patterns in its training data. It is not retrieving facts. It is predicting text. When it produces a citation, it is generating a string of characters that looks like a citation. Whether that citation corresponds to anything real is, to the model, irrelevant. Telling the model you want it to be “truthful” or to provide “real” facts does not provide any meaningful improvement to the output.

    Retrieval-Augmented Generation (RAG)

    RAG models add an index step before output generation. In simple terms, the system searches an external knowledge base (e.g., a document corpus, a database, a vector store), retrieves the most relevant passages, and injects them into your prompt before the model generates a response. The model is now working from sourced material rather than pure training data. This reduces hallucinations on basic tasks. Researchers have concluded that RAG approaches achieve a roughly 42% reduction in hallucination rates compared to baseline LLMs in controlled evaluations.

    Web-Connected Hybrid Systems.

    These are tools such as Perplexity, ChatGPT with browsing enabled, and Claude with web search enabled. Rather than searching a static corpus, these systems query the live web, retrieve current pages, and ground responses in retrieved results with inline citations. They are closer architecturally to a fast research assistant than to a static LLM request.

    Web-Connected Tools Still Get Things Wrong

    Usually, when a new model or tool is released, the architecture upgrade is cool (let's never forget the time OpenAI let us change the color of our text bubbles). But none of these updates or upgrades will be sufficient to completely replace a human analyst.

    A March 2025 Tow Center for Digital Journalism study at Columbia University tested eight generative AI search tools across 200 prompts, assessing their ability to accurately retrieve and cite news content. The findings were damning. Chatbots were generally bad at declining to answer questions they couldn’t answer accurately, offering incorrect or speculative answers instead, and premium chatbots provided more confidently incorrect answers and citations than their free counterparts.

    The failure wasn’t outright fabrication. It was a citation mismatch: the URL looks real, and the source looks authoritative, but the model attributes to it misquote the content, strip essential context, or simply make things up. The source is real. The claim isn’t.

    I have found this consistently in my own research. When working on concepts for intelligence writing, I have found that AI-enabled research tools consistently misreference, omit, conflate, or merge details from the multiple sources they cite. This issue is consistent across Claude, ChatGPT, Gemini, and Perplexity, and worsens as the number of sources increases.

    OpenAI’s own researchers published a paper in 2025 stating that hallucinations persist because standard training and evaluation procedures reward guessing over acknowledging uncertainty. In other words, models learn to bluff because confident guesses score better on benchmarks than honest abstentions.

    MIT research published in January 2025 found that models were 34% more likely to use phrases like “definitely,” “certainly,” and “without doubt” when generating incorrect information. Put simply, the more wrong the AI is, the more certain it sounds. When you consider that words of estimative probability are a cornerstone to intelligence writing, this finding is concerning.

    Web connectivity improves grounding, but it does not eliminate an LLM's probabilistic nature. The model is still predicting likely text. When it retrieves a real page and synthesizes/fuses it with three others, it is still making probability-weighted decisions about what to say. Those decisions can, and frequently do, diverge from what the sources actually contain. At present, there is no practical or meaningful signal indicating when fusion went wrong, and solutions such as "model reasoning” and “chain of thought analysis” help reduce hallucinations but are not perfect (we will get to this in a moment).

    In a domain where one fabricated source or incorrect bit of information in an assessment can redirect resources, shape policy, or cost a human life, even a 37% error rate on citation tasks — Perplexity’s rate in the Tow Center study, the best performer tested — is not a deployment threshold. It must be a disqualifying statistic.

    Hallucinations are Here to Stay

    Perhaps most important. Hallucination error rates will never reach zero. Two independent mathematical proofs have established this as fact, not an engineering failure. Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli formalized this problem and proved that LLMs cannot learn all computable functions and will therefore inevitably hallucinate if used as general problem solvers. Because the formal world AI tools model is a subset of the real world, hallucinations are inevitable for real-world LLMs as well.

    Michał P. Karpowicz, approached the same problem from a different perspective, using mechanism design theory (reverse game theory) and the Green-Laffont theorem to prove that no LLM performing non-trivial knowledge aggregation can simultaneously achieve truthful knowledge representation, semantic information conservation, and complete revelation of relevant knowledge. The conclusion is the same: hallucinations arise from the mathematical structure of information aggregation and processing itself, not from any limitation that more training data or a better architecture will fix.

    Better models, better prompts, retrieval, and web access reduce the risk of hallucinations. But no serious generative AI system can promise zero. The practical goal can not be perfection. It must be to catch errors before they get published.

    Reasoning Models

    But what about “reasoning models” or “chain of thought analysis”? The sales pitch for reasoning models is pretty straightforward. Instead of generating text in one pass, most “modern models” work through problems step by step before producing a final answer. The model “thinks” out loud, checking its own work, revising, backing up when a path doesn’t hold. It looks, from the outside, like deliberation.

    The assumption that follows is logical. If you deliberate more, you make fewer errors. If the model is reasoning, surely it’s less likely to fabricate. The data, research, and practical application say the opposite.

    OpenAI’s own benchmarks showed that o1, its first reasoning model, hallucinated 16% of the time when asked to summarize publicly available information about people. Its successor, o3, hallucinated 33% of the time on the same task. Wildly twice the rate of its predecessor, despite being the “more capable model.” o4-mini came in at 48%. OpenAI acknowledges that “more research is needed” to understand why hallucinations worsen as reasoning models scale up.

    When a model hallucinates in step three of a twenty-step reasoning process, that falsehood becomes the basis for steps four through twenty. The model, confident in its reasoning, constructs complex logical structures on fabricated information. The more detailed the reasoning chain, the more convincing the incorrect answer seems.

    Additionally, when models generate multiple candidate solutions and choose among them, longer reasoning chains are often selected, but length does not reliably indicate correctness. Research shows that verification mechanisms can systematically favor verbose but incorrect reasoning over concise but correct reasoning, creating an incentive for the model to learn that appearing to reason thoroughly is more important than actually reasoning correctly.

    Researchers have named this “reasoning hallucination” - a more subtle form of hallucination embedded within logically coherent reasoning logs, which makes incorrect information more convincing and harder to identify. It resembles the conjunction fallacy, wherein detailed but deceptive explanations are viewed as more trustworthy precisely because of their detail.

    This is extremely important when considering how one interprets the output of reasoning models. A standard LLM that fabricates a citation gives a wrong answer. But a reasoning model that fabricates in step three has now produced an incorrect answer, wrapped in twenty steps of internally consistent logic explaining why the answer is correct. Now the output becomes harder to challenge, not easier. The analyst has to peel back the reasoning chain to find where it diverged from reality, which requires knowing the subject well enough to spot the mistake. Often, this means more time and effort spent trying to understand the AI and how it reached a conclusion than if the analyst had just done the work themselves.

    The RAG Ceiling

    The improvement RAG and web-connected tools offer comes with a structural limit that is particularly acute in intelligence work. Retrieval works when you know what to retrieve. You define the corpus, write the query, and the system finds the relevant document. If the document exists and you’ve indexed it, the model can work from it. This is undoubtedly useful, and I have helped build solutions that use this methodology for intelligence purposes. However, this is also a description of a known-knowns problem. One where the questions are already formed, and the relevant sources are already identified.

    Intelligence analysis is not the problem.

    Intelligence analysis is the work of answering questions you don’t fully know how to ask yet, about things that are deliberately obscured, using sources that may not yet be identified. The unknown unknowns. The patterns nobody has named, the connections nobody has thought to look for, are precisely what a retrieval system cannot surface. You cannot index data that does not exist. RAG solves for the synthesis of known documents. The analyst’s core task remains outside of this space.

    The Model Doesn’t Understand Your Question

    There’s an additional misconception worth naming, because it shapes how people and organizations misuse or misrepresent AI tools nearly every day. It is rooted in an unspoken belief that the tool or model “understands” what you’re asking. To squash this, you must understand that an AI doesn’t “understand” you. Not in any human sense.

    Regardless of the text you see on screen, when you submit a prompt, the tool doesn’t parse your intent, assess your context, or apply judgment to your actual need in the same ways you would expect a human to. A commercially available tool may provide user feedback indicating it is "reasoning your question...” but in reality, it is simply calculating what text is statistically most likely to follow the input you provided, though perhaps through a series of complex evaluations that seem to, in some sense, mirror logical thought. The key to remember is that everything presented is based on statistical likelihood, not true deterministic reasoning.

    These calculations will produce output that looks responsive, well-reasoned, and authoritative. It can also be completely wrong. Unfortunately, the model will not tell you which one happened. It cannot — because it has no model of your intent to compare against its output. It will be confidently wrong or right every time. This is why precise prompting isn’t just good practice. It’s the only mechanism by which you actually control what the tool does. An imprecise prompt is shaped by whatever patterns in the training data look similar, which may have nothing to do with your actual task. The prose will be confident and clean regardless.

    Where AI Earns Its Keep in Analytical Workflows

    Despite the doom-and-gloom narrative, AI tools are not useless to analysts. The question we have to ask is, "What is the intended use?" I use AI tools every day, quite happily, because I also understand how they work and their limitations, and this has tempered my expectations.

    Where I have found that AI tools consistently earn their keep is in the early stages of a research task. Pulling together background, surfacing open-source articles worth following up on, and reading deeper. Drafting initial summaries that a junior or mid-level analyst can use as a starting point rather than a blank page. This is real value and can save an analyst hours of effort. You can get intern-level throughput at machine speed.

    I have also found that AI is a great resource for enforcing analytical writing standards, voicing, and styling. Ensuring active voice. Using consistent terminology. Adherence to ICD 203 and ICD 206 format standards. These are tasks where the probabilistic nature of the output is irrelevant. You’re not asking the model to generate facts, you’re asking it to reshape prose against a fixed standard. I have found that, for the most part, the models are reliable in that regard. It won’t hallucinate a passive construction. That being said, human-in-the-loop verification is still necessary.

    The Distinction That Matters

    One category of analytical tasks requires factual accuracy under uncertainty. The other requires structural compliance, pattern recognition, or synthesis of identified sources within a closed corpus.

    AI is increasingly reliable in the second category. It remains unreliable with the first. Conflating the two is where people get into trouble. Reaching for a web-connected chatbot as a substitute for analytical judgment, or treating a RAG-grounded summary as a sourced assessment, is a recipe for disaster. Architecture improvements are real, emerging fast, and do matter for procurement decisions. But, they do not change the fundamental constraint: a probabilistic system operating on retrieved text is not the same as an analyst exercising judgment on incomplete information.

    The way I see it, the analyst isn’t going to be replaced - at least not anytime soon. Rather, the analyst is being repositioned as the error-catching layer that mathematics says will always be necessary. Someone still has to evaluate the sources. Someone still has to make the call. I think the same can probably be said of any profession that requires true critical thinking, research, and analysis.

    The vendor who tells you otherwise is trying to sell you something.

    Sources

    Share this post

    Enjoyed this?

    I publish this writing for free. If it was worth your time, a coffee helps me keep it that way.

    Buy me a coffee