Baxnet Ideas · Founder note
Your AI Chatbot Hasn’t Signed the NDA
TL;DR: A company chatbot can expose a real contradiction and invent a plausible confession in the same voice. The security boundary has to sit outside the model.
A company builds an AI chatbot to answer questions about its products. Then somebody asks it questions about the company.
That is the beautifully awkward premise behind a recent More Perfect Union video. The reporter uses Amazon’s shopping chatbot while investigating marketplace products promoted as American-made. The bot appears unusually cooperative: it points out contradictions in product information and then offers a theory about why Amazon might tolerate them.
It looks as if Amazon has accidentally supplied the investigation with its own witness.
The fuller story is even more useful. The video was based on research published by Columbia’s Center for Law and Economic Studies. Its authors inspected listings, varied their prompts and preserved screenshots of the exchanges. In one test, Amazon’s chatbot refused a request using the phrase “Made in USA”, then produced results when the researchers changed the wording to “in USA”. In another, it identified a mismatch between a product’s American-made presentation and information describing it as imported.
Those are observations worth investigating. The chatbot’s explanation of Amazon’s motives is something different. It was a model-generated theory, not an internal memo.
The corporate chatbot may be an unusually cooperative witness. It is also a witness capable of inventing the crime.
The useful part is not the confession
There is already independent reason to take the marketplace issue seriously. In July 2025, the US Federal Trade Commission wrote to Amazon and Walmart about third-party listings that appeared to make deceptive US-origin claims. The problem did not begin with a chatbot answer.
But the answer cannot tell us, by tone alone, where it came from.
The model may have retrieved information the user was entitled to see. It may have connected public facts that the company had never placed together on one approved page. It may have reached material that should have been restricted. It may have inferred a plausible explanation from the wording of the question. Or it may simply have produced a convincing story.
All five can sound equally fluent.
That makes chatbot output a useful lead and a poor verdict. A revealing answer should send us towards product records, policy documents, repeatable tests and people who can establish what actually happened. The chatbot can show where to look. It cannot turn its own confidence into provenance.
This is the part of the whistleblower metaphor that breaks. A human whistleblower can understand what they witnessed, choose to disclose it and provide primary evidence. A language model has no first-hand experience and no moral decision to make. It will not worry about losing its job. It also will not recognise that a neat explanation has crossed from evidence into speculation unless the surrounding system makes that boundary visible.
An NDA is not an access control
Companies will naturally try to teach their chatbots what not to discuss. There will be system prompts, refusal rules, fine-tuning and lists of prohibited subjects. These are useful layers. They are not a confidentiality system.
The Columbia researchers did not perform some spectacular Hollywood hack. They changed the wording of a shopping request. The different result does not by itself prove that a secret guardrail existed, but it demonstrates how brittle language-level behaviour can be. A rule that holds for one phrase and disappears for a near-equivalent phrase is not a dependable security boundary.
NIST defines prompt injection as an attack that exploits the mixing of untrusted input with instructions created by a more trusted party. Not every persistent question is prompt injection. The broader lesson is that user language is hostile territory: it should not be able to grant access that the surrounding software did not grant.
AWS makes much the same point in its own security principles for agentic AI. Enforcement should sit outside the model’s reasoning loop, with deterministic controls, narrowly scoped access and audit records the agent cannot rewrite.
This produces a less theatrical security question. Do not ask whether the chatbot can be persuaded to betray the company. Ask why it could reach the sensitive information in the first place.
Imagine an internal HR assistant allowed to read the staff handbook and a salary spreadsheet. The instruction says it should answer questions only about leave. Somebody asks whether two colleagues are paid the same. If the model has to remember that salaries are off-limits, the architecture has already failed. It should never receive those rows for that request.
Confidentiality cannot be a personality trait we hope the model retains.
The risk changes when the chatbot can act
An indiscreet chatbot can reveal something. An overpowered agent can also do something.
That distinction is becoming practical. Amazon says its shopping assistant—Rufus, now called Alexa for Shopping—can use account context, add items to a cart, reorder products and, in some cases, buy an item automatically. These may be useful features. They also show why permission design cannot stop at read access.
An agent needs separate boundaries for what it can retrieve, what it can disclose, and what it can change. Being allowed to read an address does not imply permission to reveal it. Being allowed to recommend a purchase does not imply permission to complete one. A prompt should not be capable of turning one permission into the next.
The same applies inside a business. Access to a customer record should not silently include permission to email the customer, issue a refund or edit the record that will later be used to audit the decision. High-consequence actions need their own approval path, not a more emphatic sentence in the system prompt.
Personal intelligence needs smaller rooms
The stakes become more intimate when the data belongs to one person rather than a company.
A personal intelligence system may eventually hold messages, calendar entries, health records, financial history and years of private documents. Giving one conversational agent unrestricted access to that archive and instructing it to be discreet would repeat the corporate mistake at the scale of a life.
The safer model is a series of smaller rooms. Access should be limited by source, person, period, purpose and action. A relationship adviser might receive a narrowly scoped report without inheriting the underlying message archive. A health agent might use one set of records without learning what was said in a family chat. An assistant permitted to prepare a draft should still need separate approval to send it.
This is where Personal Chat Intelligence, and Mimoto specifically, should be useful: producing bounded, source-linked evidence that a person can choose to share, rather than placing an open-ended conversational layer over everything they have ever said. I wrote recently about why personal agents need permission boundaries. The chatbot-as-witness problem is the same architectural lesson viewed from the other side. A model should not be responsible for deciding whether it was allowed to see the evidence it is now describing.
There is something darkly funny about the corporate chatbot that starts answering questions its owner would prefer it not to. It has not signed an NDA. It has no career to protect, no loyalty to test and no instinct for an awkward silence.
But the durable lesson is not that AI models are natural whistleblowers. It is that secrecy cannot depend on their discretion.
The useful security question is not whether the chatbot is loyal. It is why we gave it something we needed it to be loyal about.