Baxnet Blog · technical-explainer
Privacy Choices Are Often Made in the Dark
TL;DR: Privacy choices become weaker when the person cannot see the data trade they are being asked to accept.
Privacy is still too often described as if it were a clean individual decision.
Read the policy. Review the toggle. Decide what you are comfortable sharing. Move on.
That story is tidy, but it does not describe how most data decisions actually happen. Usually the person does not know much about the real trade in front of them. They do not know what will be retained, what will be inferred, what will be linked with something else later, or how easy it will be to revoke the decision after the system has already learned from it.
That is one reason the old line “users chose this” so often feels dishonest.
In The Economics of Privacy, Alessandro Acquisti, Curtis Taylor, and Liad Wagman pull together years of economic and empirical privacy research and arrive at a point that still matters for product builders: consumers are often making privacy decisions under imperfect or asymmetric information. In plain English, one side of the transaction understands far more than the other. The company knows what it can collect, combine, predict, retain, and monetize. The person sees a prompt.
That gap matters even more when the system is asking for intimate context rather than shallow browsing data.
If a personal AI product wants access to messages, notes, calendars, location history, or relationship context, it is not enough to say that the user consented. Consent given in fog is a weak foundation. The product has to make the trade legible before it asks for trust. What stays local? What leaves the device? What is stored long term? What trains a model? What can be deleted cleanly? What can be inspected by the person in plain language rather than in policy prose?
This is where privacy stops being a legal checkbox and becomes product design.
A second paper helps sharpen the point. In the NBER working paper The Effect of Privacy Regulation on the Data Industry: Empirical Evidence from GDPR, Guy Aridor, Yeon-Koo Che, and Tobias Salz studied data from an online travel intermediary after the GDPR opt-in requirement took effect. Their result is the kind of finding that should make builders slow down before they treat regulation as the whole solution. The paper reports a 12.5% drop in intermediary-observed consumers, but it also finds that the remaining observed users became trackable for longer, increasing their value to advertisers.
The lesson is not that privacy regulation failed. It is narrower, and maybe more useful: rules can change market behavior without automatically producing a healthier relationship between the person and the system. The surface gets cleaner. Some tracking drops. But the underlying incentive is still to learn as much as possible from the users who remain observable.
That is exactly why Baxnet keeps returning to architecture and boundaries instead of treating privacy as a messaging problem.
For the kind of product world Mimoto is aiming at, trust has to be built into the mechanics. A person should not need an economics degree to understand the bargain. They should be able to tell where their data lives, what context is active, what memory is being used, and how to step back without wrecking the product’s usefulness. If the system remembers something sensitive, the person should know that it remembers. If the system is drawing on a private archive, the person should be able to inspect the source material behind the answer. If a category of data is off limits, that boundary should be real in the product, not just suggested in a footer link.
I think this is where a lot of personal AI still undershoots the problem. The industry likes to ask whether users are willing to share more data in exchange for more personalization. But that framing skips the harder question: under what conditions would a person actually understand the exchange well enough for the choice to mean anything?
Better prompts are not enough. Better architecture matters. Better visibility matters. Better defaults matter. And, importantly, less appetite for collecting everything matters.
That last point is easy to miss. Product teams often assume that trust is earned by adding reassurance around a broad data grab. Research like this suggests the opposite direction can be stronger. Narrow the scope. Explain the data path. Make the memory boundary inspectable. Reduce the amount that has to be taken on faith.
Privacy choices made in the dark are not really choices. They are wagers.
If personal AI is going to ask for the raw material of a person’s life, it should at least be designed so the person can see the table they are sitting at.
Further reading: Acquisti, Taylor, and Wagman, “The Economics of Privacy” (Journal of Economic Literature, 2016) and Aridor, Che, and Salz, “The Effect of Privacy Regulation on the Data Industry: Empirical Evidence from GDPR” (NBER Working Paper, 2020). The first source is a broad economics review rather than a study of AI assistants. The second studies an online travel intermediary and advertising value under GDPR, so the product conclusion here is an analogy about incentives and legible consent rather than direct evidence about personal AI tools.