In the last post, I wrote about how well parallel development has been working for us. The next two sessions raised a different question: what should the agents helping us build the application be able to read, and what should an LLM inside the finished product be allowed to receive?
I’m combining security and the legal and ethical considerations here because, in our project, they meet at that question of access. There are our credentials to protect during development, and there is Engestofte’s customer and business information to protect if we connect the dashboard to a model.
We haven’t settled every part of this yet. These are the decisions we’re working through.
An environment file is still a readable file#
We currently keep credentials in environment files. The application needs them to connect to services, but the coding agent doesn’t need to know their actual values to implement a feature.
Sometimes the agents look in those files. That is the part that concerns me. If a tool reads an environment file and returns its contents to the agent, a credential can become part of the context sent to the model. An API key belongs in the application’s authentication process; it has no reason to appear in a development prompt.
The lecture, Is it secret? Is it safe?, made the point that the danger isn’t the AI company stealing keys. It’s ordinary mistakes: an agent includes the .env file in a debug summary, runs a command that prints environment variables, commits a secret, runs a malicious script from a package it installed, or writes a secret to a log. None of those require bad intent from anyone.
In the CMS repository, we have added a deny list to Claude Code’s settings file, .claude/settings.json. It blocks the agent from reading .env, .env.local and .env.test, and from running cat or type on them. Claude Code enforces those rules itself, so this is stronger than asking the agent nicely in a prompt.
It is still a narrow boundary. The rules block those exact commands, so head .env or a grep across the folder isn’t covered. The file only exists in the CMS repository, not in the API or Web repositories. And only Claude Code reads it. We also use Codex and Mistral’s Vibe in the project, and neither of them knows the file exists. We need to establish what our configuration actually prevents, rather than assume the secrets are protected because a rule exists.
The important distinction is between the application using a secret and the model seeing it. A server can authenticate a request with an API key without putting that key into the prompt. During development, though, a file read or diagnostic command can expose it unnecessarily.
Where our keys live, and what’s still missing#
We want a setup where the application can get the credentials it needs while the coding agent works with variable names and placeholders. That means looking at restrictions outside the prompt: file permissions, isolated development environments, and ways to provide secrets at runtime without exposing them to the agent’s tools.
A secrets manager is one option to investigate. OWASP’s secrets-management guidance discusses centralised storage and limiting access. But moving a value out of an environment file only helps if the agent cannot retrieve it through another route. The access boundary matters as much as the storage choice.
We also need to account for command output and logs. A tool that prints an environment variable can expose the same value as opening the file directly. The setup we want to verify is one where the app can run, the agent can help diagnose problems, and the secret itself never needs to appear in the conversation.
Some of the lecture’s advice we already follow. The .env files are in .gitignore in all three repositories, so they can’t be committed by accident, and the Web and CMS repositories have a .env.example that lists which variables exist without their values. The API repository is still missing one.
In production, the real values live on the server, and the secrets our CI needs are stored in GitHub Actions. No agent has access to the server. That is our strongest safeguard, and it matches the lecture’s advice: keep the keys on the server, and never invite the AI in there.
Development is where we’re only halfway there. The lecture’s advice is to use separate test keys for development, limit them, and replace them regularly. Our OpenAI key has a spending limit, so if it leaked, the cost would be capped. But we use the same key in development as in production, so a leak would still mean replacing the key the live website depends on. A separate development key that we can simply throw away is the next change I’d make.
For now, the deny list is our main safeguard on our own machines, and stronger enforcement is something we’re still researching. The lecture’s suggestion was to treat any machine an agent runs on as potentially compromised, and to decide what belongs on it from there.
Trying to make my own chatbot spill the beans#
During the session, we tried to get the chatbots on our portfolio sites to reveal a secret key or details about the system. I tried it with mine. It couldn’t give me the key, because it never has it. The Dify API key sits in a Cloudflare Worker that forwards the chat requests, so it never reaches the browser or the model’s context. The model sees the retrieved blog content and its system prompt, and I had already published that system prompt in the chatbot post.
That was the lecture’s point: an AI can only leak what it has access to. The way to limit that is to limit its tools. A function like searchPublicDocs(query) can only ever return public documents, while a function like readFile(path) can return anything on the disk. My chatbot only has the first kind. A backend with file access would be a different story.
Johan’s data is a separate decision#
The other question concerns the product we’re building. In the project plan, we described AI assistance for extracting inquiry details and drafting replies. Those features could save Johan time, but using them with real information would mean deciding what leaves Engestofte’s systems.
We want to talk to him before connecting real customer or sales data to an LLM in the dashboard. A useful feature doesn’t automatically justify sending the information behind it to an external provider.
Customer correspondence can include names, contact details, event plans, and personal requests. Sales information and offers also belong to the business. We need to explain what a proposed integration would send, why it needs those details, and what alternatives are available. We want that conversation before making the connection.
The one AI feature we already have shows what a clear boundary looks like. The CMS sends text to OpenAI to draft the English and German translations, and the only text it sends is content that’s going on the public website anyway: pages, holiday-house descriptions, navigation, the footer and image descriptions. No customer data or sales information goes through it.
A dashboard assistant could be valuable. Johan might ask what is missing from an upcoming event, or get help preparing a reply from the agreed details. But each task needs a defined amount of information. Drafting a response about a venue visit shouldn’t require access to the entire customer database or sales history. It’s the same principle as searchPublicDocs over readFile: give the assistant a tool for the one job, not general access to the database.
Human review remains part of the design. It helps catch an incorrect draft or a misunderstood inquiry. It doesn’t undo a disclosure: by the time someone reviews the answer, the input has already been sent to the provider.
Choosing a provider is part of the discussion#
Mistral is one provider we could investigate. We’d need to evaluate the actual service, deployment, and terms rather than treat a provider name as a privacy guarantee.
Mistral publishes a data-processing addendum and documents zero-data-retention options, with conditions on which services and requests are eligible. Those are things to examine if we choose that route. Retention, training use, and processing location are separate questions, and we’d need to understand each for the service we use.
A data-processing agreement would be part of arranging how a provider handles personal data. It would still leave us responsible for deciding which information the feature needs and limiting what we send. Johan agreeing that an assistant sounds useful would be the start of that discussion, rather than the whole decision.
Telling people when AI is involved#
The session on legal and ethical considerations added another angle: transparency. Under the EU AI Act, people should be told when they’re interacting with an AI system, unless it’s obvious. My portfolio chatbot introduces itself as “Morten’s AI Assistant” and says it can make mistakes, so nobody should mistake it for me.
In the Engestofte CMS, the AI only drafts the translations. A person reads and approves them before anything is published, so a human stays responsible for what appears on the website, in every language.
The application is intended to work without AI assistance. That gives us room to get these choices right. We can keep developing the event workflow while we research credential handling and discuss the data boundary with Johan.
I still see advantages in adding an LLM, but I want to be able to explain exactly what it can see and why. That applies both to the agent writing our code and to any assistant Johan might eventually use.
This is part nine of my posts on this semester’s AI-driven applications elective, combining the sessions on security and legal and ethical considerations.