AI agents no longer just answer questions. They browse the web, call APIs, write code, and access enterprise data, often with minimal human intervention. That shift from passive tool to autonomous actor is one of the most exciting developments in the field, and it opens a class of security risk that existing frameworks were never designed to handle.
Mahesh Kumar Goyal is a senior data and AI engineer at Google, where he specializes in advanced agentic AI systems, predictive analytics, and large-scale data strategy. He leads cross-functional initiatives to build responsible AI architectures for some of the hardest problems in the industry. In this conversation he explains why the security systems built over the last forty years fall short for autonomous agents, and what enterprises should put in place before deploying them.
Mahesh Kumar Goyal is speaking independently, and the views he shares are his own.
This is the full conversation behind an issue of Offensive Engineering, the newsletter from InfoSec Relations written for engineers. Hosted by Samarpita Pattnaik, a volunteer host at InfoSec Relations. Connect with Mahesh Kumar Goyal on LinkedIn.
You can read the issue, Securing Agentic AI Against Data Leakage, drawn from this conversation.
S Pattnaik and S Eben J
The transcript below has been lightly edited for clarity.
Before we start, share a couple of lines about yourself.
Goyal: My name is Mahesh Kumar Goyal. I've been in the data space for almost fifteen years. I've been working with a series of customers to solve their data problems. In recent times, I'm working with a lot of large customers to solve their agentic solutions, and at the same time, to identify what could be a potential use case to solve with the agents and the frameworks that we are seeing in the modern times.
How significant is the threat of data leakage in autonomous AI agents compared to traditional application security risks?
Goyal: If you see the current market space, or how the technology is moving, it's much bigger. The traditional apps we've been building over the last thirty, forty years are predictable and stateless. But agents are not predictable sometimes. Depending on how the LLM reacts, how it interacts with the tools, it can follow various paths. At the same time, agents have memory. They take real actions, and they move data across multiple systems. And most of the time, they don't have a human in the loop. So with the systems where we knew how the data was going to move, it was much easier to predict how we wanted to secure it. But with these agentic systems, it's much more difficult, because it's very complex. It interacts with various systems, and all of these technologies built in the last few years are more advanced, and we haven't looked back into how we're going to secure them. So the security systems are weak. At the same time, these agents are very strong at interacting with any of the systems that we've been building for the last forty years.
What makes agentic AI systems uniquely difficult to secure from a data governance perspective?
Goyal: As I already mentioned, these agents are non-deterministic. We don't know whether it's going to be accessing a tool, and many times, since there is no human in the loop, we don't know what kind of access it has. Usually the access management we've seen in the past would start with the user, and then system-to-system interactions would happen. But with these agents, since a lot of companies have their separate data strategy, their separate modernization strategy, and a different AI strategy, this AI strategy altogether does not follow any of the data governance or the other practices we've built across data. So there's a blur. These agents usually have blurred data and control planes. The document they read can become instruction. They also read a lot of log files, a lot of trace information, personal information. There's a blurred line between what is code and what is data for an agent. So the systems we had in the past, which allow role-based access control, column-based access control, least access privilege, cryptographic boundaries, are very difficult to apply on these agentic AI systems. We definitely need an altogether different set of systems, which would be built for agents, not for data. Because the data, it just sits in a place, and then we use it for various intelligence purposes. Based on the personas and the applications, we decide how we want to grant the different sets of access, how we want to define the governance boundaries, how we want to make sure there is quality. But with agents, since it's pretty much non-deterministic, we do have to design or think about how we want to make it more governed. All the systems we've seen so far for sensitive data protection, we may have to come up with a different approach of flow for agents. The LLMs can access tools, the tools will access sub-agents, then from there it goes to an external API. So it can have a different path. That's where we have to come up with a system that has a human in the loop, and at the same time makes some kind of deterministic approach for all the agents to follow in a specific set of governance.
Are current enterprise security architectures prepared for autonomous AI workflows?
Goyal: I would say mostly not, because all the systems that have been built are for standard applications used across organizations. For example, EDR tools are built to detect anomalies in the systems, or your web traffic, or in human behaviors. But an agent that runs code perfectly maybe a hundred thousand times can make a mistake on the next occurrence. That's where identifying a specific anomaly in the agent is very tricky compared to identifying the anomaly in traditional systems. And all the systems like IAM, RBAC, DLP assume there's a human or a static service behind all of these operations. But with agents, these are dynamic. There are sub-agents, there are MCP tools. And there's a lot of context and memory within the agent itself. It runs over a period of time, and the way it builds the memory is quite difficult to identify, like where the problem would be coming from. So the security systems the current enterprise uses, I would say they're not prepared for autonomous AI agents or workflows. There are some initiatives or some stacks that are pretty modern, and they follow some of the practices we were discussing, like agent-to-agent protocol, and some of the protocols built for MCP are standard. Another that I've seen is very popular is Model Armor, which prevents any prompt injection, it looks for SQL injection. So those are the solutions getting more popular, but at the same time, wider enterprise adoption is going to take some time.
One threat vector that keeps coming up is prompt injection. I'd love your take on this.
Goyal: SQL injection has been one of the foundational problems we've had for a very long time. With these AI agents, even though the agents have long-term memory, one malicious prompt can poison the entire memory or the entire history the agent has been following. It can be a smaller output of a sub-agent, which the agent was not expecting. So that can completely derail how the agent was making progress. SQL injection had a clean pattern. We knew what kind of SQL would come. We had a lot of semantics to identify the SQL. The SQL may have a WHERE clause, or a JOIN condition that could result in an injection. But with LLMs, with prompts, it's much more difficult, because it's in human natural language text that may be coming from the user. Even though from the word-by-word perspective there are no vulnerabilities we'll be able to find, from the context perspective, it can have an all different meaning that can create a problem for how the AI workflows would run. So eventually the SQL injection would become a prompt injection, and prompt injection would become a bigger problem. We definitely need a series of tools or technologies to make sure there is a filter in the prompt, or there's an architectural control. For example, we have sandboxing control at the provenance to make sure we test all of the prompts that go to the production systems, and there is a limitation or a boundary where it verifies that this prompt has been done in the past. So we need a different set of enforcement, or an agent gateway, to make sure we can prevent the systems from prompt injections.
How should organizations approach context isolation and memory protection in AI agents?
Goyal: That's a good question, and it's one of the problems right now. If you see across different organizations, they're just treating the memory of an agent like short-term memory. But one solution I'm seeing in the market right now is to treat agent memory like a database. So we have to make sure that this data we have as part of the agent is encrypted. We encrypt it, we scope it down, like who has read access on this, who has write access on it, and then we have a way to version control it. At the same time, we have to make sure that one user session does not contaminate other user sessions. So that would be to make sure there's isolation between the different sets of users that run across the different company. Another option I see very frequently is we need to make this memory ephemeral. If it stays throughout the lifetime of an agent, then if there is any way that the data was, or the prompt was, injected, it cleans out after a period of time. And having the scoped context window, having a summarization of your context to make sure the summary is not poisoned, or the summary still aligns with how the user started this agent, that would definitely play a major role as we move forward with these agentic workflows.
What are the biggest misconceptions enterprises currently have about securing LLM-based agents?
Goyal: One of the common challenges we've seen is that companies think the guardrails or the monitoring systems or any of the tooling they're using for observability is good enough for agents as well. They might be good just to monitor the infrastructure of the agents, like where agents have been deployed, whether they have good source control, whether it's the right DevOps pipeline. But if you talk about the task or the flow within agents, they're not able to secure that with these existing systems. The organizations think an agent is just a smarter script, but that's not reality. Most of the time, the agent can uncover certain aspects from your dataset or your entire code base, which a human was not able to do in the last few years. The agent can do it in a few minutes. So it's much smarter than what we think as a script. Apart from that, we always start building the technologies or tools first. If you use any agentic workflow, we start deploying it, start making it production ready, but we think about security at a later point of time. So that's another misconception, that they think they can make the agent secure at a later point of time. All of these misconceptions the industry has right now could really be a chain, or a way, that attackers can find to corrupt or inject, to go for the prompt injection on the agents.
What governance controls should be mandatory before deploying autonomous AI agents in production environments?
Goyal: If you see some of the recent announcements from Google, what we've done is we came up with a centralized agent inventory. That's one of the core principles right now. All organizations should follow that. They should have a central repository where they manage all of the agentic systems, like their MCPs, their RAG, all the agents they run across the organization. There should be a centralized repository for it, and they should have the right set of owners, purpose, data classification, security practices, and what kind of tools they can access. Their scope should be narrowed down. At the same time, we should ensure we understand the blast radius for any agent exploitation. If an agent gets exploited specific to an organization department, does it impact the other side of the world or not? So there should always be an estimation of the blast radius for any agent that could get exploited in the future. And for governance, one recommended practice would be to have their own cryptographic identities. Make sure they have their own set of key pairs. If you want to encrypt any prompt that's been triggered on the agent, it has its own set of key pairs. It's not shared with the other agent, because that's where there could be a chance that this key can get exposed. And the similar principles we had in the past, like least privileged tool access, network segmentation, and making sure the most critical task within the agent goes to human-in-the-loop approval. So if that is always in place, we can make sure the governance for any of these autonomous AI agents can run freely in production.
Since we're talking about exposures and exploitations, should AI agents operate under zero trust principles? Are existing IAM and RBAC models sufficient, or do we need agent-specific permission architectures now?
Goyal: Zero trust only makes sense when we make sure all of the agent's processes or actions are authenticated, authorized, and verified, regardless of where they're coming from. So even if you start a prompt with an agent, we need to make sure this prompt does not have any injections, and this aligns with what the agent is built for. If there's a persona-based agent, we need to verify that this prompt makes sense for what the agent is built for. It's not going to provide an outcome or answer that is different from the core purpose of the agent. At the same time, as we narrow down these agents based on their cryptographic identities, we build short-lived tokens, which can run for a certain period of time. For example, if I want to do an MCP authentication or access data from an API or a database, we know that this query would run for a certain number of minutes only, not more than that. So we can have short-lived tokens or mTLS authentication to make sure there's a decoupled authentication in place. That's really helpful, rather than having hard-coded API keys, or a config file that has all the information or API keys plugged in with the agent. That's one of the core challenges we've seen, especially with large companies, that they accidentally leak all their code base, because they had all the API keys as part of either a log file or a test case. And if you want to follow zero trust for any of the agent-to-agent calls, we need to make sure that even for agent to agent, there is authentication in between, so it only gets the output that it is looking for. The downstream agent has its own memory context, and if that memory context is poisoned, it may impact the actual agent that is accessing the data from the sub-agent.
You've spoken about the guardrails and the governance principles we need. My last question has two parts. What security capability will be foundational for enterprise AI agents over the next three to five years, and what advice would you give to organizations deploying agentic AI and handling sensitive data today?
Goyal: I think we already discussed the challenges we're seeing as an industry and some of the solutions popping up right now. Not all of these are very popular, but overall, these are getting more attention. At the same time, a lot of big companies like Google Cloud are talking about how we need to secure the agents. I was watching a video sometime back from Cloud, which talks about how we need to prompt, and I realized that all this time I've been prompting it wrong. So the large companies are providing us a way to make the sensitive data we have within our organization more secure with agent deployment. First of all, I'll start with making sure we have an inventory for all of these agents or tools, configurations, or any systems we built for AI, and governance in terms of who has access to the right set of tools, which agents can access which sets of tools. At the same time, all of these agents have their own identities, so they secure all the communications they do within the agent, and outside of the agent to tools. To make sure that we understand the blast radius, if the system compromises, what gets impacted. So that's another assessment the industry has to do. Apart from that, have unique agent identities, make sure there's least access or privilege given to the tool or to the users. There's runtime monitoring, we need to make sure sensitive data protection is always running, so the agent is not getting access to sensitive data. And even if it is getting access, it's not exposing that sensitive data outside of it. And human in the loop is always very critical, no matter how big or complex the agent goes, or if it is solving the most complex problem a human used to do in the past. It has to have a certain level of human in the loop to make sure it does not go beyond what we are looking for. If the agent has been built for a specific task, is it able to do that? Sometimes we are able to run a finance assessment with the large agents. If the agent is missing one digit out of a finance calculation, it can make ten million into a hundred million. So for those aspects where there's a high volume of risk involved, the human is always required to validate all the information. So these are some of the things you see in the market that have been in the talk. A lot of big companies and small companies always talk about how they want to provide a solution or tooling to make sure the agents are not biased, the systems are verified, you have your golden dataset to verify all of this before you actually go and deploy on production.