As AI models evolve, a curious security paradox starts to emerge. 

The more AI advances, the more it can be used as a weapon at scale. At the same time, the more it advances, the more secure it becomes. 

More advanced models tend to be harder to break into directly. More mitigation layers. Better detection. Stronger guardrails. 

At the same time, these models become a multiplier on the other side. Greater persuasion, more automation, more scale. 

Attacks no longer need to break the model itself. It is enough to exploit the surrounding system: integrations, context, users, data. 

AI becomes more resilient internally while amplifying attacks externally. The risk does not disappear. It shifts. 

Where the risk lives today: three key areas 

So where do the biggest risks actually lie in LLM-based applications today? I would start with three: prompt injection, context poisoning, and stale data. 

1) Prompt injection 

Prompt injection is the modern version of malicious input, except now the input is natural language. 

OWASP already ranks prompt injection as a core risk in LLM applications (LLM01). 

The main concern is not making the model say something inappropriate, but making the system act incorrectly: 

• calling the wrong tool  

• retrieving data it should not access  

• leaking information  

• triggering sensitive processes  

This is not theoretical. There are already cases in enterprise copilots where a simple message or embedded content can lead to data exfiltration or bypass controls. A kind of zero-click prompt injection. 

2) Context poisoning 

If you use RAG, memory, internal wikis, or knowledge bases, you have created a context supply chain. And every supply chain can be attacked, either intentionally or accidentally. 

Context poisoning is simple in concept. The model does not need to be hacked. Someone only needs to influence the context it consumes: documents, notes, tickets, internal pages. 

The result is answers that sound correct but are based on flawed assumptions. 

3) Stale data 

Stale data is the least glamorous risk and probably the most common. 

Outdated data leads to confident but wrong decisions: outdated policies, pricing, contacts, SLAs, or processes. 

It does not look like an attack. It looks like a bug. 

Until it turns into a loss. An automated wrong decision. An authorization outside policy. A communication that is no longer valid. 

What changes in software engineering? 

Engineering has always dealt with uncertainty: networks, concurrency, distributed systems. 

LLMs introduce a different kind of uncertainty. 

That shifts the design mindset: 

• Before, we protected endpoints. Now we protect intentions.  

• Before, we validated input. Now we validate input, context, and action.  

• Before, failure was an exception. Now failure can be a coherent but completely wrong answer.  

Secure Software Development Lifecycle still matters, but it is not enough if you do not see AI as a sociotechnical system. Your application is not just a chat. It is an operator with access to data and tools. 

What is the scope of your agent? What happens in the worst-case scenario if it fails? 

The human impact: a trust crisis 

One of the side effects is a growing crisis of trust. 

Deepfakes in video calls, voice cloning, hyper-personalized phishing. Attacks are becoming more convincing, more technical, and faster. 

The Arup case, where criminals used deepfakes in a video call to trigger a multi-million dollar transfer, shows how “looking real” has become an attack vector, not proof of authenticity. 

Practical checklist (Definition of Done for AI systems) 

If I had to translate this into engineering criteria, I would start here. 

1. Scope, permissions, and action

• Classify the system: response-only, retrieval (RAG), or execution (agent)  

• Enforce true least privilege per tool: short-lived tokens, minimal scope, separated identities  

• Risk-based gating: sensitive actions require confirmation, dry runs, or human approval  

• Scope limits: how many emails, how much money, how many records per execution  

2. Isolation and safe execution

• Sandbox risky tools: restricted network, allowlisted egress, rate limits, cost quotas  

• Kill switch: ability to disable actions quickly and fall back to read-only mode  

3. Protection against promptinjectionand malicious context 

• Separate system instructions from untrusted content (documents, emails, pages)  

• Treat context as a supply chain: origin, authorship, versioning, audit trail, and write policies  

• Immutable rules: documents cannot override policies or request secrets  

4. Stale data and operational quality

• Define TTL per data type and display timestamp and source in critical responses  

• If data is outdated: alert, request confirmation, or refresh  

5. Observability, auditing, and adversarial testing

• Structured logs with redaction: RAG sources, decisions, tool calls, outputs  

• Anomaly detection: cost spikes, exfiltration attempts, injection patterns  

• Continuous red teaming with realistic abuse scenarios  

What comes next and what engineering needs to handle 

At least three movements are happening in parallel: 

• AI as critical infrastructure. Copilots and agents will become a standard product layer. Security becomes part of design, not an afterthought.  

• Context engineering as a firewall. Context will require governance, provenance, and validation, just like dependencies and packages today.  

• The verification era. We will rely more on verifiable signals: signatures, audit trails, explicit confirmations. Not out of paranoia, but because it has become too easy to fake things.  

Three questions before going to production 

This AI paradox is a reminder that no system is ever fully secure. 

What you can build is a system with safe failure, traceability, and clear boundaries, assuming a hostile environment. 

In my view, we will eventually see a major AI-related attack driven by excessive permissions, poor context governance, and uncontrolled automation. 

If you are putting AI into production, consider three questions: 

1. Have you truly enforced least privilege?  
2. Can you explain why the system acted the way it did by looking at the logs?  
   
3. Can you disable actions within minutes and fall back to read-only mode?  

If the answer is no to any of these, pay attention. The cost of this AI security paradox will show up. 

____________________________________________________________________________________ 

Rafael Dourado is the BR Operations Manager at Programmers. His main focus is ensuring end-to-end client satisfaction as organizations embrace this technological shift. With 15 years of experience in software and analytics, he is passionate about AI and its transformative potential. In his free time, Rafael enjoys playing chess with his daughter.