The AI layer nobody had security-tested

The team has reviewed its cloud account. The model provider has published security material. Authentication works, and the application passed a conventional test before the AI feature was added.

The remaining risk sits between those checks.

An AI feature can retrieve private material, mix instructions with user input, call tools, write to another system, or expose details through logs and error messages. A user does not need to break the model provider to cause harm. They need to find one path where the application trusts the wrong instruction or gives the model more access than the user should have.

That path only becomes visible when the whole application is tested as a system.

We start with the real boundary. Which users can reach the feature? What data can enter its context? Which systems can it search, and which actions can it take? Where do service accounts, retrieval stores, model providers, and application logs sit in the flow?

The map does not need to be elaborate. It needs to show where trust changes and where one identity may inherit the permissions of another.

From there, we agree the rules of engagement. The client authorises the test boundary, environments, accounts, test data, stop conditions, and escalation contacts before active testing starts.

Prompt injection and jailbreak tests matter, but a list of clever prompts is not an assessment. The useful question is what an attack can reach in this product.

Testing follows plausible paths through the application. Can untrusted content change the system's instructions? Can one user retrieve another user's material? Can a tool call cross an approval boundary? Do outputs reach downstream systems without checks? Could logs, traces, or provider settings retain data the team did not intend to keep?

We also review the controls around the feature. Identity, API access, secrets, network paths, storage, monitoring, and deployment choices can either contain an AI failure or make it worse.

Each finding needs evidence that an engineering team can reproduce. A theoretical concern without a path, impact, and fix is difficult to prioritise.

The output is not a long list sorted by generic severity. Findings are ranked against the application's data, users, actions, and operating environment.

Some fixes change permissions. Others add input boundaries, output validation, approval steps, retrieval filters, safer logging, or better monitoring. A few risks may remain by design, but the team should know who accepted them and what signal would trigger another decision.

The engagement ends with a remediation plan, a readout for the people who own the product, and clear retest criteria. If the team wants help implementing the fixes, Estma can work beside engineering and verify the result.