Executive summary
2
separate ways that data leaks, each needing a different fix
Key conclusions
- Source-system permissions do not travel with the data. A model can repeat anything it can read to anyone who asks.
- Retrieval leaks are fixed by filtering each query against the user's actual access rights. Fine-tuning leaks are prevented by excluding restricted content before training.
- Test the system as an attacker would: log in as a low-privilege user and ask for something only a high-privilege user should see.
A model does not check a user's job title before it answers. It can repeat anything it can read. The controls that kept the source systems safe were part of those systems, and the documents do not carry them.
Solving the first problem creates this second one. Plan for it at design time, because both of its causes are difficult to detect afterwards.
Data leaks through two routes, and each needs a different fix
- Retrieval returns a document the user should not see. The fix is to filter each query against the user's actual access rights.
- Fine-tuning makes the model memorise the restricted fact itself. The content is then inside the model's weights, where no retrieval filter can catch it. The fix is to exclude that content before training, because afterwards there is nothing to filter.
Apply access control to each query before it reaches the model
The lasting solution is to treat access as a property of each request instead of each document. For every request, the system checks who is asking and what they are entitled to see. It then confirms that the retrieval respects those rights before anything reaches the model.
Then test the system as an attacker would, before a real attacker does. Log in as a low-privilege user and ask for something only a high-privilege user should have. The test takes an afternoon, yet most acceptance criteria do not include it.
