Systems integration
The access control your unified data layer just lost
Connecting scattered data is step one. The moment that unified data feeds or trains a model, the permissions that used to separate finance from HR do not automatically come with it.
Executive summary
2
distinct mechanisms behind the leak, each needing a different fix
Core conclusions
- A model doesn't check job titles before answering — whatever it can read, it can repeat, and source-system permissions don't travel with the data automatically.
- Retrieval leaks are fixed by filtering at query time against the asker's actual entitlements; fine-tuning leaks require excluding restricted content before training.
- Test it the way an attacker would: ask the system, as a low-privilege user, for something only a high-privilege user should see.
A model does not check anyone's job title before it answers. Whatever it can read, it can repeat — and the controls that made the source systems safe lived in those systems, not in the documents themselves.
This is the second problem created by solving the first one. It is worth anticipating at design time, because both of the mechanisms behind it are difficult to detect afterwards.
Two mechanisms, two different fixes
- Retrieval serves a document the asker should not see. The fix is to filter at query time against the asker's actual entitlements, not to rely on what was indexed.
- Fine-tuning memorises the fact itself, so the restricted content is inside the weights and no retrieval filter can intercept it. The fix is to exclude it before training, because afterwards there is nothing to filter.
Move the control to the query
The durable pattern is to stop treating access as a property of the document and start treating it as a property of the request: who is asking, what are they entitled to, and does this retrieval respect that before anything reaches the model.
Then test it the way an attacker would, before one does. Ask the system, as a low-privilege user, for something only a high-privilege user should have. That test takes an afternoon and it is not in most acceptance criteria.
