A permission check is a snapshot. This post follows three bugs where the snapshot outlived the permission, and shows how to fix them and how to rate them.
This post is for developers and security engineers who build features that copy, cache or queue data: search indexes, AI answers, exports, background jobs. You do not need to have read the companion post, Reviewing a Google Drive Integration, though it sets up the product we use as an example: a multi-tenant SaaS app that syncs a customer's Google Drive and offers AI search over it. We take three bugs in turn. They look unrelated, and they share one cause. After the third, we look at how to decide how serious a bug like this really is.
When code asks "may this user read this document?", the answer is true at that moment. Permissions change: someone is removed from a folder, an employee leaves, a share is revoked. Any copy of the data, any saved result and any queued job keeps working from the answer it had earlier.
Security people call the general problem TOCTOU, short for "time of check, time of use". You check a condition, then act on it later, and the condition changed in between. In this post the condition is always "is this user allowed?". The related term authorization drift means the permissions your system believes in slowly differ from the real ones.
Our three examples differ in where the old answer is stored:
The platform copies Google Drive documents every six hours and indexes them for AI search. Consider this timeline:
The engineering manager says: "The next sync is at 3:00 PM and will update the permissions. A few hours is fine, because we are not reading from Google Drive anymore." The root cause is that the platform holds a copy of the document and a copy of who may read it, and the copy of the permissions is out of date. The boundary that matters is between Google's current authorization state and the one the SaaS cached. "We synced it earlier" does not justify access now, because the permission Alice had at 9:00 AM was a decision about 9:00 AM.
It is tempting to say "just re-sync more often" or "re-index the document". Look at what actually changed: the content of the document did not, but its access list did.
| Document content | Document permissions | |
|---|---|---|
| What it is | What the document says | Who may read it |
| Changed at 10:00 AM? | No | Yes, Alice was removed |
| Fixed by re-indexing the content? | Not needed | No. Re-indexing does not change who may see it |
The document also still exists and other people may still have access to it. So the answer is not "delete it from the index". The platform has to update who may retrieve the content, which is a separate piece of data from the content itself.
Use three separate mechanisms:
changes.list) and push notifications (changes.watch). Be careful with what notifications are. Google's documentation says they carry no content or metadata, only a signal that something changed, so you must call the API to find out what. Notification channels also expire (up to one week for changes) and must be recreated, so run a regular reconciliation too, in case a notification was missed.Keep three timestamps in mind: when the content was last synced, when permissions were last verified, and whether any change is pending. They are different facts.
The platform connects with one Google account's credentials. To reason about other users' access, it needs the document's full access list, and Google does not always provide it. In the Drive API, a file's permissions list is only available if the requesting user can share the file, and it is not populated for items in shared drives. So a read-only integration account may not be able to see who else has access.
Mapping is another problem: Drive permissions are written in terms of Google accounts, groups and domains, while the SaaS has its own users. Matching them reliably means linking each SaaS user to a Google identity. This means "check Google permissions at retrieval" is a design with real work behind it. Options include connecting with each user's own account (so a check uses their own access), using an administrator-level integration that can read permissions, and indexing only documents whose permissions the integration can see. Whichever you choose, decide what happens when the answer is unknown. The safe default is to hide the document.
| Approach | Good | Limit |
|---|---|---|
| Sync every 15 minutes | Simple | Permissions can be about 15 minutes stale, plus processing delay |
| Drive change notifications | Much faster detection | Not an instant guarantee. Signals only, with expiry, so reconciliation is still needed |
| Check Google permissions on every search | Freshest answer | Adds latency, uses API quota and makes search depend on Google being up |
| Local permission copy plus change feed | Fast search that scales | You must keep the copy consistent and reconcile it |
| Fail closed when freshness is unknown | No access on uncertain data | Search may be briefly unavailable |
My usual choice is a local permission copy kept current with the change feed, enforced at retrieval time, with stricter freshness rules for sensitive documents. Whatever you pick, write down the maximum acceptable revocation delay and what happens when Google is unreachable. "How fast must a revoke take effect?" is a product question, and the technical design should answer it with a number.
Is it a release blocker? Not just because the sync interval is six hours. The questions that decide it: how confidential are the documents, can one user see content another user is no longer allowed to see, how many customers are affected, how long is the exposure, did customers expect revocation to propagate promptly, and what other controls exist. For an enterprise AI search product over confidential files, launching without retrieval-time authorization and a defined revocation mechanism is something I would push hard against.
The team now enforces authorization before sending chunks to the model. Then someone adds a cache to save money:
def answer_question(user, question):
cache_key = hash(question)
cached_answer = redis.get(cache_key)
if cached_answer:
return cached_answer
chunks = vector_db.search(question)
authorized_chunks = [c for c in chunks if can_access(user, c.document_id)]
answer = llm.generate(question, authorized_chunks)
redis.setex(cache_key, 3600, answer)
return answerThe developer says: "We check authorization before generating answers, and the cache only lasts an hour, so this is secure." Look at the cache key: hash(question). It contains nothing about who asked.
The authorization check only runs on a cache miss. On a hit, the saved answer is returned directly. An optimization silently removed the control. Notice that this needs no permission change at all: two users asking the same question is enough. Answers are not like public web pages. An answer built from Tenant A's confidential documents is only fit for people who can read those documents.
Putting tenant_id in the key stops cross-tenant leaks, but two users in the same tenant can still have different document access. A manager's answer about salaries should not be served to someone in a different team. Here are the options:
| Cache key | Result |
|---|---|
| Question only | Unsafe for any answer that depends on permissions |
| Tenant and question | Fixes cross-tenant leaks. Still unsafe when people in one tenant see different documents |
| User, tenant and question | Good isolation, but a cached answer can still outlive a revoked permission |
| Permission context plus source document versions | Allows safe reuse, but needs reliable invalidation |
| No caching for sensitive answers | Simplest model, costs more per request |
For a first safe version I would cache per user and tenant with a short lifetime, and not share entries between users until the design is tested. Store, with each cached answer, the list of source documents (and their versions) it was built from. On every cache hit, check that the user can still read all of those documents, and drop the entry when any permission or document changes. Checking only one document is not enough, because an answer can combine several.
A note on retrieval order. Fetching candidate chunks before checking authorization is not a leak by itself. What matters is whether unauthorized content ever reaches the model or the user. A sound design either searches only within an already-authorized set or filters the candidates before any content is used.
Tests: Alice asks twice (reuse is fine). Bob from another tenant asks the same question (must not get Alice's answer). Bob from the same tenant without access asks (must not get restricted content). Revoke Alice's access after caching (the cached answer must not bypass it). Change a source document (the old answer must be invalidated or versioned). Take the cache offline (the system must fall back safely, not skip checks).
Users can export confidential documents. Exports can be huge, so they run in the background:
@app.post("/api/exports")
def create_export(request):
user = validate_jwt(request.headers["Authorization"])
document_ids = request.json["document_ids"]
for document_id in document_ids:
if not can_access(user, document_id):
raise Forbidden()
job_id = queue.publish({"user_id": user.id, "document_ids": document_ids})
return {"job_id": job_id}
def process_export(job):
documents = database.get_documents(job["document_ids"])
archive = create_zip(documents)
url = upload_archive_to_s3(archive)
notify_user(job["user_id"], url)The developer says: "We verify authorization for every document before queueing, and the worker only processes approved IDs, so it is secure." Now picture Alice requesting an export of 100 confidential documents at 10:00 AM. At 10:01 her access to 20 of them is revoked. At 10:05 the worker runs.
The root cause is the gap between authorization at request time and authorization at execution time. When the API checks Alice, the decision is about Alice. When the worker runs, it uses the worker's own broad database access, which says nothing about whether Alice may still read these documents. The trust boundary that changes is the one between the API and the worker.
A natural improvement is to check access right before each file is added to the ZIP. That shrinks the window, but does not close it:
The check is still a snapshot. So before designing a fix, decide which promise the product makes. Does revocation need to be respected at the time of the request, when each file is read, when the export is complete, or when the user downloads it? These are four different policies, and the engineering team has to choose one on purpose.
Even a final check has a gap before the user actually downloads. If revocation must apply to every new download, check authorization again at download time. For very strict requirements the revoke and the download need to coordinate through locking or other transactional means. And once a file has been downloaded, nothing can recall it. Which of these you need is a risk decision, not a technical default.
The worker uploads the ZIP to S3 and tells the user. Questions to ask about that last step:
job_id or object key the client sends.Also connect the lifecycle: when an employee is deprovisioned, cancel their pending jobs and invalidate unredeemed export links.
Suppose you report the export problem and the engineering manager disagrees:
"A user can only request an export for documents they are already allowed to see. The attack needs someone to revoke access while the job is running, which is an unlikely race. The ZIP is in a private S3 bucket. This is not high severity."
Your job is to turn that argument into facts. You investigate and find:
The "unlikely race" argument assumes a window of milliseconds. Here the window is up to 30 minutes in the queue, followed by a 24-hour link. And the trigger is not an unusual coincidence: it is a normal business event. Someone who is leaving requests an export, and their access is removed as part of offboarding. No special skill is needed. That moves it from "a race we hope never happens" to "a scenario that matches how people actually leave companies".
The private bucket does not change this, because the link is the key. A presigned URL grants access to whoever holds it, regardless of the bucket's privacy.
It is tempting to answer "critical" at once. A more careful position is that the evidence so far shows a potentially high-impact authorization flaw: the worker lacks a second check, and the delivery link outlives permission changes. It does not yet prove every step needed for a revoked person to actually obtain the data. For example, a user who legitimately downloaded files before revocation is not exploiting this flaw. Whether the person can still get the link after losing access depends on how the link is delivered: to an external email address, through an app session that ends at offboarding, or with the job cancelled during deprovisioning.
So I would call it potentially High, and raise it to Critical once reproduced if it reliably exposes regulated or highly sensitive data to a former user. The evidence to gather, in a staging environment with synthetic documents:
If all five steps work, you have strong evidence of a practical post-revocation disclosure.
| Factor | Question to answer | Here |
|---|---|---|
| Exploitability | Who can do it, with what prerequisites, in what window? | Any user with export rights, up to 30 minutes plus a 24 hour link |
| Impact | How sensitive is the data, and how much? | Financial and legal documents, personal data, thousands of files |
| Scope | One user or many customers? | Any customer using exports |
| Realism | Does it match a normal event? | Yes, offboarding |
| Detection | Would you notice it? | Likely not, since it looks like a normal export |
| Compensating controls | What else limits it? | Not established yet. Check. |
| Fix cost | How hard is a real fix? | Moderate: re-check, versioning, controlled download |
Disabling exports is a valid answer, but a good reviewer offers narrower choices as well, so product leaders can pick the least disruptive safe one:
| Mitigation | What it reduces | Cost |
|---|---|---|
| Disable exports temporarily | Everything, if every export path is covered | The feature is gone |
| Cancel queued jobs when access is revoked or the user is deprovisioned | Many stale jobs | Needs reliable revocation events |
| Re-check at job start and again before delivery | Stale authorization at both ends | Engineering work |
| Stop emailing direct presigned URLs. Use an authenticated, short-lived download | The 24 hour window | Some infrastructure and UX work |
| Exclude the most sensitive document classes from export | Worst-case impact | Reduced export features |
For this data I would require a mitigation that prevents the problem before launch, not only monitoring after the fact. If no narrow control can be built and verified in time, disabling the export is justified.
How I would put it to product leadership: "The export workflow relies on an authorization check done up to 30 minutes before the data is packaged, and the delivery link is valid for 24 hours. A user who is losing access could request an export while allowed, then keep the result after being removed. I am treating this as potentially high severity and will confirm by reproducing the full flow. Because the documents are financial and legal, I need a preventive control before launch. The best fix is a re-check at execution and delivery, cancelling stale jobs, and authenticated downloads. If we cannot do that safely in time, I recommend turning exports off until we can."