← mohdarif.com · All notes

Cloud Security · Incident Response

Investigating and Containing Compromised AWS STS Credentials Across Accounts

A worked incident: trace a bulk S3 download back to a session using one account's logs, then stop it without taking down the services that share the role.

By Mohammad Arif · 14 min read

An alert says a large number of objects were downloaded from a sensitive S3 bucket. The activity used a role assumed from another AWS account, and the temporary credentials are still valid. This post covers two questions an incident commander will ask: what happened, and can we stop it right now without causing an outage? The answers depend on knowing exactly what AWS logs prove and what IAM changes actually affect.

1. The scenario

• Your company uses several AWS accounts under AWS Organizations.
• An application on an EC2 instance in Account A has an IAM role (AppRole) that is allowed to assume a role in Account B.
• The role in Account B, ProductionDataReader, can read a sensitive S3 bucket. Five other production services also use it.
• An alert shows a large download from the bucket. The team suspects the EC2 instance was compromised and that its credentials were used to assume the role.
• You can see CloudTrail and IAM in Account B only. You cannot see Account A.
• The attacker's temporary credentials have about 45 minutes left.

2. Evidence: what logs prove, suggest and leave unknown

Keep these three apart in everything you write down. Mixing them up is how incident reports turn into guesses.

3. Investigation, step by step

Start from what you know for sure (the suspicious S3 activity) and work backward to the identity. Do not start by searching for AssumeRole and hoping.

Step 0: Check that the evidence exists

CloudTrail records S3 object-level actions such as GetObject only if data events are enabled for the bucket. They are off by default and billed separately, and you select each resource type on its own (general purpose buckets and S3 directory buckets use different types). Without them you can see who assumed the role, but not which objects were read. Alternatives are S3 server access logs (if enabled) and GuardDuty S3 findings. Find out in the first minutes what you have, because it decides how well you can measure the damage.

Step 1: Read the S3 events

Look at the GetObject events for the bucket in the alert window and pull out the session identity:

-- Athena, table partitioned by region, year, month, day (adjust names to your table)
SELECT eventtime, sourceipaddress, useragent,
       useridentity.accesskeyid,
       json_extract_scalar(requestparameters, '$.key') AS object_key
FROM cloudtrail_logs
WHERE eventsource = 's3.amazonaws.com'
  AND eventname = 'GetObject'
  AND useridentity.sessioncontext.sessionissuer.arn LIKE '%/ProductionDataReader'
  AND region = 'us-east-1' AND year = '2026' AND month = '11' AND day = '04'
  AND eventtime > '2026-11-04T17:00:00Z'
ORDER BY eventtime;

The table that the CloudTrail console creates for you is not partitioned, so every query scans all the log files. For incident work, use a partitioned table (region, year, month and day, with the partitions loaded) or partition projection, and always filter on those columns. CloudTrail Lake is another option. Otherwise queries are slow and costly when you need answers fast.

Group the results by accessKeyId and sourceIPAddress. Five legitimate services and one attacker will usually form separate groups. The suspicious one is the key with the unusual IP, user agent, volume or object pattern.

Step 2: Link the key to its AssumeRole event

Now find the STS call that created that key. In Account B, the AssumeRole event includes:

If the key ID is missing from the S3 events, link on the session ID (principalId) and match creationDate to the AssumeRole event time. Do not link events by IP address or timestamp alone. Several sessions can share a NAT gateway, and many sessions can start in the same minute. The access key ID is the key.

Two things to know about where the STS events are. Calls to the global STS endpoint (sts.amazonaws.com) are logged in the us-east-1 region, while calls to a regional STS endpoint are logged in that region. If you do not see the AssumeRole event where you expect it, check us-east-1 and use a multi-region trail or CloudTrail Lake. Also, CloudTrail does not log denied cross-account AssumeRole requests in the target account, so failed attempts against your role will not appear in Account B.

Step 3: Work out who called, and what that does and does not prove

In Account B, the caller appears as the source account (Account A) and a principal ID. For a role session, the principal ID normally ends with the session name, and for EC2 instance profile credentials the session name is commonly the instance ID (I did not find this stated in AWS's documentation, so confirm it in your own logs). So it may point to the instance whose credentials were used. Check the real value in your logs, because the full caller ARN is only in Account A's own CloudTrail. Treat it as a strong lead, not a conclusion.

It is still not proof of who acted. Credentials can be copied off an instance and used from anywhere. Compare the sourceIPAddress in the assume call with the instance's known public or NAT address (from Account A):

• It matches: the activity probably came from or through that network. It could still be an attacker on the instance, or another workload behind the same NAT.
• It does not match: the credentials were likely taken from the instance and used elsewhere.

An IP address is a network location, not an identity. It can belong to a NAT gateway, proxy, VPN or other shared infrastructure.

Step 4: Scope everything that key did

Search all events for that access key ID, in all services and regions, not only S3:

• Other buckets, ListBucket calls, and other services the role can reach.
• Further AssumeRole calls (role chaining into other accounts).
• IAM changes or attempts to create persistence.
• The earliest activity from that caller or IP, to find when it started.

Write down the object list, count and bytes read. Legal and privacy teams will need the exact scope, and it is only as good as the logging you confirmed in Step 0.

Can Account B alone identify the instance or the attacker?

Partly. Account B can tell you the calling account, a principal ID that may include the session name (often the instance ID), the source IP and the tool. It cannot tell you what ran on the instance, who the attacker is, or how they got the credentials. For that you need Account A: its CloudTrail, GuardDuty findings, VPC flow logs, EC2 inventory and logs from the host itself.

4. Containment: stop it without an outage

The credentials still have 45 minutes left and five legitimate services use the same role. First, the facts that decide what works.

Changing the trust policy does not stop existing sessions

The trust policy controls who can assume the role. Tightening it blocks new assumptions only. Credentials already issued keep working until they expire. That is 1 hour by default, up to 12 hours if the role's maximum session duration allows, and role chaining caps a session at 1 hour. In this scenario an instance role is calling AssumeRole, which is role chaining, so the session in Account B lasts at most 1 hour. Tightening the trust policy is worth doing to stop the attacker getting fresh credentials, but it does nothing about the session they already hold.

Your options

Option 1: Deny the specific session

If the bad session can be told apart from the good ones, deny just that. For an assumed-role session, aws:userid has the form <role-id>:<session-name>, so a unique session name works. aws:SourceIdentity and session tags can work too, if you set them up beforehand.

{
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "StringLike": { "aws:userid": "AROAEXAMPLEROLEID:suspicious-session-name" }
  }
}

This only works if the attribute is genuinely unique. The caller chooses the session name, and it can change when roles are chained. If legitimate services use the same name (for example a default like app), a deny on it would cut them off as well. Check the real values in CloudTrail before relying on this.

Tip: the exact value to deny is already in your evidence. Copy userIdentity.principalId from the suspicious S3 event. It has the form AROA...:session-name, where AROA... is the role's unique ID, not its name. It also appears as assumedRoleUser.assumedRoleId in the AssumeRole event. You can look up the role ID with aws iam get-role --role-name ProductionDataReader --query Role.RoleId.

This does not stop the attacker from getting a new session. The caller chooses the session name (2 to 64 characters). If they still hold valid credentials for the calling role, they can assume the role again under a different name, and this deny will not match. Pair it with a change to the role's trust policy that blocks the compromised caller, and with isolating the host (section 5).

Option 2: Deny everything issued before a cutoff

If you cannot isolate the session, deny all sessions issued before a chosen time. The condition key aws:TokenIssueTime records when the temporary credentials were created.

{
  "Effect": "Deny",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::sensitive-data/*",
  "Condition": {
    "DateLessThan": { "aws:TokenIssueTime": "2026-11-04T18:00:00Z" }
  }
}

Walk through three sessions with a cutoff at 18:00 UTC:

Three limits you must state out loud:

• Scope. The example above denies only s3:GetObject on one bucket. The attacker's credentials can still call DynamoDB, other buckets or other APIs the role allows. During an active incident, widen it to "Action": "*" for old sessions, or review what else the role can do. The IAM console's "Revoke active sessions" button does the wide version: it attaches an inline deny-all policy named AWSRevokeOlderSessions to the role, with a cutoff about 30 seconds in the future to allow for propagation, and it applies to every user of the role.
• Collateral. Every legitimate session issued before the cutoff is blocked until it gets new credentials. Whether that is an outage depends on how each service refreshes.
• New sessions. The cutoff only covers sessions issued before it. If the attacker still holds the caller's credentials, they can assume the role again after the cutoff and receive a session that is not denied. Also add a deny for the compromised caller to the role's trust policy, and isolate the host. Also, the cutoff only affects temporary credentials, because aws:TokenIssueTime is absent from requests signed with long-term access keys. It will not stop an IAM user key the attacker may have created.

Use an explicit deny, not just a removal of permissions. If a bucket policy also allows the role, removing the role's own permissions does not stop access. An explicit deny wins over any allow.

The safe order for the legitimate services

1. List the workloads that use the role and how each gets its credentials.
2. Have them obtain fresh sessions after the cutoff: through their normal credential refresh, or a controlled restart where needed. People using the AWS CLI may need to clear its credential cache (~/.aws/cli/cache).
3. Verify in CloudTrail that new AssumeRole events appeared with creation times after the cutoff, and check application health.
4. Apply the deny. AWS says policy updates can take a few minutes to take effect, so do not assume it is instant.
5. Confirm the attacker's access key now gets AccessDenied in CloudTrail and the services are still healthy.

The fast, narrow option: protect the bucket

An explicit deny in the bucket policy protects the data at once and does not depend on the role. For example, deny GetObject except for approved roles and expected network paths, and keep an exception for incident responders so you do not lock yourself out. It buys time while you work out the precise session deny.

What about IP and network restrictions?

They can help as an extra layer, but do not rely on them:

• If the attacker works through the same compromised instance or the same NAT gateway as legitimate traffic, an IP allowlist lets them through.
• Production services may reach AWS through VPC endpoints, not fixed public IPs. Use aws:SourceVpce or aws:SourceVpc for those. aws:SourceIp does not apply to requests that come through a VPC endpoint.
• Adding legitimate IPs one at a time while watching for failures, during an incident, risks outages and mistakes. Establish the real network paths first, in advance if you can.

If you cannot tell the attacker from legitimate sessions

Then no precise deny exists. Stop the exfiltration first (bucket-level deny or the cutoff deny), accept a controlled disruption, and rebuild the legitimate services' access in the right order afterwards.

5. Contain the source in Account A

You cannot see Account A, but you can ask its owners to act:

• Isolate the instance. Swap its security group for one with no outbound access. Do not terminate it, because you need its disk and memory for forensics. Snapshot its volumes.
• Revoke the role's sessions in Account A only after isolating the host. EC2 rotates instance role credentials automatically and makes new ones available at least five minutes before the old ones expire. A revoke denies the credentials issued before its cutoff, but anyone still on the host receives a fresh, valid set at the next rotation. To change quickly what an instance can do, replace its instance profile. AWS advises against simply removing the role from the profile, because that change can take up to an hour to take effect.
• Find the way in. Common causes are a web flaw that reaches the instance metadata service (SSRF), remote code execution, or a leaked key. In Account A's CloudTrail, the ec2RoleDelivery field on the instance's calls is 1.0 when the credentials came from IMDSv1, which is consistent with metadata theft. Fix the cause.
• Rotate every secret that was on the instance or reachable from it, and check other instances that use the same role.
• Look for the GuardDuty findings UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS (instance credentials used from an IP outside AWS) and .InsideAWS (used from another AWS account). GuardDuty can stop raising these for continued activity from the same remote source, so a quiet GuardDuty does not mean the activity stopped.

6. Make the next incident easier

• Turn on S3 data events (or server access logs) for sensitive buckets before you need them.
• Give each service its own role and a distinct, meaningful session name. A shared role like ProductionDataReader is what made this containment hard.
• Require sts:SourceIdentity on cross-account role assumption, so sessions carry a name that cannot be changed later. Use sts:RoleSessionName to require a meaningful session name too.
• Restrict the trust policy to specific principals and your organization (aws:PrincipalOrgID).
• Keep session durations short.
• Require IMDSv2 on instances to make metadata credential theft harder. A hop limit of 1 keeps the token response on the instance, but it can break containers, so test first. The ec2:RoleDelivery condition key can enforce it in policies, and CloudTrail's ec2RoleDelivery field shows whether a session's credentials came from IMDSv1 or IMDSv2.
• On sensitive buckets, require access through a VPC endpoint and named roles.
• Alert on AssumeRole from unusual IPs or tools, and on unusual download volume.
• Send CloudTrail to a separate log archive account with log file validation enabled.
• Write and practice the containment steps above, so nobody learns IAM behavior during an incident.

7. Briefing the incident commander

Use this order: decision, mechanism, verification, impact. For example:

8. Things that sound right but are not

Summary

• Start from the known S3 activity, then link it to the STS session by access key ID, not by IP or time.
• Say clearly what the logs prove, what they suggest, and what is unknown.
• Changing a trust policy does not end sessions already issued.
• Use a targeted deny only when an attribute is truly unique. Otherwise use a cutoff deny with aws:TokenIssueTime, and know its scope and collateral damage.
• Neither deny stops the attacker from assuming the role again, so also block the compromised caller and isolate the source.
• If data is actively leaving, stop it first and coordinate the disruption.
• Contain the source in Account A, but isolate the host before revoking its role.

References

• AWS docs: Revoking IAM role temporary security credentials
• AWS docs: Logging IAM and AWS STS API calls with CloudTrail
• AWS STS API reference: AssumeRole
• AWS docs: CloudTrail userIdentity element
• AWS docs: Global condition context keys
• Amazon GuardDuty: IAM finding types
• Amazon Athena: Query AWS CloudTrail logs