IAM & Permission Troubleshooting

Cloud Troubleshooting & Support

Chapter 3 · IAM & Permission Troubleshooting

Following Chapter 2's connectivity flowchart, this chapter tackles the second-most-common category of cloud support ticket: "access denied." It builds directly on cloud1-6's IAM foundations, turning that chapter's concepts into a genuine, practiced investigation.

Restating the Foundation — Cloud1-6's AuthN vs. AuthZ Triage

Recall the split: is this a login problem, or a "logged in but blocked" problem? This chapter assumes authentication is already confirmed working, and focuses entirely on authorization — the much larger and more nuanced category of real IAM tickets in practice.

Reading a Policy Error Message

Every provider's "access denied" error carries real diagnostic information, often skipped past by a frustrated user or engineer. Worth reading in full, every time, before doing anything else:

  • The specific action that was denied (e.g. s3:GetObject), not just "something failed."
  • The specific resource it was attempted against — the exact identifier, not just "a bucket."
  • Sometimes, which specific policy or organizational boundary actually caused the denial — modern provider tooling increasingly surfaces this directly.
Read the full error message before guessing
One of the most common, easily avoidable time-wasters in real IAM troubleshooting is seeing "access denied," assuming a cause, and starting to make changes — rather than reading the specific action and resource the error already names. The error message is frequently the fastest path to the actual answer.

The Debugging Order

Revisiting cloud1-6's explicit-deny-always-wins nuance, structured as an actual investigation sequence:

  1. Confirm the identity is actually who you think it is — the role or account actually in use, not assumed.
  2. List all policies attached to that identity — direct, via a group, via an assumed role — not just the obvious one.
  3. Check for any explicit deny across all of them — one deny anywhere wins, regardless of allows granted elsewhere (cloud1-6).
  4. Check resource-level restrictions and condition keys — is the allow scoped to a different specific resource than the one actually being accessed?
  5. Check for a permission boundary or organization-level restriction that could override even a correctly-configured identity-level policy.

Permission Boundaries & Organization-Wide Policies

A layer cloud1-6 didn't have room to cover properly: Service Control Policies (AWS), Azure Policy, and GCP Organization Policies apply at the organization or account level — above and entirely independent of any individual identity's own IAM policies. Even a user with a policy explicitly granting full administrator access can still be blocked by an org-level restriction, which is genuinely surprising to someone only checking the identity's own policies. This is a real, common source of confusing tickets where "the policy looks completely correct" and access is still denied — the missing piece is invisible at the identity level entirely.

A Common Real Scenario — Cross-Account/Cross-Resource Access

Revisiting cloud1-6's cross-account role-assumption material in a troubleshooting context: a genuinely common real ticket pattern involves two separate policies that both need to be correct.

Trust policy vs. permission policy — a common, subtle confusion
A trust policy on the target role controls who is allowed to assume it. A permission policy controls what they can do once assumed. These are separate, independently configurable settings — a misconfigured trust policy prevents the role from ever being assumed at all (an error at the assumption step itself), while a misconfigured permission policy allows assumption to succeed but denies specific actions afterward. Confusing which one is broken wastes real troubleshooting time.

Practical Tools for IAM Debugging

Each provider offers policy simulation or testing tools — AWS's IAM Policy Simulator, Azure's "Check access" feature, GCP's Policy Troubleshooter — genuinely useful for confirming exactly why an action would or wouldn't be allowed, without needing to actually attempt the action for real against production.

A Support-Relevant Workflow — Putting It Together

1. CONFIRM IDENTITY -- who is actually making this request 2. GATHER ALL POLICIES -- direct, group, role — not just the obvious one 3. CHECK FOR EXPLICIT DENY -- one deny anywhere wins 4. CHECK RESOURCE/CONDITION SCOPING 5. CHECK ORG-LEVEL/BOUNDARY POLICIES 6. USE THE PROVIDER'S POLICY SIMULATOR TO CONFIRM

This is genuinely IAM troubleshooting's own version of Chapter 2's connectivity flowchart — the same disciplined, layer-by-layer structure, applied to a completely different kind of problem.

Hands-On Exercises

Exercise 1

A user has a policy that appears to grant full access to a service, and no explicit deny is found on any attached policy — yet access is still denied. What layer, not covered by checking the identity's own policies, should be checked next?

📄 View solution
Exercise 2

Explain the difference between a trust policy and a permission policy in the context of assuming a role in another account, and describe what a misconfiguration in each would actually look like as a symptom.

📄 View solution
Exercise 3

Explain why reading the full "access denied" error message — rather than just seeing "denied" and starting to guess — saves real time, using the specific pieces of information this chapter says such an error typically contains.

📄 View solution

Chapter 3 Quick Reference

  • This chapter assumes AuthN already works — it's entirely about AuthZ (cloud1-6)
  • Read the full error: specific action + specific resource + (sometimes) which policy caused the denial
  • Debugging order: confirm identity → gather all attached policies → check for explicit deny → check resource/condition scoping → check org-level/boundary policies
  • SCPs/Azure Policy/GCP Org Policies sit above identity-level policies entirely — a correct-looking identity policy can still be overridden
  • Trust policy (who can assume a role) ≠ permission policy (what they can do once assumed) — two separate, both-required settings
  • Use the provider's own policy simulator (IAM Policy Simulator/Check access/Policy Troubleshooter) to confirm without testing against production
  • Next chapter: Reading Logs & Metrics Under Pressure — building on cloud1-8's metrics-then-logs workflow