Testing Conditional Access without locking yourself out: report-only mode and the What If tool
A poorly scoped Conditional Access policy can block thousands of users or, worse, every administrator in the tenant. That fear often explains why policies sit disabled for months. Entra ID nevertheless provides the tools to test before enforcing; they just need to be used methodically.
Why testing should be systematic
Conditional Access evaluates every sign-in against all enabled policies. Effects add up: a new policy may look harmless on its own and still block a scenario nobody anticipated, such as a service account, a legacy application or an external partner. Testing is therefore not an optional precaution; it is what allows you to enforce ambitious policies without backing down at the first incident.
Report-only mode
A policy in report-only mode is evaluated on every sign-in, but its controls are not enforced. The result is recorded in the sign-in logs, in a dedicated tab, with one of the following outcomes: success, failure, user action required or not applied. You can see, on real sign-ins, who would have been blocked or prompted for MFA.
A few precautions:
- keep the policy in report-only long enough to cover business cycles: month-end, on-call duty, travel;
- a policy requiring a compliant device can, even in report-only, trigger certificate selection prompts on some platforms; check the current Microsoft documentation;
- Microsoft-managed policies are initially deployed in report-only: review them rather than letting them switch on without analysis.
The What If tool
The What If tool simulates a sign-in from parameters you choose: user, application, IP address, platform, client app, risk level. It shows which policies would apply, which would not, and why. It is the ideal tool to check an exclusion, understand a block reported by a user, or confirm that an emergency access account is not covered by any blocking policy.
It does have limits: it simulates a theoretical situation and does not reproduce every signal of a real sign-in. It complements report-only mode rather than replacing it.
The insights and reporting workbook
To analyse the impact of a policy at scale, the Conditional Access insights and reporting workbook aggregates results per policy: number of users affected, applications involved, breakdown of outcomes. It requires sending sign-in logs to a Log Analytics workspace, which carries a storage cost you should plan for. It is what answers the key question before enforcement: how many users would be blocked, and which ones?
A staged rollout
- Create the policy in report-only, with the necessary exclusions from the start, especially emergency access accounts.
- Analyse the results over one or more weeks, and address every failure: justified exclusion, configuration fix or user support.
- Enable it for a pilot group that is representative and includes members of the IT team.
- Expand in waves, watching the logs and helpdesk calls.
- Document the policy: purpose, scope, exclusions and owner.
Emergency accounts. Before any enforcement, use the What If tool to confirm that your break-glass accounts are not caught by any policy that could block them. It is the only guarantee that you can fix a configuration mistake without opening a ticket with Microsoft.
What an audit checks
- The number of policies left in report-only or disabled for a long time, with no decision made.
- Effective exclusion of emergency access accounts from every blocking policy.
- Sign-in logs sent to a Log Analytics workspace so that impact can be analysed.
- Documentation of exclusions and their periodic review.
- A progressive rollout process for new policies.