Testing and Troubleshooting Conditional Access Policies

How to test conditional access policies using report-only mode, monitor conditional access results using sign-in logs and how to troubleshoot policies using the what-if tool

Testing Conditional Access Policies

Assign the policy to a small group of users for testing first, then assign to all users if there are no issues

Create an AD group for a test group of users
Example: CAP Azure Require MFA for Administrators Test

Example: applying a policy to a test group

Users and groups - Include - Test AD group

Enable the policy in report-only mode

Enable the policy in report-only mode, then monitor the sign in logs to make sure the policy is working as expected

Enable policy - report-only

Monitoring sign-in logs

Azure AD - Monitoring - Sign-ins

Select the sign-in request and check the conditional access tab to see the policy results

In this example the policy "Grant Guest Access" was not applied, you can get more info by clicking on show details

In this example the policy conditions were not met as the user is not a guest user so the policy was not applied

Check the report-only tab to see the conditional access policy report-only results

In this example we can see that the policy "Require MFA for Administrators" was not applied because the policy is in report-only mode but the policy result would be "User action required" which means that the user would be prompted to setup MFA at logon

Clicking on show details will give you more info about the report-only results

In this example the details show that the policy would be applied as the user has an admin role and the access control would be granted only if the user completes MFA "User action required"

Testing policies using the what-if tool 

You can use the What If tool to troubleshoot conditional access policies. The What If tool allows you to see the result of conditional access policies that will apply with a selected user and conditions.

Conditional - Access - Policies - What If

Select the user and the conditions that you want to test e.g. cloud apps, location, client app etc

In this example I've selected a user, any cloud app and browser as the client app and I can see the result of the conditional access policy that will apply "Require MFA for Administrators"

You can also see policies that will not apply

Enable policy - On