Under attack? Get incident response help

Identity and access

Roll out Conditional Access in report-only mode first

How a small Microsoft 365 team can trial MFA, legacy authentication and risky sign-in policies in report-only mode, read the logs and enforce safely.

Also in:TürkçeDeutschFrançaisEspañolItalianoPolskiNederlandsEesti

Illustration: Roll out Conditional Access in report-only mode first

Create every baseline Conditional Access policy in report-only state, read what it would have done to real sign-ins, fix the exclusions, and only then switch it to On. In report-only mode the system evaluates the policy at sign-in but does not enforce it, so nobody is prompted or blocked, with one exception for compliant-device checks covered below. That is the cheapest way to find the service account or old mail client that would otherwise break.

Check what Microsoft has already staged

You may not be starting from zero. Microsoft-managed policies are created in eligible tenants in a Report-only state, and Microsoft enables them no less than 30 days after introduction if you leave them alone. Email and Message center posts warn you 2 weeks before. In some cases the delay is shorter, and the notices and the policy details say so.

Open Entra ID > Conditional Access > Policies as at least a Conditional Access Administrator and look for entries with Microsoft in the Created by column. The baseline set relevant here includes:

  • Block legacy authentication
  • Multifactor authentication for admins accessing Microsoft Admin portals (14 admin roles)
  • Multifactor authentication for all users
  • Multifactor authentication and reauthentication for risky sign-ins
  • Block device code flow

You can change their state and exclusions, but not rename or delete them. For more changes, duplicate the policy and edit the copy. Setting a managed policy to Off opts you out, so do that on purpose and write down why.

Licensing matters. Risk-based policies need Microsoft Entra ID Protection, a Microsoft Entra ID P2 feature. Creating Conditional Access policies also prevents you from enabling security defaults, so the two are not meant to be combined.

Set exclusions before you read anything

Exclude your emergency access (break-glass) accounts from every policy, including the Microsoft-managed ones. If a policy is misconfigured and all administrators are locked out, those accounts are how you get back in.

Service accounts need a decision of their own. Calls made by service principals are not blocked by policies scoped to users. Microsoft's planning guidance suggests replacing accounts used in scripts or code with managed identities, and using Conditional Access for workload identities where you need to target service principals.

A policy that combines Block and All resources can lock out admins, and exclusions cannot be configured for important endpoints such as Microsoft Graph, as Microsoft's planning guide warns.

One quirk applies if you add a compliant-device requirement. Report-only policies that check device compliance can prompt users on macOS, iOS and Android to select a device certificate, repeatedly, until the device is compliant. Exclude those platforms from report-only policies that check compliance.

Read the results

Creating or changing policies needs the Conditional Access Administrator role. Use a test user in a test group before pointing a policy at everyone.

Then open a sign-in event in the sign-in logs and go to the Report-only tab. Each policy shows one of these results:

ResultWhat it tells you
Report-only: SuccessConditions and non-interactive controls were met, for example an MFA claim already in the token
Report-only: FailureConditions matched but the controls were not met, for example a block control or a non-compliant device
Report-only: User action requiredThe user would have had to do something, such as an MFA challenge. They were not prompted
Report-only: Not appliedThe conditions did not match, for example the user is excluded

Failure is the line to chase. For a legacy authentication block, a failure is an old client or protocol still in use, and Microsoft names Office 2010 and IMAP, SMTP and POP3 as typical sources. Open Policy impact (preview), choose 7 days or 1 month, and follow the link to sample sign-in events for the legacy policy. Record the client and protocol from each event and name an owner before you enforce.

Policy impact covers interactive sign-ins over the past 24 hours, 7 days or 1 month. The Conditional Access Insights and Reporting workbook compares report-only and enforced results side by side. It needs an Azure Monitor subscription, Microsoft Entra ID P1 and a Log Analytics workspace that receives your sign-in logs.

Report-only has gaps. It does not cover policies scoped to the User Actions scope. And the risky sign-in policy does not include guest users, so do not assume it protects them.

Enforce one policy at a time

After you confirm the results, move the Enable policy toggle from Report-only to On. Do it one policy at a time, in order of how much breakage you expect. Tell users first what changes, when, and where to get support. A prompt nobody expected becomes a helpdesk ticket. If your team is small, our consulting service can review the policy set and the logs with you.

When you later change an enforced policy, copy it, put the copy in report-only with the proposed change, and compare its results with the original. Once you are satisfied, update the enforced policy and remove the copy. Mind the limit of 240 policies per tenant, which counts policies in any state, so clean up copies.

Turn on protected actions as well, so that creating, changing or deleting a Conditional Access policy requires fresh verification.

Where to start

  1. Check Entra ID > Conditional Access > Policies for Microsoft-managed policies, and note the enablement date from the Message center or email notice.
  2. Create or confirm a break-glass exclusion on every policy, and decide how each service account will be handled or replaced.
  3. Make sure you have an Azure Monitor subscription and Microsoft Entra ID P1, then stream sign-in logs to a Log Analytics workspace so the Insights and Reporting workbook works.
  4. Put the legacy authentication block and the admin MFA policy in report-only, then review Failure and User action required results in the Report-only tab.
  5. Fix or migrate each failing client or account, tell users, then switch one policy to On and watch the sign-in logs.
  6. Repeat for MFA for all users and the risky sign-in policy (P2 only), and enable protected actions.

How we can help

Consulting and training

We assess where you stand, set priorities with you and train your staff to recognise attacks.

Contact us
AngelForce team

Written by the AngelForce team: security consultants with CEH, LPT Master and CISA certifications who test, monitor and defend systems for organisations across Europe.

Written with AI assistance from public sources listed below, and reviewed by the AngelForce team before publication.

Sources

  1. learn.microsoft.com/en-us/entra/identity/conditional-access/concept-conditional-access-report-only
  2. learn.microsoft.com/en-us/entra/identity/conditional-access/plan-conditional-access
  3. learn.microsoft.com/en-us/entra/identity/conditional-access/managed-policies
  4. learn.microsoft.com/en-us/entra/identity/conditional-access/overview

Articles about Identity and access

Tell us what you need to protect

Describe your environment and what you want to achieve. We will reply by email.

Contact us
info@angelforce.eu +372 5705 2044

AngelForce OÜ · Tallinn, Estonia