Report-only rollout and rollback

Pre-flight checks, pilot design, enforcement criteria, and how to get back out safely.
Published by Urbannerd Consulting · Reviewed 2026-07-29

Rollout and rollback

benchmark guidance prescribes: create in report-only → run at least one week → review sign-in logs →

enable. Microsoft publishes a three-phase, ~4-week schedule (see

). Both agree on the one-week

report-only minimum. This file is the operational detail around that.

Microsoft's four gate conditions before deploying any policy: break-glass excluded from

all policies · tested with a pilot group · users have registered the required authentication

methods · changes communicated with supporting documentation.

The sequence

0. Emergency access accounts exist, excluded, and sign-in TESTED
1. Design and peer review
2. Pre-flight data collection
3. Report-only  (minimum 1 week per benchmark guidance; longer for risk and device policies)
4. Sign-in log analysis
5. Pilot group enforcement
6. Staged wider enforcement
7. Full enforcement
8. Post-deployment monitoring and exclusion review

Step 0 is a gate, not a step. Nothing proceeds until a break-glass account has actually

been signed into successfully. A configuration screenshot is not a test.

Step 2 — pre-flight data collection

Collect before report-only, because report-only tells you what would have happened, not

whether users could comply.

Step 3 — report-only

• Minimum one week per benchmark guidance. Extend to two to four weeks for: sign-in risk policies

(behaviour varies seasonally — month-end, travel, holidays), device compliance (enrolment

lags), anything touching guests.

• Report-only must cover a full business cycle. A week that misses month-end close

misses the finance team's batch jobs.

• Report-only does not evaluate everything. Known limits worth stating: it does not

apply to some non-interactive flows, and session-control outcomes are not fully

represented. Treat report-only as necessary, not sufficient. [VERIFY] the current

documented limitations before asserting specifics.

• Use the What If tool alongside report-only for targeted "would this user be blocked"

questions, especially for location and geographic policies (benchmark guidance explicitly recommends

this for 5.2.2.15).

Step 4 — sign-in log analysis

Review, for the report-only period:

• Sign-ins where the policy result is Would have failed — grouped by user, app, client

app, device state and location.

• Whether failures cluster on a population (a department, a device type, guests) or are

scattered.

• Non-interactive and service-principal sign-ins, which people routinely forget to check.
• Legacy client app sign-ins that would be blocked.
• Guest sign-ins.

The decision to enforce is: **every "would have failed" is either understood and accepted,

or has a remediation.** Not "the number looks small".

Step 5 — pilot group design

• 5–10% of the population, or 20–50 users, whichever is smaller at first.
• Must be representative, not volunteers: include at least one of each device type,

one remote worker, one frontline/shared-device user, one heavy mobile user, and — where

in scope — one guest and one service scenario.

• Include IT, but IT must not be the only pilot group. IT is the least representative

population in any organisation.

• Never pilot on executives first. They have the lowest tolerance and the highest escalation.
• Run at least one week enforced before widening.

Test the exclusions, not just the inclusions. Microsoft is explicit about this and it is

the step almost everyone skips:

Ensure you test the exclusion criteria of a policy... test whether the excluded users are

prompted for MFA, because **the combination of other policies can require MFA for those

users**.

An exclusion in one policy is not an exemption from the estate. This applies directly to

break-glass validation — excluding the account from Policy A proves nothing if Policy B still

catches it. The only proof is an actual successful sign-in.

Microsoft's other caveat worth repeating: a simulated run via What If *"gives you a good idea

of how a Conditional Access policy affects sign-in, but it doesn't replace an actual test run

in a properly configured development environment."* Do not oversell What If.

Step 6 — staged enforcement

Widen in tranches: pilot → one department → a business unit → all. Between tranches,

check helpdesk ticket volume and sign-in failure rate. Stop widening if either moves.

Communication

Registration campaigns must run before enforcement, not alongside it.

Rollback

Fastest safe rollback: set the policy to Report-only rather than Off. It stops

enforcement immediately while preserving the policy and continuing to generate data about

what it would have done.

Microsoft's three rollback options, in order: disable the policy · **exclude the affected

user or group** (with their caution: *use exclusions sparingly, only where the user is

trusted; add users back as soon as possible*) · delete it if disabled and no longer needed.

Rollback facts the agent must state accurately:

• Policy changes apply to new token requests. Users already holding valid tokens keep

access until those expire or continuous access evaluation revokes them, so rollback is

not instantaneous for signed-in users — and conversely, "it still works for me" is not

evidence the rollback failed.

• **Deleted policies and named locations can be restored within a 30-day soft-delete

period**, by a Conditional Access Administrator. Most admins do not know this. Offer it

whenever someone reports an accidental deletion.

• Revoke refresh tokens for affected users after any window where controls were relaxed —

especially privileged accounts. Without it, sessions established under the weaker posture

survive the restoration.

• If administrators are locked out, break-glass is the route. This is the one moment the

accounts exist for, and using them must trigger the benchmark guidance alert. **If break-glass does

not exist and no admin can reach the policy, the only remaining route is a Microsoft

support request** — Microsoft reviews and, after confirming, updates the policies. It is

slow, and saying so plainly is the strongest argument for configuring break-glass first.

• Record every rollback: what triggered it, who authorised it, what changed.

Contingency policies — prepared in advance, held in report-only

Distinct from rollback. Rollback undoes your change; a contingency policy restores access

when a control becomes unavailable — an MFA provider outage, a device-compliance evaluation

failure. Microsoft's full treatment is in .

The shape: pre-build backup policies that omit the failing control but **compensate by

narrowing** — restrict the population, restrict the platform, restrict the network, close the

legacy path — then disable the primary policy. Hold them in report-only when not in use so

their impact is visible and they are one toggle from active.

Naming: EMnnn - ENABLE IN EMERGENCY: [Disruption][i/n] - [Apps] - [Controls] [Conditions]

Prerequisite the agent should ask for: **which applications are Category 1 (minutes),

Category 2 (hours), Category 3 (days)?** Microsoft is explicit that business, security, legal

and leadership must agree this list before an incident, not during one.

During a degraded window: document every change and the prior state · assume attackers will

password-spray and phish while MFA is off, and reset critical users' passwords before

disabling MFA for them · archive all sign-in activity · triage every risk detection.

Rollback decision triggers — define these before enforcing, not during an incident:

Step 8 — post-deployment

• Watch sign-in failures for two weeks after each enforcement step.
• Exclusion review: every exclusion added during rollout needs an owner and a removal

date. Put the exclusion groups under an access review (benchmark guidance/5.3.3). This is the

control that stops "temporary exclusions that became permanent".

• Re-run the MFA capable report to confirm the population moved.
• Confirm break-glass monitoring is firing — test it .
• Schedule a quarterly review of the whole policy set.

Per-policy deployment template

The agent produces this for each policy in a baseline:

1. Prerequisites — what must be true first
2. Pilot scope — who, and why they are representative
3. Report-only duration — with the reason for the length
4. Logs to review — the specific fields and filters
5. Known exclusions — with owner and review date
6. User communication — audience and timing
7. Enforcement criteria — the measurable condition for going live
8. Rollback — trigger conditions and the exact steps