Hybrid Identity Automation

Each customer has its own BigFix, AD, Graph and Exchange credentials, resolved at runtime by a naming convention keyed off the customer identifier — so onboarding is a few credential records, not a code change. The decision that mattered was where the secret is allowed to exist. Credentials live in ServiceNow’s encrypted store and are resolved and injected by the MID Server, so they’re never written into the BigFix action that lands on the customer’s DC and never appear in an endpoint log — the endpoint receives a script and data, never a password. The reset path is the sharpest case: the new password is generated on the domain controller itself, so it never travels inbound, never sits in a payload, and is delivered to the user by SMS. The rejected alternative — embedding credentials in the dispatched action, as the naive BigFix integration does — would scatter every customer’s secrets across endpoint action-histories and logs. The newer sibling automations standardised this into a single Credential Resolver (WCM, with sources such as Azure Key Vault behind it); this flow predates that and resolves from the encrypted credential tables directly.

A request to act on “user X” still has to resolve to some customer’s domain controller. The easy version trusts the page — the front-end says which tenant, the engine obeys — which is a cross-tenant privilege-escalation hole: anyone who can shape a request points it at another customer’s AD. So for the portal the target customer, DC, BigFix site and credentials are resolved server-side from the signed-in user’s customer association, never from anything the browser sent. The browser is treated as hostile. (The one caller permitted to name a tenant is the server-to-server REST front-end, which authenticates as an OAuth 2.0 confidential client — a trusted backend, not an end user.) The cost is wiring each user to a customer at onboarding; the benefit is that no crafted request can cross a tenant boundary, because the boundary isn’t carried in the request.

Onboarding a customer means learning their directory: FQDN, NetBIOS name, base DN, the default OU for new users, the email domain, whether they’re Exchange-hybrid, and which OUs are privileged. The two obvious paths are a manual intake form (slow, error-prone, and customers often don’t know the answers) or blind auto-configuration (dangerous). I built a third: a PowerShell survey that runs on the DC itself and infers each fact, emitting it as a {value, confidence, evidence} triple — not a bare value, but the value, how sure it is, and the AD object it read to decide. A human adjudicates the low-confidence ones. The machine does the forensics; the person makes the call — and onboarding a new customer becomes a config row rather than a code change.

The model is fail-closed from the first insert: creating a customer’s config stamps deny-first rules — MSP staff get scoped access, a customer’s own users default to deny. On top of that, a protected set — the well-known privileged SIDs (Domain Admins, Enterprise Admins, the Tier-0 accounts), plus any restricted OUs the onboarding survey flagged — is blocked for everyone, including MSP staff, and the protection is transitive: nested membership is caught with the LDAP matching-rule-in-chain OID 1.2.840.113556.1.4.1941, so a Tier-0 account can’t be smuggled in behind a benign-looking group. Crucially the gate is enforced in PowerShell on the domain controller — re-resolving SIDs, walking the chain, locale-independent — not in the UI. The browser guard only mirrors it; the real one runs where the change actually happens, because anything the client enforces, the client can bypass.

BigFix is fire-and-forget: you POST an action and get a dispatch acknowledgement — no webhook, no completion event, no return channel. The trap is treating the 200 on dispatch as “finished”; it only means “accepted.” So completion is gated on data freshness, not transport status: the endpoint script writes a result.json stamped with this run’s RITM, and ServiceNow reads it back through the BigFix Client Query API in a Do-While loop — 15 s for AD ops, 5 min for the AAD sync — comparing the embedded RITM before trusting the file, so a stale result from a previous run can never be mistaken for this one. Client Query, not the older Analysis-property approach, because it’s built for one-shot interactive reads. This drop-and-read round-trip became the standard ServiceNow-to-BigFix pattern across the estate, factored into a reusable Poll for Result subflow.

The focused view below isolates just that round-trip — the ad-hoc BigFix action and the Client Query read-back that pre-populate the form:

The portal felt slow because an on-load call warmed a catalog synchronously, holding a shared platform semaphore for several seconds — and the hold scaled with viewers, not actions, so a handful of admins opening the portal at once could starve a node. The tempting fix is to tune the poll interval, but that’s a constant-factor win that leaves the thread-hold in place. The real fix was to take the synchronous work off the load path entirely: the portal does an instant, tenant-scoped cache read while the catalog warms on a background worker, out of band. Onboarding fires an async event and returns rather than blocking on the build, and if a load ever hits a cold cache it doesn’t error — it shows a graceful “indexing…” and triggers the warm itself, so the system repairs its own cold-start. A read that held a thread for seconds now returns in milliseconds, and concurrency stopped being a scaling axis.

The brittle way to run PowerShell on a DC through BigFix is to inline the script as a string — but BigFix ActionScript, the JSON payload and the BES XML envelope each claim {} and XML entities as their own delimiters, so an inline script means three nested layers of escaping fighting each other (plus the Windows-1252 characters in Scandinavian names that the BES parser rejects outright). Instead the scripts live in a table as versioned records, are Base64-transported so no layer can misread them, and are decoded and run on the endpoint. Escaping stops being a problem because nothing is ever escaped — the payload is opaque bytes until it lands on disk. The bonus: every endpoint runs a known, auditable, versioned script, not a string assembled at dispatch time.

Every membership field is {Added:[], Removed:[]}, never “the desired full list”. A replayed or partial HR update therefore can’t silently strip a user out of groups that weren’t part of the change — the worst case is a no-op, not data loss. The rejected alternative, full desired-state sync, is cleaner on paper but wipes any membership the caller didn’t know to include, which for a directory you don’t fully own is unacceptable. The protected groups are also un-removable through the payload: a well-formed Removed entry targeting them is dropped server-side. Names Phase 1 can’t find on-prem aren’t treated as failures either — they route forward: an Entra group goes to Graph, a mailbox to Exchange Online, so the caller never has to know which system owns each name.

Config over code. Customers describe the same thing differently — DK vs Denmark, the c / co / countryCode triple that has to stay consistent — so attribute mapping is declarative, normalised by a resolver with a safe fallback rather than branching code per customer. Onboarding variance is absorbed as configuration, not engine changes.

A few platform edges I had to engineer around, kept here for the record:

• Cookie stripping. Cookie-based session auth to BigFix worked in direct testing but failed through the MID Server — the ECC Queue strips Cookie headers when it serialises a REST call into a queue record. Fix: Basic auth with the credential decrypted in-script (GlideEncrypter) and the Base64 header built by hand, because a standard header survives serialisation.
• Password(2Way) corruption. Flow Designer’s two-way password variables arrived truncated and garbled between steps. Rather than fight the platform, every script that needs a secret resolves it at the moment of use — more code, but deterministic and observable.
• Exchange Online needs a real host. Connect-ExchangeOnline won’t load inside ServiceNow, so Phase 3’s mailbox work runs on the MID Server itself, authenticating with a per-customer certificate pinned in its Windows certificate store; the cert never leaves the box and the app registration is scoped to mailbox management only, so a compromise elsewhere can’t forge mailbox permissions.