Blake Drumm - Technical Blog

Sending Emails with Azure Communication Services (ACS) — A C# Console Application with Custom Headers

March 6, 2026

💡 Introduction

Azure Communication Services (ACS) provides a powerful Email API that allows developers to send transactional and notification emails directly from applications without managing SMTP infrastructure.

While the official SDK documentation demonstrates the basic steps required to send an email, real-world implementations often require additional capabilities such as:

Custom email headers
Reply-to address handling
HTML email templates
Tracking identifiers for correlation
Logging and troubleshooting visibility
Automation-friendly execution behavior

To explore these scenarios, I built a small C# console application that demonstrates sending a formatted email using the ACS Email SDK while implementing several practical features that are commonly required in production environments.

This post explains how the application works, how to run it, and includes a deeper look at the internal logic used to send and track emails through ACS.

💾 Download the Code

You can download the full source code used in this article here:

🔗 Download the Email Quickstart Console Application
https://files.blakedrumm.com/EmailQuickstart-ACS.zip

After downloading, extract the archive and open the project folder in your preferred IDE such as Visual Studio or VS Code.

📬 What the Application Does

This console application sends a formatted HTML email using the Azure Communication Services Email SDK.

Key capabilities include:

Sending HTML and plain-text email bodies
Adding custom email headers
Supporting reply-to addresses
Generating a tracking identifier for correlation
Polling the EmailSendOperation until completion
Logging timestamps and send duration
Supporting interactive and automated execution modes

Because of this functionality, the tool can be used as both:

a developer testing utility
a troubleshooting tool
a validation tool for ACS email configuration

🏗️ High-Level Architecture

The application flow mirrors how the ACS Email service processes messages.

Console Application
        ↓
Azure.Communication.Email SDK
        ↓
Azure Communication Services
        ↓
Recipient Mail Server

The console application constructs an email message, submits it through the ACS Email SDK, and waits for the asynchronous send operation to complete.

⚙️ Prerequisites

Before running the application, ensure the following requirements are met.

1. Install .NET

The application requires the .NET SDK.

https://dotnet.microsoft.com/download

2. Create an Azure Communication Services Resource

In the Azure Portal:

Create an Azure Communication Services resource
Navigate to Keys
Copy the connection string

Example format:

endpoint=https://<resource>.communication.azure.com/;accesskey=<key>

3. Configure ACS Email

Your ACS resource must have:

An email domain configured
A verified sender address

Without this configuration, email sends will fail.

4. Install the Required SDK

The project references the ACS Email SDK.

If you are creating a new project manually, install it using:

dotnet add package Azure.Communication.Email

🔑 Configure the Connection String

The application reads the ACS connection string from an environment variable.

Environment variable name:

COMMUNICATION_SERVICES_CONNECTION_STRING

Windows (Command Prompt)

setx COMMUNICATION_SERVICES_CONNECTION_STRING "endpoint=https://<resource>.communication.azure.com/;accesskey=<key>"

PowerShell

$env:COMMUNICATION_SERVICES_CONNECTION_STRING="endpoint=https://<resource>.communication.azure.com/;accesskey=<key>"

When the application starts, it validates that this variable exists before attempting to send an email.

▶️ Build the Application

Navigate to the project directory and run:

dotnet build

▶️ Run the Application

You can run the application using:

dotnet run

Or execute the compiled binary:

EmailQuickstart-WithCustomHeaders.exe

When executed, the application sends a test email and logs progress in the console.

🖥️ Interactive Mode

If the application detects it is running in a terminal session, it enters interactive mode.

Available commands:

Ctrl + R  → resend email
Enter     → exit application

This allows rapid testing without restarting the program.

If the application runs in non-interactive mode, it sends a single email and exits automatically.

📧 Email Template and Message Content

The application constructs both:

an HTML message body
a plain-text fallback body

The HTML template includes common elements used in production email layouts:

Responsive layout
Preheader text
Call-to-action button
Tracking identifier
Optional “view in browser” link

The template contains placeholder tokens that are dynamically replaced before sending.

Examples include:

##TRACKING_ID##
##VIEW_IN_BROWSER_URL##
##REPLY_TO_EMAILS##

🏷️ Custom Email Headers

The project demonstrates the use of custom email headers to include metadata with each email.

Examples include:

Importance: normal
X-Mailer
X-Campaign
X-Environment

Custom headers can help with:

campaign tracking
environment identification
message classification

🔎 Tracking and Correlation

Each email send generates a unique tracking identifier.

Guid.NewGuid()

This identifier can be inserted into:

email headers
message body
console logs

This allows correlation between:

application logs
Azure Monitor diagnostics
external email systems

⚙️ How It Works Internally (Deep Dive)

Step 1 — Initialize the Email Client

The application creates an EmailClient using the ACS connection string.

var emailClient = new EmailClient(connectionString);

Step 2 — Build the Email Message

An EmailMessage object is constructed containing:

sender
recipients
subject
HTML body
text body
reply-to addresses
custom headers

Step 3 — Submit the Send Request

The EmailClient submits the message to Azure Communication Services.

ACS returns an EmailSendOperation, which represents the asynchronous send request.

Step 4 — Poll the Operation

Because email sends are asynchronous, the application polls the operation until completion.

During this process it logs:

operation state
elapsed time
completion status

Step 5 — Log the Results

Structured logs are written to the console including timestamp and execution duration.

Example output:

[2026-01-16T20:03:14Z +2.31s] [INFO] Email send operation started.

🧪 When This Tool Is Useful

This console application is helpful for:

validating ACS email configuration
testing custom headers
troubleshooting delivery failures
verifying domain and sender configuration
generating diagnostic logs
validating HTML email templates

It effectively acts as a developer testing harness for Azure Communication Services Email.

⚠️ Common Issues When Testing ACS Email

401 Unauthorized

Usually caused by:

incorrect connection string
expired or regenerated access key

Sender Not Verified

The sender address must belong to a verified domain configured in ACS.

Email Never Arrives

Possible causes include:

recipient spam filtering
missing SPF or DKIM records
invalid recipient address
message size or attachment restrictions

🧠 Final Thoughts

Azure Communication Services makes it easy to send email programmatically, but real-world implementations often require more than minimal SDK examples.

This console application demonstrates several useful capabilities including:

HTML email construction
custom header support
reply-to handling
asynchronous operation polling
structured logging

Tools like this make it easier to validate deployments, troubleshoot issues, and experiment with ACS email functionality before integrating it into larger systems.

Deep-Dive Email Delivery Analysis with KQL - Azure Communication Services (ACS)

February 10, 2026

💡 Introduction

Email troubleshooting in Azure Communication Services (ACS) can be deceptively complex.

A message may appear to be sent successfully, yet arrive late, get quarantined, bounce with minimal explanation, or fail silently due to downstream mail server policies. While ACS exposes rich diagnostics in Azure Monitor, those signals are spread across multiple tables and often require correlation to tell a complete story.

This post introduces a production-ready KQL query that joins ACS email send and status logs to provide a single, enriched view of email delivery behavior — including latency, message characteristics, sender and recipient context, and clear interpretations of SMTP and enhanced SMTP failure codes.

📋 Prerequisites

To use this query, you’ll need:

Access to Azure Monitor Log Analytics workspace with ACS diagnostic logs enabled
At least Log Analytics Reader role on the workspace
ACS Email diagnostic logs flowing to the workspace (both ACSEmailSendMailOperational and ACSEmailStatusUpdateOperational tables)
Basic familiarity with KQL (Kusto Query Language)

📩 The Problem with Raw ACS Email Logs

Out of the box, ACS email diagnostics present a few challenges:

Send and delivery status events live in separate tables
Correlating send and delivery events requires understanding MessageId/CorrelationId relationships
SMTP codes are numeric and often unclear without protocol knowledge
Enhanced SMTP codes are easy to miss or misinterpret
Message size and attachment details are not immediately obvious
Bounce behavior (hard vs. soft) requires manual interpretation
Correlating events across systems is difficult without message IDs

When customers ask questions like:

“Why did this email bounce?”
“Why did delivery take 30 seconds instead of 2?”
“Is this a content issue, a mailbox issue, or a policy block?”

You need more than raw logs — you need context.

🔍 What This Query Does

This KQL query was built to answer those questions directly.

At a high level, it:

Joins send events with final delivery status events
Measures end-to-end delivery latency
Enriches results with sender, recipient, and message metadata
Classifies bounces and failures
Translates SMTP and enhanced SMTP codes into readable explanations

All results are returned in a single, sortable dataset.

⚙️ Data Sources Used

The query relies on two ACS diagnostic tables:

1️⃣ ACSEmailSendMailOperational

Used to capture:

Send timestamp
Message correlation ID
Message size (in megabytes, not bytes)
Attachment count
Recipient counts (To, Cc, Bcc, unique)

2️⃣ ACSEmailStatusUpdateOperational

Used to capture:

Final delivery status
SMTP and enhanced SMTP status codes
Sender identity
Failure reason and message
Bounce classification
Recipient mail server details
InternetMessageId for cross-system correlation

📜 The Complete Query

Here’s the full KQL query. You can copy and paste this directly into Azure Monitor Log Analytics:

// -----------------------------------------------
// Description: Analyzes Azure Communication Services email delivery by
//              joining send and status logs to measure delivery latency 
//              and surface clear explanations for SMTP and enhanced SMTP
//              failure codes.
//
// Created by: Blake Drumm ([email protected])
// Date Created: January 21st, 2026
// Enhanced: February 10th, 2026
// Enhancements:
//   - Added sender information (SenderUsername, SenderDomain)
//   - Added message characteristics (Size in MB, AttachmentsCount, recipient counts)
//   - Added detailed failure information (FailureMessage, FailureReason)
//   - Added bounce classification (IsHardBounce, BounceType)
//   - Added recipient domain extraction and mail server tracking
//   - Added InternetMessageId for cross-system correlation
//   - Corrected Size field interpretation (megabytes, not bytes)
// -----------------------------------------------
(ACSEmailSendMailOperational
| project 
    SendTime = TimeGenerated, 
    MessageId = CorrelationId, 
    MessageSize = Size,  // Size is in megabytes (MB)
    AttachmentsCount,
    ToRecipientsCount,
    CcRecipientsCount,
    BccRecipientsCount,
    UniqueRecipientsCount,
    _ResourceId)
| join kind=inner (
    ACSEmailStatusUpdateOperational
    | where isnotempty(RecipientId)
    | where DeliveryStatus in ("Delivered","Failed","Bounced","Suppressed","Quarantined","FilteredSpam")
    | summarize arg_max(TimeGenerated, DeliveryStatus, SmtpStatusCode, EnhancedSmtpStatusCode, 
                        SenderDomain, SenderUsername, FailureMessage, FailureReason, 
                        IsHardBounce, RecipientMailServerHostName, InternetMessageId)
        by MessageId = CorrelationId, RecipientId, _ResourceId
    | project
        MessageId,
        RecipientId,
        _ResourceId,
        FinalStatusTime = TimeGenerated,
        FinalDeliveryStatus = DeliveryStatus,
        SmtpStatusCode = tostring(SmtpStatusCode),
        EnhancedSmtpStatusCode = tostring(EnhancedSmtpStatusCode),
        SenderDomain,
        SenderUsername,
        FailureMessage,
        FailureReason,
        IsHardBounce,
        RecipientMailServerHostName,
        InternetMessageId
) on MessageId, _ResourceId
| where FinalStatusTime >= SendTime
| extend DurationMilliseconds = datetime_diff("millisecond", FinalStatusTime, SendTime)
| extend DurationSeconds = round(todouble(DurationMilliseconds) / 1000.0, 3)
// Extract recipient domain for analysis
| extend RecipientDomain = tolower(extract(@"@(.+)$", 1, RecipientId))
// Categorize message size (Size field is in megabytes)
| extend MessageSizeCategory = case(
    MessageSize < 0.1, "Tiny (<100KB)",
    MessageSize < 1.0, "Small (100KB-1MB)",
    MessageSize < 5.0, "Medium (1-5MB)",
    MessageSize < 10.0, "Large (5-10MB)",
    "Very Large (>10MB)"
)
// Categorize bounce type (handle both "True"/"False" and "true"/"false")
| extend BounceType = case(
    FinalDeliveryStatus == "Delivered", "Not Applicable",
    tolower(IsHardBounce) == "true", "Hard Bounce",
    tolower(IsHardBounce) == "false", "Soft Bounce",
    FinalDeliveryStatus in ("Bounced", "Failed") and isempty(IsHardBounce), "Unknown (not populated)",
    "Unknown"
)
| extend SmtpStatusCode =
    iff(isempty(SmtpStatusCode) and FinalDeliveryStatus == "Delivered", "<not applicable>", SmtpStatusCode)
| extend EnhancedSmtpStatusCode =
    iff(isempty(EnhancedSmtpStatusCode) and FinalDeliveryStatus == "Delivered", "<not applicable>", EnhancedSmtpStatusCode)
| extend SmtpStatusCodeDetails = case(
    SmtpStatusCode == "<not applicable>", "Not applicable (Delivered)",
    SmtpStatusCode == "421", "Service not available; try again later (temporary failure)",
    SmtpStatusCode == "450", "Mailbox unavailable (temporary); retry expected",
    SmtpStatusCode == "451", "Local error in processing (temporary); retry expected",
    SmtpStatusCode == "452", "Insufficient system storage (temporary); retry expected",
    SmtpStatusCode == "500", "Syntax error; command unrecognized",
    SmtpStatusCode == "501", "Syntax error in parameters or arguments",
    SmtpStatusCode == "502", "Command not implemented",
    SmtpStatusCode == "503", "Bad sequence of commands",
    SmtpStatusCode == "504", "Command parameter not implemented",
    SmtpStatusCode == "550", "Mailbox unavailable / policy / access denied (permanent failure)",
    SmtpStatusCode == "551", "User not local; please try forwarding address",
    SmtpStatusCode == "552", "Storage allocation exceeded (mailbox full or message too large)",
    SmtpStatusCode == "553", "Mailbox name not allowed / invalid recipient",
    SmtpStatusCode == "554", "Transaction failed (often policy/content rejection)",
    isempty(SmtpStatusCode), "Unknown / not provided",
    strcat("Unmapped SMTP code: ", SmtpStatusCode)
)
| extend EnhancedSmtpStatusCodeDetails = case(
    EnhancedSmtpStatusCode == "<not applicable>", "Not applicable (Delivered)",
    EnhancedSmtpStatusCode == "5.1.1", "Bad destination mailbox address (recipient doesn't exist)",
    EnhancedSmtpStatusCode == "5.1.0", "Bad destination mailbox address",
    EnhancedSmtpStatusCode == "5.2.2", "Mailbox full",
    EnhancedSmtpStatusCode == "5.2.1", "Mailbox disabled / not accepting mail",
    EnhancedSmtpStatusCode == "5.4.1", "Cannot route / no answer from host / destination routing issue",
    EnhancedSmtpStatusCode == "5.7.1", "Rejected by policy (SPF/DKIM/DMARC/reputation/policy)",
    EnhancedSmtpStatusCode == "4.7.1", "Temporary policy/reputation issue; retry expected",
    EnhancedSmtpStatusCode == "4.4.1", "Connection timed out / temporary network issue; retry expected",
    isempty(EnhancedSmtpStatusCode), "Unknown / not provided",
    strcat("Unmapped enhanced code: ", EnhancedSmtpStatusCode)
)
| project
    MessageId,
    InternetMessageId,
    RecipientId,
    RecipientDomain,
    SenderUsername,
    SenderDomain,
    FinalDeliveryStatus,
    BounceType,
    SendTime,
    FinalStatusTime,
    DurationSeconds,
    DurationMilliseconds,
    MessageSize,
    MessageSizeCategory,
    AttachmentsCount,
    ToRecipientsCount,
    CcRecipientsCount,
    BccRecipientsCount,
    UniqueRecipientsCount,
    SmtpStatusCode,
    SmtpStatusCodeDetails,
    EnhancedSmtpStatusCode,
    EnhancedSmtpStatusCodeDetails,
    FailureReason,
    FailureMessage,
    RecipientMailServerHostName,
    _ResourceId
| order by SendTime desc

⏱️ Delivery Latency Measurement

The query calculates delivery latency using:

SendTime (from send logs)
FinalStatusTime (from status logs)

It then derives:

DurationMilliseconds
DurationSeconds

This makes it easy to:

Identify delayed deliveries
Compare performance across recipient domains
Spot transient vs. systemic delays

📦 Message Characteristics and Size Handling

One common pitfall is assuming the ACS Size field is reported in bytes.

It is not.

This query correctly treats Size as megabytes (MB) and adds:

A MessageSizeCategory field:
- Tiny (<100KB)
- Small (100KB–1MB)
- Medium (1–5MB)
- Large (5–10MB)
- Very Large (>10MB)

Combined with attachment and recipient counts, this helps identify:

Messages that may trigger size-based rejections
Large fan-out sends with higher delivery latency

Note: ACS has a default maximum email size of 10MB (including attachments), though this can be increased to 30MB via support request. Messages exceeding limits will fail before logging.

💥 Bounce Classification

The query explicitly categorizes bounce behavior:

Hard Bounce – permanent failure (invalid mailbox, policy block, etc.)
Soft Bounce – temporary failure (mailbox full, throttling, transient policy)
Not Applicable – delivered successfully

This makes it trivial to separate:

Retry-worthy failures
Permanent delivery issues
Clean deliveries

📜 SMTP and Enhanced SMTP Code Translation

Raw SMTP codes are not very helpful on their own.

This query maps:

Standard SMTP status codes (e.g., 550, 554)
Enhanced SMTP status codes (e.g., 5.1.1, 5.7.1)

Into clear, human-readable explanations, such as:

“Bad destination mailbox address (recipient doesn’t exist)”
“Rejected by policy (SPF/DKIM/DMARC/reputation)”
“Temporary network issue; retry expected”

Delivered messages automatically show:

<not applicable> for SMTP fields

This removes ambiguity and speeds up root-cause analysis.

🌐 Recipient and Server Intelligence

The query extracts and surfaces:

Recipient domain
Recipient mail server hostname
InternetMessageId

This allows you to:

Group failures by destination domain
Identify problematic recipient mail systems
Correlate ACS activity with external mail logs or SIEM data

🎯 Common Usage Patterns

Filter by Specific Sender

| where SenderUsername == "[email protected]"

Find All Bounces to a Specific Domain

| where RecipientDomain == "gmail.com" and FinalDeliveryStatus == "Bounced"

Identify Slow Deliveries (>10 seconds)

| where DurationSeconds > 10 and FinalDeliveryStatus == "Delivered"

Messages with Large Size or Attachments

| where MessageSize > 5.0 or AttachmentsCount > 0

All Hard Bounces in Last 24 Hours

| where BounceType == "Hard Bounce"
| where SendTime > ago(24h)

📊 Aggregation Examples for Dashboards

Delivery Success Rate by Recipient Domain

// Add this to the end of the base query
| summarize 
    Total = count(),
    Delivered = countif(FinalDeliveryStatus == "Delivered"),
    Bounced = countif(FinalDeliveryStatus == "Bounced")
    by RecipientDomain
| extend SuccessRate = round(100.0 * Delivered / Total, 2)
| order by Total desc

Average Delivery Time by Hour

// Add this to the end of the base query
| where FinalDeliveryStatus == "Delivered"
| summarize AvgDeliverySeconds = avg(DurationSeconds)
    by bin(SendTime, 1h)
| render timechart

Top Failure Reasons

// Add this to the end of the base query
| where FinalDeliveryStatus != "Delivered"
| summarize Count = count() by FailureReason, EnhancedSmtpStatusCodeDetails
| order by Count desc
| take 10

Bounce Rate by Sender

// Add this to the end of the base query
| summarize 
    TotalSent = count(),
    HardBounces = countif(BounceType == "Hard Bounce"),
    SoftBounces = countif(BounceType == "Soft Bounce")
    by SenderUsername
| extend HardBounceRate = round(100.0 * HardBounces / TotalSent, 2)
| order by HardBounceRate desc

Message Size Distribution

// Add this to the end of the base query
| summarize Count = count() by MessageSizeCategory
| render piechart

⚠️ Limitations & Known Issues

Subject Lines Are Not Available

Azure Communication Services does not log email subject lines or body content in diagnostic logs for privacy and security reasons. Only metadata (sender, recipient, size, status codes) is captured.

If you need to track subjects, implement custom logging in your application before sending:

// Example: Log subject before sending
logger.LogInformation("Sending email - MessageId: {messageId}, Subject: {subject}", 
    correlationId, emailMessage.Subject);

IsHardBounce Casing

The IsHardBounce field returns “True”/”False” (capital T/F), not “true”/”false”. The query handles this with case-insensitive comparison using tolower().

Time Range Performance

For optimal performance on large datasets, limit queries to specific time ranges (e.g., last 24-48 hours) rather than scanning all historical data. Add this at the start:

| where TimeGenerated > ago(24h)

Correlation Timing

The query filters for FinalStatusTime >= SendTime to ensure logical ordering. In rare cases where clock skew exists, some records may be excluded.

🧪 When to Use This Query

This query is ideal for:

Investigating bounced or delayed emails
Analyzing delivery performance trends
Supporting customer escalations
Validating ACS email configuration
Understanding policy-based rejections
Building workbooks or dashboards for email observability
Identifying problematic recipient domains or mail servers

🚀 Next Steps

Now that you have the query, here are some ways to extend it:

Create an Azure Workbook – Build interactive dashboards using this query as a data source
Set Up Alerts – Create alerts for high bounce rates, delivery latency spikes, or specific failure patterns
Join with Application Logs – Use MessageId or InternetMessageId to correlate with your application’s custom logging
Export to Power BI – For long-term trend analysis and executive reporting
Integrate with SIEM – Feed results into Azure Sentinel or third-party SIEM tools

🧠 Final Thoughts

Azure Communication Services provides excellent email diagnostics — but the real power comes from correlating and enriching that data.

This KQL query turns raw ACS logs into a structured, actionable dataset that answers the questions customers actually ask:

What happened to my email?
Why did it fail?
How long did it take?
Is this temporary or permanent?

If you work with ACS Email regularly, this query is a strong foundation for troubleshooting, reporting, and long-term monitoring.

Feel free to adapt, extend, and share this query with your team. Happy troubleshooting! 🎉

Custom IAM Roles Are Not Supported - Azure Communication Services (ACS)

January 20, 2026

💡 Introduction

When deploying or securing Azure Communication Services (ACS), it’s natural to want to follow least-privilege access principles by creating custom IAM (RBAC) roles.

This is especially common when customers want to:

Separate operational access from administrative access
Grant narrowly scoped permissions to applications or automation
Avoid over-assigning built-in roles

However, Azure Communication Services behaves differently than many other Azure resource providers, and this often leads to confusion during role design.

🔐 The Issue

A customer attempted to create and assign a custom IAM role for an Azure Communication Services resource.

The goal was to grant limited permissions for managing or interacting with ACS without assigning a broad built-in role.

Despite the role being created successfully in Azure, it did not function as expected when assigned to users or service principals.

🔍 The Root Cause

Custom IAM roles are not supported for Azure Communication Services.

While Azure allows you to define custom roles at a subscription or resource scope, ACS does not honor custom role definitions, even if the permissions appear valid.

This means:

Custom roles may assign successfully
Role assignments appear correct in the portal
Access attempts still fail at runtime

This is not a configuration issue or a customer mistake. It is a platform limitation.

✅ The Resolution

To manage or interact with Azure Communication Services, you must use a predefined built-in role.

✔ Required Built-In Role

Communication and Email Service Owner

This role provides the permissions required to:

Manage ACS resources
Access keys and endpoints
Configure calling, SMS, email, and related features

At this time, there is no supported alternative that allows partial or custom permission sets for ACS.

⚠️ Important Notes About ACS RBAC

Here are a few important behaviors to be aware of when working with ACS permissions:

ACS does not support custom RBAC roles
DataActions are not exposed or configurable for ACS
Least-privilege designs are limited to built-in roles only
Role assignments must be validated using real API calls, not just portal visibility

This can be surprising if you’re used to other Azure services that fully support custom role definitions.

🧭 Design Guidance

If you need separation of duties or reduced blast radius, consider these alternatives:

Use separate ACS resources per environment or workload
Assign the built-in role only to dedicated service principals
Scope access tightly at the resource level, not subscription-wide
Use Azure Monitor and diagnostics to audit usage instead of restricting permissions

While not ideal, these approaches align with the current ACS security model.

🧠 Final Thoughts

Azure Communication Services has a powerful feature set, but its RBAC model is intentionally constrained today. Attempting to use custom IAM roles will lead to confusing access failures that are difficult to troubleshoot unless this limitation is known upfront.

If you encounter authorization issues with ACS:

Verify the role is Communication and Email Service Owner
Do not rely on custom role definitions
Validate access using real SDK or REST calls

If this behavior changes in the future, it will require a platform update from the ACS product group.

Outbound Calling with a Lightweight Console Tool - Azure Communication Services (ACS)

December 5, 2025

:bulb: Introduction

Azure Communication Services (ACS) provides powerful APIs for building telephony workflows — from simple outbound calls to fully automated IVR systems. But when you’re validating a new ACS deployment, troubleshooting access keys, or checking if a purchased phone number is correctly configured, you often need something much simpler:

A tool that can place a real outbound PSTN call on demand.

To streamline that process, I created a small C# console application that uses the ACS Call Automation API to place a call from an ACS-owned phone number to any valid target phone number. This post provides an overview of how the tool works, why it exists, and what to know when using ACS programmatically for calling.

:telephone_receiver: Why Build a Tool for ACS Outbound Calling?

When helping customers troubleshoot ACS calling issues, one of the most common challenges is isolating whether a problem is:

a misconfigured ACS resource
an invalid or inactive phone number
a missing callback endpoint
incorrect API usage
version mismatch in the SDK
firewall or networking issues blocking callbacks

A lightweight console caller removes all other moving parts so you can focus on validating:

✔ The ACS resource is reachable
✔ The access key is correct
✔ The Call Automation API accepts your request
✔ The ACS phone number is capable of PSTN outbound traffic
✔ Call events are flowing to the configured callback URL (if applicable)

It’s the ACS equivalent of “ping” for telephony — quick, simple, and reliable.

:mag: Understanding ACS CallAutomation Behavior

One important part of building this tool was understanding how ACS behaves across SDK versions. The CreateCall API:

Does not provide real-time call status
Does not block while waiting for the call to be answered
Does not include a call state object in the response

Instead, ACS gives you:

CallConnectionId

These identifiers uniquely represent the server-side call object and can be used:

to correlate logs
to track call events
inside Call Automation actions (hang up, play audio, etc.)
for troubleshooting in Azure Monitor or diagnostics views

Any real-time events — like CallConnected or CallDisconnected — are always delivered asynchronously to your configured callback URL, not through the tool itself.

This design is intentional and foundational to ACS at scale.

:gear: What the Tool Does

The console tool performs the following steps:

Accepts two command-line parameters:
- ACS source phone number
- Target PSTN phone number
Creates a CallAutomationClient using your ACS connection string.
Builds a CallInvite using the source and target numbers.
Sends a CreateCall request to the ACS endpoint.
Prints the CallConnectionId and ServerCallId returned by ACS.
Exits immediately — because ACS handles all call progression asynchronously.

What It’s Ideal For

Verifying ACS outbound calling works
Testing newly purchased phone numbers
Troubleshooting 401/403/BadRequest failures
Ensuring your ACS access key is correct
Validating resource deployment issues
Proving connectivity before building more complex automation

What It’s Not Designed For

Monitoring call state in real-time
Handling media, DTMF, or recording
Complete IVR or call-flow logic
Replacing full Call Automation workflows

This tool is intentionally focused and minimal.

:electric_plug: Callback URL Requirements

ACS requires a public callback endpoint for call events. This is used to deliver:

CallConnected
ParticipantsUpdated
CallDisconnected
Media events

Even if you aren’t using callbacks yet, the callbackUri must be:

a valid URI
publicly accessible
HTTPS recommended

If you’re just validating call initiation, you can temporarily specify a placeholder endpoint.
For full automation, you’ll want a real listener (Azure Functions, App Service, Ngrok, etc.).

:warning: Common Issues When Testing ACS Calls

Here are some real-world issues I’ve seen while helping customers:

❌ 401 Unauthorized

Typically caused by:

Incorrect access key
Expired regenerated key not updated in code
Wrong endpoint region

❌ 403 Forbidden

Usually means:

The ACS phone number is not enabled for PSTN
Phone number is missing from the resource
You’re calling a restricted destination

❌ Call never arrives

Often due to:

Incorrect E.164 formatting (ACS is strict)
Callback endpoint unreachable
Resource not provisioned for PSTN

The console tool makes diagnosing these issues much easier — if an outbound call fails, the response error tells you why.

:floppy_disk: Download the Tool

You can download the tool or review the source code here:

🔗 Download the ACS Outbound Calling Tool Executable
https://files.blakedrumm.com/ACS-CallExample-Console-Executable.zip

🔗 Source Code Repository
https://github.com/blakedrumm/ACS-CallExample-Console

Feel free to fork it, extend it, or incorporate it into your own automation workflows.

:speech_balloon: Final Thoughts

Azure Communication Services is extremely powerful once you understand its event-driven architecture. Tools like this console caller make it easier to validate your environment, test scenarios quickly, and remove uncertainty during deployments.

If you’re working with ACS and need help diagnosing issues or extending call workflows, feel free to reach out — I’m always happy to help troubleshoot or share lessons learned.

Cleaning Orphaned WindowsPatchExtension Status Files - Azure Update Manager

June 18, 2025

:bulb: Introduction

While helping customers troubleshoot patching issues with Azure Update Manager, I ran into a recurring problem—leftover .status files from the WindowsPatchExtension.

These orphaned files can confuse log analysis and clutter VMs over time. This blog post explains how they get there, why they matter, and how to clean them up using a simple PowerShell script.

:mag: The Problem — Orphaned `.status` Files

When the WindowsPatchExtension runs, it generates .settings and .status files in the following directories:

Settings Path
C:\Packages\Plugins\Microsoft.CPlat.Core.WindowsPatchExtension\<version>\RuntimeSettings
Status Path
C:\Packages\Plugins\Microsoft.CPlat.Core.WindowsPatchExtension\<version>\status

Under normal conditions, these files are created and removed together. But if patching is interrupted, retried, or a new version of the extension is installed, some .status files may remain—even if they no longer correspond to a .settings file.

:warning: Real-World Scenario

In a recent case, the customer deployed VMs from a custom image that already had the WindowsPatchExtension installed. The image unknowingly included stale .status files, which then propagated to every new VM built from that image.

This mismatch between .settings and .status files caused confusion during patch runs, where the extension logs no longer lined up with the expected state.

:thinking: Why This Matters

These orphaned .status files:

❌ Do not represent the current patch status
🤔 Can cause confusion when checking plugin behavior
🧹 Accumulate over time and clutter the file system

While they don’t break the extension’s functionality, they make troubleshooting harder—especially when using automation or analyzing logs across many VMs.

:wrench: The Fix — Compare and Clean with PowerShell

To resolve this, I wrote a PowerShell script that compares .status and .settings file indexes and deletes any .status files that don’t match.

What It Does

Finds the highest-index .settings file in the RuntimeSettings folder
Deletes any .status files with a higher index

This keeps only the relevant .status files and ensures consistency.

:floppy_disk: How to Use the Script

🔗 Script Link: PowerShell Gist - Cleanup Status Files

Disclaimer: Always test in a development environment before applying to production.

1. Save the Script Locally

Save the content of the Gist above as a .ps1 file on your VM:

Remove-UpdateManagerOldStatusFiles.ps1

2. Run PowerShell as Administrator

Because the script accesses protected directories, you must run it with admin rights.

Right-click PowerShell → Run as Administrator

3. Execute the Script

Navigate to the script location and run:

.\Remove-UpdateManagerOldStatusFiles.ps1

The script will:

Detect the latest version of the WindowsPatchExtension
Find the highest .settings file
Remove .status files with a higher index

You’ll get output confirming what (if anything) was removed.

Have you encountered similar cleanup scenarios with the WindowsPatchExtension or Update Manager?
Did you solve it a different way?

Let me know! I’m always interested in how others handle extension state management at scale.

Setup OpenWeb UI AI Webpage - Azure Container Apps

September 7, 2024

:bulb: Introduction

Deploying OpenWeb UI on Azure allows you to host an interactive AI webpage. However, due to OpenWeb UI’s architecture—where both the frontend and backend are housed in a single container—scalability is limited. This guide will walk you through setting up OpenWeb UI using Azure Container Apps, discuss its limitations regarding scaling, and compare this solution to using Kubernetes.

:hammer: Prerequisites

Before starting, make sure you have the following:

Azure Account: An active Azure subscription with permissions to create new resources.
OpenWeb UI Container Image: The container image from GitHub Container Registry (ghcr.io/open-webui/open-webui:main).
OpenAI API Key: [Optional] In order to utilize OpenAI in OpenWeb UI, you will need an OpenAI API Key. https://platform.openai.com/api-keys

:rocket: Setting Up OpenWeb UI with Azure Container Apps

Step-by-Step Guide (Portal)

1. Access the Azure Portal

2. Create a Resource Group

Search for Resource groups and create a new one named OpenWebUI-ContainerApp-RG.
Select a region like East US be sure you are consistent with the region through this guide.

3. Create a Storage Account

Search for Storage accounts, create a new one (For example: openwebuistorageaccount (lowercase required)) in the OpenWebUI-ContainerApp-RG resource group.
- During creation of the Storage account there are configuration changes required:
  - Basics tab
    1. Primary service select Azure Files.
    2. Performance you can set Standard unless you have a specific need for Premium.
    3. Redundancy you can set Locally-redundant storage (LRS) unless you have a specific need for the other options.
  - Networking tab
    1. Network access select Enable public access from all networks (default option) unless you have a specific need for the other options (the other options require additional configuration which will not be covered in this guide).
  - Other tabs
    1. Leave the defaults and create the storage account.
- After the storage account is created there are additional steps required:
  1. Search for Storage accounts. Locate the storage account (openwebuistorageaccount) and select it.
  2. Expand Data storage and go to the File shares blade. Add a new file share.
    1. Name set to ai-storage-file-share.
    2. For Access tier select Transaction optimized.
    3. On the Backup tab I disable backup, as I do not want to incur any additional charges. But if you can afford it, keep it enabled to keep your files safe in the event of data loss.
    4. Create the file share.

4. Create a Container App

Search for Container Apps, create a new container app.

Basics tab
1. Container app name set the name to ai-openwebcontainer.
2. Deployment source set to Container image.
3. Region set East US.
4. Container Apps Environment create a new container apps environment. There is additional configuration needed for the container app environment:
  1. Set the Environment Name to OpenWebContainerEnvironment.
  2. Leave the rest of the options as the defaults.
  3. Create the Container Apps Environment.
Proceed to Next: Container.
Container tab
1. Image source set to Docker Hub or other registries
2. Image type set to Public
3. Registry login server set to ghcr.io
4. Image and tag set to open-webui/open-webui:main
5. CPU and Memory set to what you are comfortable with, I set to 2 CPU cores, 4 Gi memory but this can be lower if needed.
6. Leave the rest of the options as the defaults (we will configure the environmental variables in a few steps).
Proceed to Next: Bindings.
Bindings tab
1. Nothing needs to be modified on this tab.
Proceed to Next: Ingress.
Ingress tab
1. Toggle Ingress to Enabled
2. Ingress traffic set to Accept traffic from anywhere: Applies if 'internal' setting is set to false on the Container Apps environment
3. Ingress Type set to HTTP
4. Target port set to 8080
5. Leave the rest of the options as the defaults.
Select Review + create to create the Container App.

5. Gather data from Storage Accounts

Search for Storage accounts, select the storage account we created openwebuistorageaccount.
Expand Security + networking and select Access keys.
Click show on one of the 2 keys listed, copy one for us to use in the next step.

6. Link Azure Files to Container Apps Environment

Search for Container Apps Environments.
Expand Settings, then select Azure Files, we are going to Add a new file share to the container apps environment.
1. Name set a name (openwebcontainerfileshare) for the file share in the container apps environment.
  1. Storage account name set to openwebuistorageaccount
  2. Storage account key set to the access key we copied in the previous step.
  3. File Share set to ai-storage-file-share
  4. Access mode set to Read/Write

Search for Container Apps, select the container app we created ai-openwebcontainer
Expand Application, select Containers. Select Edit and deploy.
- First, you will need to select the Volumes tab.
- Volumes tab
  1. Select + Add
  2. Volume type set to Azure file volume
  3. Name set to whatever name you would like (I used ai-openweb-volume).
  4. File share select the file share we created openwebcontainerfileshare
  5. Mount options set to nobrl more information
- Now select the Container tab.
- Container tab
  1. Name / suffix set the name of the revision to something you will recognize. (I used live)
  2. Click on the container image ai-openwebcontainer shown in the Container Image table
  3. The Edit a container menu will open.
    1. In the Basics tab you can add your Environment variables at the bottom of the tab. Here are all the environmental variables that OpenWeb UI Supports: https://docs.openwebui.com/getting-started/env-configuration/
      - Add a new environmental variable (keep in mind this method will expose your API key in plain text. If you want security you need to save it as a secret):
        
        Name: OPENAI_API_KEY
        
        Source: Manual entry
        
        Value: <OpenAI-API-Key> https://platform.openai.com/api-keys
    2. Select the Volume mounts tab.
    3. Select the dropdown under volume name and select the Azure file volume that we created in the Volumes tab.
      - Volume name: ai-openweb-volume
      - Mount path: /app/backend/data
      - Sub path (optional): Leave this empty
    4. Click save
- Lastly, you will need to select the Scale tab.
- Scale tab
  1. Min replicas set to 1 (If you want the instance to spin up on demand and deallocate when not in use, set this to 0 instead. Personally, I prefer the application to remain running, so I don’t have to wait for Azure Container Apps to activate the container.)
  2. Max replicas set to 1 (the max cannot be more than 1 due to design of Docker container for OpenWeb UI)
- Select Create to create the revision.

8. Access the Application:

In the Container App view, expand Application and select Revisions and replicas.
Click the Active revision that ends with live (or whatever you configured your revision name to)
In the Revision details menu select the Revision URL this is the published URL for your container app revision. (the main URL is in the Overview blade of your container app. Its called the Application Url.)

:gear: Scaling Limitations of OpenWeb UI

Due to OpenWeb UI’s architecture, where both frontend and backend are housed in the same container, scaling the application horizontally (adding more replicas) is problematic. Here’s why:

Single Container Design: The combined frontend and backend make it difficult to scale individual components like you would in a decoupled architecture.
Session Handling: Without advanced session management, multiple instances of the backend can cause inconsistent behavior, as there’s no mechanism to manage session persistence across replicas.
Best Fit: For now, deploying OpenWeb UI with 1 replica in Azure Container Apps is the most efficient way to ensure stability.

Possible Solutions

Decoupling Services: If you want to scale, consider splitting the frontend and backend into separate containers and using Kubernetes for scaling them independently.
Kubernetes Deployment: Kubernetes can help manage scaling by separating components, introducing load balancing, and handling session persistence. However, this adds complexity and cost.

:moneybag: Cost-Effective Hosting: Container Apps vs. Kubernetes

When comparing Azure Container Apps and Azure Kubernetes Service (AKS), here’s what you need to know about cost-effectiveness:

Azure Container Apps

Serverless Pricing: You only pay for the CPU and memory your app uses. It’s cost-efficient for small to medium workloads and apps with unpredictable traffic.
Low Management Overhead: Azure manages the underlying infrastructure, so there’s no need to worry about managing virtual machines (VMs), which reduces operational costs.
Best Use: Ideal for apps like OpenWeb UI that don’t require complex scaling and benefit from Azure’s autoscaling features.

Azure Kubernetes Service (AKS)

VM-Based Pricing: You pay for VMs even when your app is idle. Scaling adds complexity since you need to manage pods, nodes, and networking.
Scalability: While AKS offers more flexibility in scaling, the cost is higher, especially for apps that don’t need continuous scalability.
Best Use: Suitable for large-scale applications that need fine-grained control over scaling and architecture.

Conclusion: Azure Container Apps is the more cost-effective solution for hosting OpenWeb UI due to its serverless pricing and minimal management requirements. AKS is better suited for complex, highly scalable applications, but it comes with a higher cost and management overhead.

:mag: Conclusion

In this guide, we deployed OpenWeb UI using Azure Container Apps and explored its scalability limitations. For most users, sticking with a single-instance deployment in Container Apps is the most cost-effective and stable approach. If scaling is essential, consider moving to the Kubernetes-based solution where the frontend and backend can be decoupled.

:thought_balloon: Feedback

Have you tried deploying OpenWeb UI on Azure? What’s your experience with its scaling capabilities? Share your insights in the comments below.

PowerShell: SCOM's Best Friend - Insights from Scomathon 2024

June 18, 2024

:book: Introduction

At Scomathon 2024, I had the privilege of presenting on a topic that is near and dear to my heart: “PowerShell: SCOM’s Best Friend.” In this blog post, I’ll be sharing the key takeaways from my presentation, along with some practical PowerShell scripts that can significantly enhance your SCOM environment. Whether you’re just starting with SCOM or are a seasoned professional, there’s something here for everyone.

For those who couldn’t attend, you can watch my presentation here: https://www.youtube.com/watch?v=ewcJpC0iFqY

:classical_building: Why PowerShell for SCOM?

PowerShell is a powerful scripting language that can automate and simplify many tasks within System Center Operations Manager (SCOM). From managing alerts and notifications to performing regular maintenance and configuration tasks, PowerShell can be your go-to tool for enhancing the efficiency and functionality of your SCOM environment.

:page_with_curl: Key Benefits

Automation: Schedule and automate repetitive tasks to save time and reduce errors.
Flexibility: Modify and extend scripts to meet your specific needs.
Integration: Seamlessly integrate with other Windows services and applications.

:hammer_and_wrench: Tools and Techniques

During the presentation, I covered a variety of tools and techniques that are particularly useful for SCOM administrators. Below are the links to the tools discussed. Please note that these scripts require compliance with the MIT license.

System Center Operations Manager Data Warehouse Grooming Tool

Read more

This tool analyzes your Data Warehouse Grooming to see what is being stored and how much of your Data Warehouse is filled with the related data. It is available in multiple formats: MSI, EXE, or source PS1.

SCOM Data Collector

Read more

With the SCOM Data Collector, you can collect useful data to analyze and troubleshoot your SCOM Environment. It’s an essential tool for any SCOM Admin who wants a holistic view of their configuration and setup.

Reconfigure System Center Operations Manager for Database Move Tool

Read more

This application aids in updating the configuration when migrating SCOM SQL Instances.

Remove Data from the SCOM Database Instantly - The PowerShell Way!

Read more

This guide allows for quick removal of specific data from the SCOM Database without needing SQL Server Management Studio.

SCOM Certificate Checker Script

Read more

Check the validity of SCOM Certificates on an Agent, Gateway, or Management Server.

Enforce TLS 1.2 & TLS 1.3 in SCOM - The PowerShell Way!

Read more

Configure your SCOM Environment for a more secure configuration with TLS 1.2 and TLS 1.3.

:star: Best Practices

Here are some best practices to keep in mind when using PowerShell with SCOM:

Test Scripts: Always test scripts in a non-production environment before deploying them live.
Error Handling: Implement robust error handling to manage exceptions and failures gracefully.
Documentation: Keep your scripts well-documented to ensure they are understandable and maintainable.

:rocket: Conclusion

PowerShell truly is SCOM’s best friend, offering unparalleled flexibility and control. By leveraging the tools and techniques discussed in my Scomathon 2024 presentation, you can enhance your SCOM environment, automate tedious tasks, and focus on what truly matters—optimizing your IT operations.

:link: Additional Resources

Leave some feedback if this helped you! :v:

Resolving Azure Maintenance Configuration Authorization Errors - Azure Update Manager

May 15, 2024

:book: Introduction

While working with Azure, I encountered an authorization error while attempting to create a maintenance configuration. This post discusses the error, its implications, and how to resolve it using a custom role.

:x: Error text

While creating a maintenance configuration, I encountered the following error:

The client ‘[email protected]’ with object id ‘8b4fd821-05b6-4938-9949-18782893c5ed’ does not have authorization to perform action ‘Microsoft.Resources/deployments/validate/action’ over scope ‘/subscriptions/a1b2c3d4-5e6f-7g8h-9i0j-1k2l3m4n5o6p/resourceGroups/DeleteThisResourceGroup/providers/Microsoft.Resources/deployments/TestConfiguration_1715820611290’ or the scope is invalid. If access was recently granted, please refresh your credentials. (Code: AuthorizationFailed)

This error message indicates a lack of permissions necessary to perform specific actions within the Azure portal.

:gear: Solution

To address this error we will need to create a custom role that specifically includes the necessary permissions for managing maintenance configurations and validating deployments.

:arrow_down: How to get it

GitHub Gist: Create-MaintenanceConfigManagerRole.ps1

I developed a PowerShell script that creates a custom role to grant a user the necessary permissions for managing Maintenance Configurations within a specific Resource Group.

# Create-MaintenanceConfigManagerRole.ps1
# Script to create a custom role for managing Azure Maintenance Configurations
# Author: Blake Drumm ([email protected])
# Website: https://blakedrumm.com/blog/resolve-azure-maintenance-configuration-error/
# Date created: May 15th, 2024

# Define custom variables
$subscriptionId = "a1b2c3d4-5e6f-7g8h-9i0j-1k2l3m4n5o6p"
$resourceGroupName = "ResourceGroupName"
$customRoleName = "Maintenance Configuration Manager - Resource Group: $resourceGroupName"
$userPrincipalName = "[email protected]"

# Define the custom role as a JSON string with the subscription ID and resource group name directly replaced
$customRoleJson = @"
{
    "Name": "$customRoleName",
    "IsCustom": true,
    "Description": "Allows management of maintenance configurations, validate and write deployments, read and write virtual machines, and write configuration assignments.",
    "Actions": [
        "Microsoft.Maintenance/maintenanceConfigurations/read",
        "Microsoft.Maintenance/maintenanceConfigurations/write",
        "Microsoft.Maintenance/maintenanceConfigurations/delete",
        "Microsoft.Resources/deployments/validate/action",
        "Microsoft.Resources/deployments/write",
        "Microsoft.Maintenance/configurationAssignments/write",
        "Microsoft.Compute/virtualMachines/read",
        "Microsoft.Compute/virtualMachines/write"
    ],
    "NotActions": [],
    "AssignableScopes": [
        "/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName"
    ]
}
"@

# Convert the JSON string to a PowerShell object
$customRole = $customRoleJson | ConvertFrom-Json

# Create the custom role
New-AzRoleDefinition -Role $customRole

# Define the scope
$scope = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName"

# Assign the custom role to the user
New-AzRoleAssignment -RoleDefinitionName $customRoleName -UserPrincipalName $userPrincipalName -Scope $scope

<#
    Copyright (c) Microsoft Corporation. MIT License
    
    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:
    
    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.
    
    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
    WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
    CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#>

:bulb: How to use it

To implement this solution:

Modify the script with your specific details, including subscription ID, resource group name, and user principal name.
Execute the script in your PowerShell environment to create the custom role and assign it.
Verify the permissions are correctly applied by attempting to create a Maintenance Configuration again.

:speech_balloon: Conclusion

By understanding and addressing Azure permission errors through custom role creation, administrators can ensure smoother operation, enhance security, and reduce manual workload. Feel free to modify the script to fit your specific needs and share your feedback on how it worked for you! :v:

Automated Azure Role Assignment Reports via Email - Azure Automation

April 23, 2024

:book: Introduction

Azure cloud services management often requires monitoring and auditing user roles and group memberships. This script automates the generation and email distribution of detailed Azure subscription role assignments. It utilizes managed identity for Azure login, fetches role assignments, and compiles them into a comprehensive report sent via email.

:arrow_down: How to get it

You can download the script from the following links:

Get-AzRoleAssignmentReport.ps1 :arrow_left: Direct Download Link
Personal File Server - Get-AzRoleAssignmentReport.ps1 :arrow_left: Alternative Download Link
Personal File Server - Get-AzRoleAssignmentReport.txt :arrow_left: Text Format Alternative Download Link

:classical_building: Argument List

Parameter	Type	Description
EmailUsername	String	The username used to authenticate with the SMTP server.
EmailPassword	SecureString	The secure password used for SMTP authentication.
From	String	The email address from which the report will be sent.
To	Array	Array of recipient email addresses to whom the report will be sent.
Cc	Array	Array of CC recipient email addresses.
Subject	String	The subject line of the email.
Body	String	The body text of the email, describing the contents of the report. Can be HTML or plain text.
SMTPServer	String	The SMTP server used for sending the email.
SubscriptionIds	Array	Array of Azure subscription IDs to be included in the report.
WhatIf	Switch	A switch to simulate the script execution for testing purposes without performing any actual operations.

:key: Configuring Permissions for Managed Identity

To enable the PowerShell script to retrieve detailed user information, such as ObjectType and DisplayName from Azure Active Directory, the UserManagedIdentity needs the “Directory Readers” permission. This role-based access control (RBAC) is assigned at the Microsoft Entra ID level (formerly known as Azure Active Directory), not at the subscription level. Follow these steps to assign the correct permissions:

Identify the Object ID:
- System Assigned Identity
  - Navigate to your Azure Automation Account -> Identity, select the System assigned tab. Copy the Object ID of the System Assigned identity.
- User Assigned Identity
  - Navigate to your Azure Automation Account -> Identity, select the User assigned tab. Click on the name of the user assigned identity you want to gather the id from. Copy the Object ID of the User Assigned identity.
Set Azure role assignments
- Select Azure role assignments
- Select Add role assignment
- Set the scope to: Subscription
- Select the subscription.
- Set the role to (use what your company allows here, this is just what I used in my testing): Reader
Assign the Role:
- Open Microsoft Entra Id -> Roles and Administrators.
  Azure Portal - Roles and Administrators
- In the roles list, find and click on Directory Readers.
- Click + Add Assignments to start the role assignment process.
Add Managed Identity to Role:
- In the assignment interface, you might not see app registrations or managed identities by default.
- Paste the Object ID (from step 1) into the search field. This should display the name and ID of your Azure Automation Account.
- Select your account and confirm the assignment.
Verify Permissions:
- Once the “Directory Readers” permission is assigned, the script will be able to pull the Object Type and DisplayName along with other outputs from Get-AzRoleAssignment.

This configuration is essential for the script to function correctly and securely access the necessary Azure AD data!

:page_with_curl: How to use it

In order to utilize this script in your Automation Runbook, you will need to set an encrypted variable inside of the Automation Account. This will be so we can pass the EmailPassword variable securely to the script. The script has the ability to gather this password automatically if you perform the following steps.

Go to Automation Accounts -> Select the Automation Account -> Variables.
Click + Add a variable
New Variable
- Name: EmailPassword
- Description: This is the password for the Email Account used in SMTP for an Azure Automation Runbook.
- Type: String
- Value: <YourPassword>
- Encrypted: Yes

Leave some feedback if this helped you! :v:

Toggle VM Power by Tag - Azure Automation

March 31, 2024

:bulb: Introduction

Managing the power state of Azure VMs can be a cumbersome task, especially when dealing with a large number of instances. In scenarios where VMs need to be started or stopped based on certain criteria—like environment tags—it’s crucial to have an automated solution. This guide introduces a PowerShell script designed to toggle the power state of Azure Virtual Machines (VMs) based on their tags, offering a streamlined approach for bulk operations.

:wrench: The Script

The PowerShell script enables you to start or stop Azure VMs in bulk by specifying their subscription IDs, a tag name, and a tag value. It also supports a -WhatIf parameter for dry runs, allowing you to preview the changes without applying them.

:label: Preparing Your VMs

To ensure your Azure VMs respond correctly to the script, you must tag them with specific key-value pairs. The script identifies VMs to start or stop based on these tags. Here’s how to set up your VM tags:

Tagging VMs

Access the Azure Portal: Navigate to the Azure Portal and find the Virtual Machines section.
Select a VM: Choose the VM you want to manage with the script.
Add Tags:
- Click on the Tags section in the VM’s menu.
- Add a new tag with the TagName you plan to use in the script. For example, if your script uses Environment as the tag name, you might set the tag value to Development for all development VMs.

Example Tag Configuration

Tag Name: Environment
Tag Value: Development

With this configuration, if you run the script with Environment as the $TagName and Development as the $TagValue, it will target all VMs tagged as part of your development environment.

This setup allows for precise control over which VMs are affected by the script, enabling you to manage VM power states efficiently across different environments or projects.

:arrow_down: How to get it

GitHub Gist: Toggle-VMPowerByTag.ps1

param
(
	[Parameter(Mandatory = $true)]
	[String[]]$SubscriptionIds,
	[Parameter(Mandatory = $true)]
	[String]$TagName,
	[Parameter(Mandatory = $true)]
	[String]$TagValue,
	[Parameter(Mandatory = $true)]
	[Boolean]$PowerState, # true for start, false for stop
	[Boolean]$WhatIf # test how the script will work, without making any changes to your environment
)

# Script to toggle Azure VM power states based on tags
# Requires PowerShell 7.2 or higher
# Author: Blake Drumm ([email protected])
# Website: https://blakedrumm.com/
# Date created: March 31st, 2024
# Date modified: April 16th, 2024

# Ensures you do not inherit an AzContext in your runbook
$disableAzContextAutosave = Disable-AzContextAutosave -Scope Process

# Connect using a Managed Service Identity
try
{
	# Connect to Azure
	$AzConnection = Connect-AzAccount -Identity -AccountId cb02e61e-d392-48d7-936a-9b44bbf5f312 -ErrorAction Stop
	Write-Output "$((Get-Date).ToLocalTime()) - Connected to Azure (Account Id: $($AzConnection.Context.Account.Id))"
}
catch
{
	# Log the error and exit
	Write-Output "$((Get-Date).ToLocalTime()) - Connection failed: $_"
	exit 1
}

$SubscriptionIds | ForEach-Object {
	# Initialize variables
	$PS = $PowerState
	$SubId = $_
	try
	{
		# Set the subscription context
		Write-Output "$((Get-Date).ToLocalTime()) - Subscription Id: $SubId"
		$setAzContext = Set-AzContext -SubscriptionId $SubId -ErrorAction Stop
	}
	catch
	{
		# Log the error and exit
		Write-Output "$((Get-Date).ToLocalTime()) - Encountered error: $_"
		exit 1
	}
	# Fetch VMs with the specified tag
	$vms = Get-AzResource -ResourceType Microsoft.Compute/virtualMachines -TagName $TagName -TagValue $TagValue
	# Start or stop the VMs based on the desired state
	$vms | ForEach-Object -Parallel {
		$vm = $_
		$WhatIf = $using:WhatIf
		$Power = $using:PS
		# Fetch VM status
		$x = 0
		do
		{
			$x++
			try
			{
				$vmStatus = Get-AzVM -ResourceGroupName $vm.ResourceGroupName -Name $vm.Name -Status -ErrorAction Stop
				# This will only be set to 2 if the command above is able to successfully fetch the VM status
				$x = 2
			}
			catch
			{
				Write-Output "$((Get-Date).ToLocalTime()) - Encountered error: (VM Name: $($vm.Name)) $_"
			}
		}
		until ($x -eq 2)
		
		
		# Extract the power state of the VM
		$vmPowerState = $vmStatus.Statuses | Where-Object { $_.Code -like 'PowerState/*' } | Select-Object -ExpandProperty Code
		
		# Start or stop VM based on the desired state		
		if (($Power -eq $true) -and ($vmPowerState -ne 'PowerState/running'))
		{
			if ($WhatIf)
			{
				Write-Output "$((Get-Date).ToLocalTime()) - What if: Starting $($vm.Name)"
			}
			else
			{
				# Start the VM
				Start-AzVM -Name $vm.Name -ResourceGroupName $vm.ResourceGroupName -Verbose
				Write-Output "$((Get-Date).ToLocalTime()) - Starting $($vm.Name)"
			}
		}
		elseif ($Power -eq $false -and $vmPowerState -notmatch 'PowerState/(deallocated|stopped)')
		{
			if ($WhatIf)
			{
				Write-Output "$((Get-Date).ToLocalTime()) - What if: Stopping $($vm.Name)"
			}
			else
			{
				# Stop the VM
				Stop-AzVM -Name $vm.Name -ResourceGroupName $vm.ResourceGroupName -Force -Verbose
				Write-Output "$((Get-Date).ToLocalTime()) - Stopping $($vm.Name)"
			}
		}
		else
		{
			# Log the VM is already in the desired state
			Write-Output "$((Get-Date).ToLocalTime()) - VM $($vm.Name) is already in the desired state (PowerState: $Power). (Current VM PowerState: $vmPowerState)"
		}
	} -ThrottleLimit 4 # Example throttle limit
}
Write-Output "$((Get-Date).ToLocalTime()) - Script completed!"

<#
    Copyright (c) Microsoft Corporation. MIT License
    
    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:
    
    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.
    
    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
    WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
    CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#>

:gear: How It Works

The script first disables AzContext autosave to ensure a clean environment. Then, it attempts to connect to Azure using a Managed Service Identity. For each subscription ID provided, it sets the Azure context and fetches VMs that match the specified tag name and value. Depending on the desired power state ($PowerState), it starts or stops the fetched VMs, respecting the -WhatIf parameter to only simulate actions if specified.

:key: Key Features

Tag-Based Filtering: Selectively start or stop VMs based on tags, allowing for flexible management of different environments or deployment stages.
Bulk Operations: Perform actions on multiple VMs across different subscriptions, saving time and effort.
Safe Testing with -WhatIf: Preview the impact of the script without making actual changes to your VMs, enhancing control and safety.

:mag: Use Cases

This script is particularly useful for scenarios like:

Cost Optimization: Automatically stop non-essential VMs outside of business hours.
Environment Management: Quickly start or stop all VMs within a specific environment (e.g., development, testing) based on tagging.

:thought_balloon: Feedback

Have you tried using the script in your Azure environment? Any suggestions for improvement or additional features you’d find useful? Share your thoughts and experiences to help enhance this script for everyone.

Assessment Failures - Azure Update Manager

February 27, 2024

:bulb: Introduction

I had a case today where Azure Update Manager was showing errors regarding checks for assessments, upon checking the Windows Updates installed on the machine we noticed the updates were successfully installed.

:x: Error text

1 errors reported. The latest 100 errors are shared in details. To view all errors, review this log file on the machine: [C:\ProgramData\GuestConfig\extension_logs\Microsoft.Software Update Management.WindowsOsUpdateExtension\1.telemetryLogs]. Failed to apply patch installation. Reason: [Failed to assess the machine for available updates: Activityld = [4242741a-0959-4b6c-886b-d7385dfc62f1], Operation=[Patching], Reason:[Windows update API threw an exception while assessing the machine for available updates. HResult: 0x80244022.. For information on diagnosing this error, see: https://aka.ms/TroubleshootVMGuestPatching.].].

:mag: Cause

Details on the above exception:

Hexadecimal Error Code	Decimal Error Code	Symbolic Name	Error Description	Header
0x80244022	-2145107934	WU_E_PT_HTTP_STATUS_SERVICE_UNAVAIL	Same as HTTP status 503 - the service is temporarily overloaded.	wuerror.h

The issue above in my customers circumstance was due to the following registry key being present:
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU

Specifically this key was changed to 1 which needed to be 0 in order to utilize Microsoft Updates:

Entry name	Data type	Values
UseWUServer	Reg_DWORD	1 = The computer gets its updates from a WSUS server.
		0 = The computer gets its updates from Microsoft Update.
		The WUServer value is not respected unless this key is set.

More information on the registry keys for Automatic Updates and WSUS:
https://learn.microsoft.com/windows/deployment/update/waas-wu-settings#configuring-automatic-updates-by-editing-the-registry
https://github.com/vFense/vFenseAgent-win/wiki/Registry-keys-for-configuring-Automatic-Updates-&-WSUS

:wrench: Resolution

Modifying the registry key above (UseWUServer) registry key to 0 instead of 1 allowed us to successfully utilize Azure Update Manager.

:computer: PowerShell Script

# The following code will allow you to set the UseWUServer registry key to 0
# Which allows you to utilize Microsoft Updates instead of Software Update Services (WSUS)
Set-ItemProperty -Name UseWUServer -Value 0 -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU"

Leave some feedback if this helped you! :v:

Configure periodic checking for missing system updates on azure virtual machines - Azure Update Manager Policy

February 14, 2024

:bulb: Introduction

I had a case today were my customer was experiencing an error with the built-in Azure Update Manager Policy named: Configure periodic checking for missing system updates on azure virtual machines. I decided it would be a good idea to script out how to fix this so it can be automatically fixed for anyone who needs this fix in the future.

:x: Error text

Failed to remediate resource: '/subscriptions/a84b857e-2f4b-4b79-8b92-c6f1093b6d4f/resourceGroups/rg-app-production/providers/Microsoft.Compute/virtualMachines/vmProdServer01'. The 'PUT' request failed with status code: 'Forbidden'. Inner Error: 'The client 'f4e3d2c1-b6a5-4f09-b8ed-9c2a3b1c9e4d' with object id 'f4e3d2c1-b6a5-4f09-b8ed-9c2a3b1c9e4d' has permission to perform action 'Microsoft.Compute/virtualMachines/write' on scope '/subscriptions/a84b857e-2f4b-4b79-8b92-c6f1093b6d4f/resourceGroups/rg-app-production/providers/Microsoft.Compute/virtualMachines/vmProdServer01'; however, it does not have permission to perform action(s) 'Microsoft.Compute/diskEncryptionSets/read' on the linked scope(s) '/subscriptions/a84b857e-2f4b-4b79-8b92-c6f1093b6d4f/resourceGroups/rg-encryption-keys/providers/Microsoft.Compute/diskEncryptionSets/desProdKeySet' (respectively) or the linked scope(s) are invalid.', Correlation Id: '9e2d4b3c-a1b2-c3d4-e5f6-7g8h9i0j1k2l'.

:mag: Cause

Built-in policy for Configure periodic checking for missing system updates on azure virtual machines does not utilize a role that includes the permissions needed for Disk Encryption Set read.

:wrench: Resolution

The below PowerShell script is designed to duplicate the Virtual Machine Contributor role and will add Disk Encryption Set read access to the new custom role. Then the script duplicates the Azure Policy: Configure periodic checking for missing system updates on azure virtual machines

Lastly the script edits the policy and replaces the assigned Role to the newly created custom role created with the script. You will need to edit line 22 to set the subscription name. On line 23 you can see the name of the new Policy that will be created with the script. You can verify the Role Definition is created correctly by searching the display name in your roles: Virtual Machine Contributor (with Disk Encryption Set read)

:computer: PowerShell Script

https://gist.github.com/blakedrumm/1ed524f72e9bd2456bed96ff409ae17d

# ============================================================================
# Name: VM Management Role and Update Compliance Policy Setup Script
# ------------------------------------------------------------------
# Description: This PowerShell script automates the creation or updating of a custom Azure role and policy definition 
# for managing virtual machine (VM) security and compliance. It ensures VMs within specified subscriptions are managed 
# with enhanced permissions, including disk encryption set reading, and comply with system update policies. If the 
# targeted custom role does not exist, it creates one by extending the "Virtual Machine Contributor" role. It then 
# duplicates a built-in Azure policy for system update assessments, integrating the custom role to enforce update 
# compliance. Designed for Azure administrators, this script streamlines VM management, security, and compliance 
# within Azure environments.
# ============================================================================
# Author: Blake Drumm ([email protected])
# Date Created: February 2nd, 2024
# Date Modified: February 6th, 2024
# ============================================================================

# Edit Variables Below

# Type in the Subscription Scope you want to target for the new Role
# (You can comma separate Subscriptions you want to apply this role to)
# (Leave this empty if you want to select all subscriptions)
$SubscriptionsScope = 'Visual Studio Enterprise Subscription'
$PolicyDefinitionName = 'Configure periodic checking for missing system updates on azure virtual machines (with Disk Encryption Set read)'

# ============================================================================

$error.Clear()
try
{
	$SubscriptionScopeDetails = Get-AzSubscription -SubscriptionName $SubscriptionsScope -ErrorAction Stop
}
catch
{
	Write-Warning "Experienced an error while gathering the subscription(s): $error`n`n$(Get-AzContext | Out-String)"
	break
}
$SubscriptionScopeId = ($SubscriptionScopeDetails).Id

do
{
	$Question = Read-Host "Current subscriptions selected ($($SubscriptionScopeId.Count)): `n • $($SubscriptionScopeDetails.Name -join "`n • ")`n`n Are you sure you want to perform actions against the above subscriptions? (Y/N)"
}
until ($Question -match "^Y|^N")

if ($Question -match "^N")
{
	Write-Output "Exiting the script."
	return # Use `return` instead of `break` outside of loops to exit the script
}

$RoleDefinitionName = "Virtual Machine Contributor (with Disk Encryption Set read)"

$CheckIfRolePresent = Get-AzRoleDefinition -Name 'Virtual Machine Contributor (with Disk Encryption Set read)' -ErrorAction SilentlyContinue -WarningAction SilentlyContinue

Write-Output "`n-----------------------------------------------------------------------------"

if (-NOT $CheckIfRolePresent)
{
	# Get the built-in "Virtual Machine Contributor" role definition
	$role = Get-AzRoleDefinition "Virtual Machine Contributor"
	
	# Clone the role to create a new custom role
	$customRole = $role | ConvertTo-Json | ConvertFrom-Json
	
	# Set properties for the new custom role
	$customRole.Id = $null
	$customRole.Name = $RoleDefinitionName
	$customRole.Description = "Lets you manage virtual machines, but not access to them, and not the virtual network or storage account they're connected to. Also lets you read disk encryption sets."
	$customRole.IsCustom = $true
	
	# Replace Actions with a new collection including Disk Encryption Sets read permission
	$customRole.Actions = @(
		"Microsoft.Authorization/*/read",
		"Microsoft.Compute/availabilitySets/*",
		"Microsoft.Compute/locations/*",
		"Microsoft.Compute/virtualMachines/*",
		"Microsoft.Compute/virtualMachineScaleSets/*",
		"Microsoft.Compute/cloudServices/*",
		"Microsoft.Compute/disks/write",
		"Microsoft.Compute/disks/read",
		"Microsoft.Compute/disks/delete",
		"Microsoft.DevTestLab/schedules/*",
		"Microsoft.Insights/alertRules/*",
		"Microsoft.Network/applicationGateways/backendAddressPools/join/action",
		"Microsoft.Network/loadBalancers/backendAddressPools/join/action",
		"Microsoft.Network/loadBalancers/inboundNatPools/join/action",
		"Microsoft.Network/loadBalancers/inboundNatRules/join/action",
		"Microsoft.Network/loadBalancers/probes/join/action",
		"Microsoft.Network/loadBalancers/read",
		"Microsoft.Network/locations/*",
		"Microsoft.Network/networkInterfaces/*",
		"Microsoft.Network/networkSecurityGroups/join/action",
		"Microsoft.Network/networkSecurityGroups/read",
		"Microsoft.Network/publicIPAddresses/join/action",
		"Microsoft.Network/publicIPAddresses/read",
		"Microsoft.Network/virtualNetworks/read",
		"Microsoft.Network/virtualNetworks/subnets/join/action",
		"Microsoft.RecoveryServices/locations/*",
		"Microsoft.RecoveryServices/Vaults/backupFabrics/backupProtectionIntent/write",
		"Microsoft.RecoveryServices/Vaults/backupFabrics/protectionContainers/protectedItems/*/read",
		"Microsoft.RecoveryServices/Vaults/backupFabrics/protectionContainers/protectedItems/read",
		"Microsoft.RecoveryServices/Vaults/backupFabrics/protectionContainers/protectedItems/write",
		"Microsoft.RecoveryServices/Vaults/backupPolicies/read",
		"Microsoft.RecoveryServices/Vaults/backupPolicies/write",
		"Microsoft.RecoveryServices/Vaults/read",
		"Microsoft.RecoveryServices/Vaults/usages/read",
		"Microsoft.RecoveryServices/Vaults/write",
		"Microsoft.ResourceHealth/availabilityStatuses/read",
		"Microsoft.Resources/deployments/*",
		"Microsoft.Resources/subscriptions/resourceGroups/read",
		"Microsoft.SerialConsole/serialPorts/connect/action",
		"Microsoft.SqlVirtualMachine/*",
		"Microsoft.Storage/storageAccounts/listKeys/action",
		"Microsoft.Storage/storageAccounts/read",
		"Microsoft.Support/*",
		"Microsoft.Compute/diskEncryptionSets/read"
	)
	
	# Replace NotActions with an empty array if needed, or customize as necessary
	$customRole.NotActions = @()
	
	# Replace assignable scopes with a new collection
	$customRole.AssignableScopes = @()
	foreach ($id in $SubscriptionScopeId)
	{
		$customRole.AssignableScopes += "/subscriptions/$id"
	}
	Write-Output "`n`t• Getting ready to create Azure Role Definition: $RoleDefinitionName"
	$error.Clear()
	try
	{
		Write-Verbose -Verbose "Custom Role Variable:`n$($customRole | Out-String)"
		# Create the new role definition
		$newAzRole = New-AzRoleDefinition -Role $customRole -ErrorAction Stop
		Write-Output "`n`t• Successfully created Azure Role Definition!"
	}
	catch
	{
		$ScriptError = $_
		"-- Custom Role:"
		$customRole
		$ScriptError
		Write-Verbose -Verbose " == Aborting script / See above for error == "
		return
	}
}
else
{
	$newAzRole = $CheckIfRolePresent
	Write-Output "`n`tRole Definition is already created.`n"
}
Write-Output $newAzRole

Write-Output "`n-----------------------------------------------------------------------------`n"

# Step 1: Capture the New Role Definition ID
$newRoleDefinitionId = $newAzRole.Id

# Step 2: Identify the Built-in Azure Policy to Duplicate
# For example, let's assume you've identified a policy by its definition ID
$existingPolicyDefinitionId = '59efceea-0c96-497e-a4a1-4eb2290dac15'

# Get the existing policy definition
$existingPolicyDefinition = Get-AzPolicyDefinition -Name $existingPolicyDefinitionId

# Step 3: Duplicate and Update the Policy with the New Role Definition ID
# Convert the existing policy rule to an object to make it editable
$policyRuleObject = $existingPolicyDefinition.Properties.PolicyRule

# Assuming the policy involves role assignments, update the rule to include your new role definition ID
# Note: The specific changes depend on the policy's structure. This is a generic example.
# If the policy assigns roles, you might find a section in the policy rule to insert the role ID.
# Correctly format the roleDefinitionIds as an array
$policyRuleObject.then.details.roleDefinitionIds = @("/providers/Microsoft.Authorization/roleDefinitions/$newRoleDefinitionId")

# Convert the modified policy rule back to JSON
$modifiedPolicyRuleJson = $policyRuleObject | ConvertTo-Json -Depth 20

Write-Output "`t• Creating/Updating new Policy Definition.`n`t  • Name: $PolicyDefinitionName"

# Step 4: Create the New Policy Definition
$error.Clear()
try
{
	$newPolicyDefinition = New-AzPolicyDefinition -Name "PeroidicCheckMissingAzureVMs" `
												  -DisplayName $PolicyDefinitionName `
												  -Description "Configure auto-assessment (every 24 hours) for OS updates on native Azure virtual machines. You can control the scope of assignment according to machine subscription, resource group, location or tag. Learn more about this for Windows: https://aka.ms/computevm-windowspatchassessmentmode, for Linux: https://aka.ms/computevm-linuxpatchassessmentmode." `
												  -Policy $modifiedPolicyRuleJson `
												  -Mode $existingPolicyDefinition.Properties.Mode `
												  -Parameter ($existingPolicyDefinition.Properties.Parameters | ConvertTo-Json) `
												  -Metadata ($existingPolicyDefinition.Properties.Metadata | ConvertTo-Json) `
												  -ErrorAction Stop
	
	# Output the new policy definition for verification
	Write-Output "`n`t• New policy definition created/updated.`n`t  • Resource Id: $($newPolicyDefinition.ResourceId)"
}
catch
{
	Write-Warning "Experienced an error while creating the Policy Definition:`n`n$error"
	return
}


<#
    Copyright (c) Microsoft Corporation. MIT License
    
    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:
    
    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.
    
    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
    WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
    CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#>

Leave some feedback if this helped you! :v:

How to check core usage of ESU licenses - Azure Arc

January 24, 2024

:bulb: Introduction

This is my first blog post regarding the Azure space! On my blog today, I want to shed light on a common issue customers face: tracking the usage of Cores across their licenses. This is vital for understanding how many Cores are being utilized versus what’s available. Currently, Azure Portal lacks a direct report view for this, forcing customers to manually check each license and linked server to sum up the Core count, which is a hassle.

Good news, though. When a customer asks for such a report, there’s a simpler method: querying the Azure Resource Graph (ARG). The query I’ve got does the heavy lifting. It counts the Cores from each linked Arc machine, adjusts the count to meet the minimum requirements (8 vCores per VM and 16 pCores per physical server), and then presents this data alongside the license info.

This approach is a game-changer. It not only saves time but also gives a more accurate picture of Core usage.

:chart_with_upwards_trend: Steps to Gather Data

:mag: Open Resource Graph Explorer

Within the Azure Portal open Resource Graph Explorer.

:memo: Kusto Query (KQL)

resources
| where type =~ "microsoft.hybridcompute/licenses"
| extend sku = tostring(properties.licenseDetails.edition)
| extend totalCores = tostring(properties.licenseDetails.processors)
| extend coreType = case(
    properties.licenseDetails.type =~ 'vCore','Virtual core',
    properties.licenseDetails.type =~ 'pCore','Physical core',
    'Unknown'
)
| extend status = tostring(properties.licenseDetails.state)
| extend licenseId = tolower(tostring(id)) // Depending on what is stored in license profile, might have to get the immutableId instead
| join kind=inner(
    resources
    | where type =~ "microsoft.hybridcompute/machines/licenseProfiles"
    | extend machineId = tolower(tostring(trim_end(@"\/\w+\/(\w|\.)+", id)))
    | extend licenseId = tolower(tostring(properties.esuProfile.assignedLicense))
) on licenseId // Get count of license profile per license, a license profile is created for each machine that is assigned a license
| join kind=inner(
    resources
    | where type =~ "microsoft.hybridcompute/machines"
    | extend machineId = tolower(id)
    | extend coreCount = toint(properties.detectedProperties.coreCount) 
) on machineId // Get core count by machine
    | extend adjustedCoreCount = iff(sku =~ "Standard" 
                                , case(coreCount < 8, 8, coreCount) , 
                                    iff(sku =~ "DataCenter"
                                        , case(coreCount < 16, 16, coreCount)
                                        ,coreCount))  
| extend machineName = tostring(split(machineId, "/")[-1])
| project machineName, machineId, licenseId, name, type, location, subscriptionId, resourceGroup, sku, totalCores, coreType, status, adjustedCoreCount
| summarize UsedCoreCount = sum(adjustedCoreCount) by licenseId, name, type, location, subscriptionId, resourceGroup, sku, totalCores, coreType, status

SCOM Certificate Error - ASN1 Bad Tag Value Met

November 14, 2023

:book: Introduction

Today I encountered an issue where SCOM fails to generate a certificate for Unix/Linux agents with an error message stating “ASN1 bad tag value met”.

Error Description

Task invocation failed with error code -2130771918. Error message was: The SCXCertWriteAction module encountered a DoProcess exception. The workflow “Microsoft.Unix.Agent.GetCert.Task” has been unloaded.

Module: SCXCertWriteAction
Location: DoProcess
Exception type: ScxCertLibException
Exception message: Unable to create certificate context
; {ASN1 bad tag value met.
}
Additional data: Sudo path: /etc/opt/microsoft/scx/conf/sudodir/

Management group: SCOM2019
Workflow name: Microsoft.Unix.Agent.GetCert.Task
Object name: UNIX/Linux Resource Pool
Object ID: {7B5B80D1-5C4A-6643-762D-60F46FB70CB8}

:mag: Possible Causes and Resolution

Possible Causes

Sudoers permissions are missing.
Errors in SSHCommandProbe.log (look for errdata)

3: 08/08/22 12:01:59 : Entering RunSSHCommand
3: 08/08/22 12:01:59 : Using su command: su - root -c
3: 08/08/22 12:01:59 : Using sudo command: ${SUDO_PATH}sudo sh -c
3: 08/08/22 12:01:59 : sending: if [ -x /etc/opt/microsoft/scx/conf/sudodir/sudo ]; then
SUDO_PATH=/etc/opt/microsoft/scx/conf/sudodir/; export SUDO_PATH
else
if [ -x /opt/sfw/bin/sudo ]; then
SUDO_PATH=/opt/sfw/bin/; export SUDO_PATH
else
SUDO_PATH=/usr/bin/; export SUDO_PATH
fi
fi
echo “Sudo path: ${SUDO_PATH}”
${SUDO_PATH}sudo sh -c ‘cat /etc/opt/microsoft/scx/ssl/scx.pem’

3: 08/08/22 12:01:59 : Enter SSHFacade::RunCommand
3: 08/08/22 12:02:02 : Leave SSHFacade::RunCommand
3: 08/08/22 12:02:02 : returned: Sudo path: /etc/opt/microsoft/scx/conf/sudodir/

errdata: sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
In the secure log file on the Linux Server, you should see messages like:

Aug 8 12:32:00 rhel8-5 sshd[2749309]: pam_unix(sshd:session): session opened for user scxmaint by (uid=0)
Aug 8 12:32:00 rhel8-5 sudo[2749343]: pam_unix(sudo:auth): conversation failed
Aug 8 12:32:00 rhel8-5 sudo[2749343]: pam_unix(sudo:auth): auth could not identify password for [scxmaint]
Aug 8 12:32:02 rhel8-5 sudo[2749343]: scxmaint : command not allowed ; TTY=unknown ; PWD=/home/scxmaint ; USER=root ; COMMAND=/bin/sh -c cat /etc/opt/microsoft/scx/ssl/scx.pem
Aug 8 12:32:02 rhel8-5 sshd[2749309]: pam_unix(sshd:session): session closed for user scxmaint

How to fix it

Check and verify the sudoers are setup correctly:
https://learn.microsoft.com/system-center/scom/manage-security-unix-linux-sudoers-templates

Leave some feedback if this helped you! :v:

SCOM Scheduled Reports Fail with AlertConnectionString, PerfConnectionString, or StateConnectionString Error

October 31, 2023

:warning: Symptoms

In SCOM 2016, 2019, or 2022, scheduled reports fail to run. If you check report status by opening the Scheduled Reports view in the Operations Console, the status message shows one of the following:

“Default value or value provided for the report parameter ‘AlertConnectionString’ is not a valid value.”
“Default value or value provided for the report parameter ‘PerfConnectionString’ is not a valid value.”
“Default value or value provided for the report parameter ‘StateConnectionString’ is not a valid value.”

:bulb: Cause

This can occur if the <appSettings> element is missing from the SQL Server Reporting Services (SSRS) configuration file.

By default, this configuration file is in the following folder on the SCOM Reporting server:

C:\Program Files\Microsoft SQL Server Reporting Services\SSRS\ReportServer\bin\ReportingServicesService.exe.config

The issue described in the Symptoms section occurs when the below element is missing from the configuration file:

  <appSettings>
  	<add key="ManagementGroupId" value="management_group_GUID"/>
  </appSettings>

:wrench: Resolution

PowerShell Automatic Method

What Does This PowerShell Script Do?

This script automates the process of configuring SQL Server Reporting Services (SSRS) in a System Center Operations Manager (SCOM) environment. Below are the key steps:

Step 1: Gather System Information

Fetch SSRS Info: Gathers essential details about the SSRS installation using Windows Management Instrumentation (WMI).
Fetch SCOM Info: Retrieves System Center Operations Manager Data Warehouse database settings from the Windows registry.

Step 2: Fetch Management Group ID

SQL Connection: Establishes a connection to the System Center Operations Manager Data Warehouse database.
Fetch ID: Executes a SQL query to obtain the Management Group ID.

Step 3: Locate and Check the Configuration File

Locate Config: Determines the path of the SSRS configuration file (ReportingServicesService.exe.config).
Check Config: Checks if the configuration file already contains the correct Management Group ID.

Step 4: Update Configuration

Backup Config: Creates a backup of the original configuration file.
Update Config: If necessary, updates the SSRS configuration file with the new Management Group ID.

Step 5: Finalize Changes

Save Changes: Saves the updated configuration.
Restart Service: Restarts the SSRS service to apply the changes.

Run the following script on your System Center Operations Manager Reporting Server. Be sure the PowerShell window is opened as Administrator and you have rights to query the Operations Manager Data Warehouse Database:

 #Author: Blake Drumm ([email protected])
 #Date Created: 10/31/2023
 $error.Clear()
 try
 {
 	$RS = "root\Microsoft\SqlServer\ReportServer\" + (Get-CimInstance -Namespace root\Microsoft\SqlServer\ReportServer -ClassName __Namespace -ErrorAction Stop | Select-Object -First 1).Name
 	$RSV = $RS + "\" + (Get-CimInstance -Namespace $RS -ClassName __Namespace -ErrorAction Stop | Select-Object -First 1).Name + "\Admin"
 	$RSInfo = Get-CimInstance -Namespace $RSV -ClassName MSReportServer_ConfigurationSetting -ErrorAction Stop
    	
 	# Output or use $RSInfo as needed
 	$SSRSConfigPath = $RSInfo.PathName
    	
 	$DWDBInfo = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\System Center Operations Manager\12\Reporting' -ErrorAction Stop | Select-Object DWDBInstance, DWDBName
    	
 }
 catch
 {
 	Write-Host "An error occurred: $error" -ForegroundColor Red
 	return
 }
    
 # User-defined SQL Server Instance for the Operations Manager database
 $SQLInstance = $DWDBInfo.DWDBInstance
 $DatabaseName = $DWDBInfo.DWDBName
    
 # Get the Management Group ID using SQL query
 $connectionString = "Server=$SQLInstance;Database=$DatabaseName;Integrated Security=True;"
 $connection = New-Object System.Data.SqlClient.SqlConnection
 $connection.ConnectionString = $connectionString
 $connection.Open()
    
 $command = $connection.CreateCommand()
 $command.CommandText = "SELECT ManagementGroupGuid from vManagementGroup"
 $reader = $command.ExecuteReader()
    
 $managementGroupId = $null
 if ($reader.Read())
 {
 	$managementGroupId = $reader["ManagementGroupGuid"]
 }
 $reader.Close()
 $connection.Close()
    
 # Check if we got the Management Group ID
 if ($null -eq $managementGroupId)
 {
 	Write-Host "Failed to get Management Group ID." -ForegroundColor Red
 	return
 	#exit 1
 }
    
 $SSRSParentDirectory = Split-Path $SSRSConfigPath
    
 $error.Clear()
 try
 {
 	# Path to ReportingServicesService.exe.config
 	$configPath = (Resolve-Path "$SSRSParentDirectory\bin\ReportingServicesService.exe.config" -ErrorAction Stop).Path
 }
 catch
 {
 	Write-Warning "Unable to access '$SSRSParentDirectory\bin\ReportingServicesService.exe.config' : $error"
 	return
 	#exit 1
 }
 # Load the XML content of the config file
 [xml]$configXml = Get-Content -Path $configPath
    
 # Check if the appSettings element already exists and has the correct ManagementGroupId
 $appSettings = $configXml.SelectSingleNode("/configuration/appSettings/add[@key='ManagementGroupId']")
 if ($null -ne $appSettings -and $appSettings.GetAttribute("value") -eq $managementGroupId)
 {
 	Write-Host "Configuration is already up to date." -ForegroundColor Green
 	return
 	#exit 0
 }
    
 # Create a backup of the existing config file
 Copy-Item -Path $configPath -Destination "$configPath.bak"
    
 # Update or create the appSettings element
 if ($null -ne $appSettings)
 {
 	$appSettings.SetAttribute("value", $managementGroupId)
 }
 else
 {
 	$appSettings = $configXml.CreateElement("appSettings")
 	$addKey = $configXml.CreateElement("add")
 	$addKey.SetAttribute("key", "ManagementGroupId")
 	$addKey.SetAttribute("value", $managementGroupId)
 	$appSettings.AppendChild($addKey)
 	$configXml.SelectSingleNode("/configuration").AppendChild($appSettings)
 }
    
 # Save the modified XML back to the config file
 $configXml.Save($configPath)
    
 # Restart the SQL Server Reporting Services service
 Restart-Service -Name $RSInfo.ServiceName
    
 Write-Host "Configuration updated successfully." -ForegroundColor Green

Manual Method

To resolve the issue, edit the SSRS configuration file to add a valid <appSettings> element.

The following are the steps:

On any computer that has the SCOM Operations Console installed, run the following PowerShell commands to get the management group ID:
```
Import-Module OperationsManager
(Get-SCOMManagementGroup).id.guid
```
Create a copy of the existing ReportingServicesService.exe.config file. By default, this file is in the following folder on the SCOM Reporting server:

C:\Program Files\Microsoft SQL Server Reporting Services\SSRS\ReportServer\bin\ReportingServicesService.exe.config
Open the ReportingServicesService.exe.config file in a text editor.

Add the following element to the file immediately before the closing </configuration> tag:

 <appSettings>
   <add key="ManagementGroupId" value="management_group_GUID"/>
 </appSettings>

In the above XML, replace management_group_GUID with the management group ID you obtained in Step 1.

For example, the element will look like the following:

 <startup useLegacyV2RuntimeActivationPolicy="true">
 	<supportedRuntime version="v4.0"/>
 </startup>
 <appSettings>
 	<add key="ManagementGroupId" value="7f263180-e7d2-9c12-a1cd-0c6c54a7341c"/>
 </appSettings>
 </configuration>

Save the configuration file.
Restart the SQL Server Reporting Services service.

SCOM SDK not staying on - AzMan Issue

October 12, 2023

:book: Introduction

Today I came across a problem with a customer that had an issue with the System Center Data Access Service (SDK) on all Management Servers crashing immediately after being started. The below errors can be found in the Operations Manager event log:

1st Event

Log Name:      Operations Manager 
Source:        OpsMgr SDK Service 
Date:          10/12/2023 3:10:10 PM 
Event ID:      26380 
Level:         Error 
Computer:      MS01-2019.contoso.com 
Description: 
The System Center Data Access service failed due to an unhandled exception.  
The service will attempt to restart. 
Exception: 
 
Microsoft.EnterpriseManagement.ConfigurationReaderException: Feature of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.IAuthorizationFeature, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' cannot be added to the container. ---> System.ServiceModel.FaultException\`1\[Microsoft.EnterpriseManagement.Common.UnknownAuthorizationStoreException]: The creator of this fault did not specify a Reason. 
   at Microsoft.EnterpriseManagement.Mom.Sdk.Authorization.AzManHelper.Initialize(String pathToStore, String appName, AzManHelperModes helperMode, String storeDesc, String appDesc) 
   at Microsoft.EnterpriseManagement.Mom.Sdk.Authorization.AuthManager.Initialize(AuthManagerModes authMode) 
   at Microsoft.EnterpriseManagement.ServiceDataLayer.AuthorizationFeatureImplementation.InitializeAzmanAccessCheckObject() 
   at Microsoft.EnterpriseManagement.ServiceDataLayer.AuthorizationFeatureImplementation.Initialize(IContainer container) 
   at Microsoft.EnterpriseManagement.SingletonLifetimeManager\`1.GetComponent\[K]() 
   at Microsoft.EnterpriseManagement.FeatureContainer.GetFeatureInternal\[T](Type type, String featureName) 
   at Microsoft.EnterpriseManagement.FeatureContainer.AddFeatureInternal\[T,V](ActivationContext`1 context, String featureName) 
   --- End of inner exception stack trace --- 
   at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ReadFeatures(XPathNavigator navi, IContainer container) 
   at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.Process() 
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(InProcEnterpriseManagementConnectionSettings configuration) 
   at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(Object state) 
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) 
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) 
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) 
   at System.Threading.ThreadHelper.ThreadStart(Object obj)

2nd Event

Log Name:      Operations Manager 
Source:        OpsMgr SDK Service 
Date:          10/12/2023 3:10:10 PM 
Event ID:      26339 
Level:         Error 
Computer:      MS01-2019.contoso.com 
Description: 
An exception was thrown while initializing the service container. 
 Exception message: Initialize 
 Full exception: Feature of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.IAuthorizationFeature, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' cannot be added to the container

3rd Event

Log Name:      Operations Manager 
Source:        OpsMgr SDK Service 
Date:          10/12/2023 3:10:10 PM 
Event ID:      26325 
Level:         Error 
Computer:      MS01-2019.contoso.com 
Description:   An authorization store exception was thrown in the System Center Data Access service. Exception message: Unable to perform the operation because of authorization store errors.

:page_with_curl: How to fix it

First you want to verify this problem is definitely related to Authorization Manager (AzMan).

What is Authorization Manager

Authorization Manager, often referred to as AzMan or AzMan.msc, is a Microsoft Management Console (MMC) snap-in that provides a flexible framework for integrating role-based access control (RBAC) into applications. It’s a part of Windows and can be used to manage authorization policies using roles and tasks for applications.

Here’s a basic overview of its functionalities:

Role-Based Access Control (RBAC): Authorization Manager allows for the definition of roles, operations, tasks, and scopes to achieve RBAC. This makes it easier for administrators to manage user permissions based on their roles within an organization.

Store Policies in AD, AD LDS, and XML: Authorization policies can be stored in Active Directory (AD), Active Directory Lightweight Directory Services (AD LDS), or in an XML file. This flexibility allows for the storage and retrieval of policies in a manner best suited to the application’s needs.

Application Integration: Applications can be developed to use Authorization Manager for enforcing access control. By doing this, the application offloads the management of roles and permissions to AzMan, making it easier to manage and update permissions without changing application code.

Scriptable Interface: Authorization Manager provides a scriptable interface, which means you can automate many administrative tasks using scripts.

Auditing: Authorization Manager can be configured to log access requests, which can then be used for audit purposes.

Delegation of Administrative Duties: Administrators can delegate certain administrative duties to others, allowing for distributed management of roles and permissions without giving full administrative access.

You can either run a PowerShell Script to check the AzMan connectivity or you can use the GUI.

PowerShell Method

Open a PowerShell window as Administrator on the SCOM Management Server, copy and paste the below script:

try
{
	$SQLServerAddress = (Get-ItemPropertyValue -Path "HKLM:\SOFTWARE\Microsoft\System Center\2010\Common\Database" -Name "DatabaseServerName" -ErrorAction Stop)
	$SQLServerOpsDatabase = (Get-ItemPropertyValue -Path "HKLM:\SOFTWARE\Microsoft\System Center\2010\Common\Database" -Name "DatabaseName" -ErrorAction Stop)
}
catch
{
	Write-Host "Error automatically gathering the SCOM SQL instance name: $($_.Exception.Message)" -ForegroundColor Red
}

# Uncomment the below lines to set the SQL Server manually instead of trying to automatically grab the information from the registry
# $SQLServerAddress = "SQL01-2019"
# $SQLServerOpsDatabase = "OperationsManager"

# Load the AzMan assembly
[Reflection.Assembly]::LoadWithPartialName("AzRoles")

# Create a new AzAuthorizationStore object
$azStore = New-Object -ComObject AzRoles.AzAuthorizationStore

try
{
	# Initialize the authorization store; replace with your SQL AzMan store connection string
	$azStore.Initialize(0, "mssql://Driver={SQL Server};Server={$SQLServerAddress};/$SQLServerOpsDatabase/AzmanStore")
	Write-Host "Connection to AzMan store successful." -ForegroundColor Green
	
	# Optionally, list applications in the AzMan store to further verify connectivity
	#$azStore.GetApplications() | ForEach-Object { Write-Host $_.Name }
}
catch
{
	# Catch any exceptions and display the error message
	Write-Host "Connection failed: $($_.Exception.Message)" -ForegroundColor Red
}

If you are affected you will see an output of:

Connection failed: The security ID structure is invalid. (Exception from HRESULT: 0x80070539)

GUI Method

Open Authorization Manager. Open a run box and type in:
```
AzMan.msc
```
Press Enter and you will see the Authorization Manager Window.
Right Click the text Authorization Manager in the left box, select Open Authorization Store…
Select Microsoft SQL and enter into the Store name text box:
```
mssql://Driver={SQL Server};Server={<SQLServer\Instance,OptionalPort>};/<OpsDatabaseName>/AzmanStore
```
Press Enter.
You will see the following pop-up:
Cannot open the authorization store. The following problem occurred: The security ID structure is invalid.

:mag: Find and fix the issue in SQL

Run the following query to determine the DB Owner for the databases:

SELECT name AS DatabaseName, SUSER_SNAME(owner_sid) AS DatabaseOwner
FROM sys.databases;

OR
You can also see the DB Owner via SQL Server Management Studio.

Locate the Operations Manager or Operations Manager Data Warehouse Database in the Object Explorer view.
Right Click the Database and select Properties.
Click on Files and on the top is the DB Owner.
Verify the account is not a local SQL account. I usually just set this to sa, as it is a built-in account that will not expire. But be aware as it can be disabled.

Run the following query to determine if a local sql account has db_owner permission on the Operations Manager Database (the query is read only):

DECLARE @DB_Users TABLE
(DBName sysname, UserName sysname, LoginType sysname, AssociatedRole varchar(max),create_date datetime,modify_date datetime)

INSERT @DB_Users
EXEC sp_MSforeachdb

'
use [?]
SELECT ''?'' AS DB_Name,
case prin.name when ''dbo'' then prin.name + '' (''+ (select SUSER_SNAME(owner_sid) from master.sys.databases where name =''?'') + '')'' else prin.name end AS UserName,
prin.type_desc AS LoginType,
isnull(USER_NAME(mem.role_principal_id),'''') AS AssociatedRole ,create_date,modify_date
FROM sys.database_principals prin
LEFT OUTER JOIN sys.database_role_members mem ON prin.principal_id=mem.member_principal_id
WHERE prin.sid IS NOT NULL and prin.sid NOT IN (0x00) and
prin.is_fixed_role <> 1 AND prin.name NOT LIKE ''##%'''

SELECT
dbname,username ,logintype ,create_date ,modify_date ,
STUFF(
(
SELECT ',' + CONVERT(VARCHAR(500),associatedrole)
FROM @DB_Users user2
WHERE
user1.DBName=user2.DBName AND user1.UserName=user2.UserName
FOR XML PATH('')
)
,1,1,'') AS Permissions_user
FROM @DB_Users user1
WHERE LoginType = 'SQL_USER' and
UserName != 'dbo (sa)' and
UserName != 'MS_DataCollectorInternalUser'
GROUP BY
dbname,username ,logintype ,create_date ,modify_date
ORDER BY DBName, username

You can use the following query to edit the User Mapping for the local SQL account and remove the db_owner role:
(Example Local Account Name: LocalSQLAccount <– Replace with your SQL account)
```
 USE [OperationsManager];
 GO
 EXEC sp_droprolemember N'db_owner', N'LocalSQLAccount';
 GO
```
OR
You can edit via SQL Server Management Studio.
1. Go to Security -> Logins, locate the local SQL account, Right Click and go to Properties.
2. Go to User Mapping and select the Operations Manager or Operations Manager Data Warehouse database.
3. Scroll in the Database role membership panel until you see db_owner, uncheck it and press OK.
Restart the System Center Operations Manager Data Access Service (omsdk) on the Management Servers:
```
Restart-Service -Name OMSDK
```

Leave some feedback if this helped you! :v:

SCOM License Expired

September 25, 2023

:book: Introduction

We attempted to activate the expired license for SCOM but we were unable due to the expiration of the product. The following errors would show when attempting to activate via PowerShell. These errors would show every time we try to activate:

Set-SCOMLicense : Unable to proceed with the command. Ensure you are connecting to correct Management Server and have sufficient privileges to execute the command. 
At line:1 char:1
+ Set-SCOMLicense -ManagementServer localhost -ProductId BZX70-NQZUT-SS ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (BZX70-NQZUT-SSFEQ-7PWWZ-9ZOQC:String) [Set-SCOMLicense], InvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.SystemCenter.OperationsManagerV10.Commands.Commands.AdministrationCmdlets.SetSCOMLicense

At the same time two consecutive events are logged in the Application Log:

Log Name:      Application 
Source:        .NET Runtime
Event ID:      1026
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Description:
Application: MonitoringHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.UnauthorizedAccessException
   at Microsoft.EnterpriseManagement.Common.Internal.ExceptionHandlers.HandleChannelExceptions(System.Exception)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.CreateEndpoint[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](Microsoft.EnterpriseManagement.EnterpriseManagementConnectionSettings, Microsoft.EnterpriseManagement.Common.Internal.SdkChannelObject`1<Microsoft.EnterpriseManagement.Common.Internal.IDispatcherService>)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.ConstructEnterpriseManagementGroupInternal[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](Microsoft.EnterpriseManagement.EnterpriseManagementConnectionSettings, Microsoft.EnterpriseManagement.DataAbstractionLayer.ClientDataAccessCore)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.RetrieveEnterpriseManagementGroupInternal[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](Microsoft.EnterpriseManagement.EnterpriseManagementConnectionSettings, Microsoft.EnterpriseManagement.DataAbstractionLayer.ClientDataAccessCore)
   at Microsoft.EnterpriseManagement.Common.Internal.SdkDataLayerProxyCore.Connect[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](Microsoft.EnterpriseManagement.EnterpriseManagementConnectionSettings, Microsoft.EnterpriseManagement.DataAbstractionLayer.ClientDataAccessCore)
   at Microsoft.EnterpriseManagement.ManagementGroup.InternalInitialize(Microsoft.EnterpriseManagement.EnterpriseManagementConnectionSettings, Microsoft.EnterpriseManagement.ManagementGroupInternal)
   at Microsoft.EnterpriseManagement.ManagementGroup.Connect(System.String)
   at Microsoft.EnterpriseManagement.Mom.ValidateAlertSubscriptionModule.ValidateAlertSubscriptionDataSource.ValidateAlertSubscriptions(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.TimerQueueTimer.CallCallback()
   at System.Threading.TimerQueueTimer.Fire()
   at System.Threading.TimerQueue.FireNextTimers()

and

Log Name:      Application 
Source:        Application Error
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Description:
Faulting application name: MonitoringHost.exe, version: 10.19.10014.0, time stamp: 0x5c45e51b
Faulting module name: KERNELBASE.dll, version: 10.0.14393.5582, time stamp: 0x63882301
Exception code: 0xe0434352
Fault offset: 0x0000000000026ea8
Faulting process id: 0xaf4
Faulting application start time: 0x01d937237975b89c
Faulting application path: C:\Program Files\Microsoft System Center\Operations Manager\Server\MonitoringHost.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: 21ffc48d-66ca-4aab-9967-47b2201d498f
Faulting package full name: 
Faulting package-relative application ID: 

:page_with_curl: How to fix it

Follow the below steps to allow you to fix the Activation issue:

Run the following PowerShell script on your Management Server to unregister the Time Service, set the date 48 hours ahead of the date SCOM was installed on the Management Server, and finally restarting the SCOM services.

 # Unregister the Time Service
 W32tm /unregister

 # Stop the Time Service
 Stop-Service -Name w32time

 # Set the Operating System date 48 hours ahead of the InstalledOn date that is in the registry
 Set-Date ([DateTime]::ParseExact($((Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup').InstalledOn), 'M/d/yyyy-HH:mm:ss', $null).AddHours(48))

 # Restart the SCOM Services (System Center Data Access Service, System Center Management Configuration, Microsoft Monitoring Agent)
 Restart-Service -Name omsdk, cshost, healthservice

Verify the time has been changed.
Open the SCOM Console and navigate to the top of the Console window, click on Help -> About
Click on Activate and type in your License key:
Accept the license agreement:
Successfully activated!

Register the Time Service to revert the changes made above, close the SCOM Console, and finally restart the SCOM SDK Service (System Center Data Access Service):

# Register the Time Service
W32tm /register

# Close the SCOM Console
Stop-Process -Name Microsoft.EnterpriseManagement.Monitoring.Console -Force
   
# Restart the SCOM SDK Service
Restart-Service -Name omsdk

Open the SCOM Console and verify if you go to Help -> About. Do you see (Eval) in the Console version?
If you see (Retail) (as shown below), you are activated! :sun_behind_small_cloud:

SCOM UNIX/Linux - Failed to find a matching agent kit to install

August 14, 2023

What is the problem?

The latest version of the UNIX/Linux Management Pack includes a hotfix for compiler mitigated OMI support. The KB (5028684) needs to be applied, which needs to be applied on the SCOM Management Server and SCOM Console components installed in your environment.

UNIX/Linux versions

Linux Agent version: 1.7.1-0
SCOM 2022 (10.22.1052.0): https://www.microsoft.com/download/details.aspx?id=104213
SCOM 2019 (10.19.1226.0): https://www.microsoft.com/download/details.aspx?id=58208
SCOM 2016 (7.6.1185.0): https://www.microsoft.com/download/details.aspx?id=29696

How to fix

Download the hotfix listed here:
https://support.microsoft.com/kb/5028684

You must apply the hotfix on the SCOM Management Server and SCOM Console components installed in your environment.

SCOM ETL Trace Gathering Script

June 30, 2023

:book: Introduction

This Tool will assist you in gathering ETL Traces. You have the options of selecting specific Tracing to gather with this script.

The script will perform the following, in this order:

Stops any existing ETL Traces
- Optional: Stops the SCOM Services
Starts the ETL Trace
- Optional: Starts the SCOM Services back up
Script will wait for issue to occur
- Default: Pauses Script, waits until you press Enter
- Optional: Sleeps for x Seconds (-SleepSeconds 10)
- Optional: Script will loop until an Event ID is detected (-DetectOpsMgrEventID)
- Optional: Script can sleep for any amount of seconds after an Event is detected. (-SleepAfterEventDetection)
Stops ETL Trace
Optional: Script can attempt to Format the ETL Trace (-FormatTrace)
Zips Up Output and Opens Explorer Window for Viewing File

:arrow_down: How to get it

You can get a copy of the script here:
Start-ScomETLTrace.ps1 :arrow_left: Direct Download Link
or
Personal File Server - Start-ScomETLTrace.ps1 :arrow_left: Alternative Download Link
or
Personal File Server - Start-ScomETLTrace.txt :arrow_left: Text Format Alternative Download Link

:page_with_curl: How to use it

Open Powershell Prompt as Administrator:

Examples

All Available Commands
.\Start-ScomETLTrace.ps1 -GetAdvisor -GetApmConnector -GetBID -GetConfigService -GetDAS -GetFailover -GetManaged -GetNASM -GetNative -GetScript -GetUI -VerboseTrace -DebugTrace -NetworkTrace -SleepSeconds -RestartSCOMServices -DetectOpsMgrEventID
Get Verbose Native ETL Trace
.\Start-ScomETLTrace.ps1 -GetNative -VerboseTrace
Get Verbose Native ETL Trace and Format the trace
.\Start-ScomETLTrace.ps1 -GetNative -VerboseTrace -FormatTrace
Gather Verbose ETL Trace and detect for 1210 Event ID (Sleep for 30 Seconds between checks)
.\Start-ScomETLTrace.ps1 -VerboseTrace -DetectOpsMgrEventID 1210 -SleepSeconds 30
Gather Verbose ETL Trace and detect for 1210 Event ID (Sleep for 30 Seconds between checks) and sleep for 10 seconds after finding the Event ID
.\Start-ScomETLTrace.ps1 -VerboseTrace -DetectOpsMgrEventID 1210 -SleepSeconds 30 -SleepAfterEventDetection 10
Restart SCOM Services after starting an ETL Trace. Sleep for 2 Minutes and stop the Trace Automatically
.\Start-ScomETLTrace.ps1 -Sleep 120 -RestartSCOMServices
Get All ETL Traces

Get Verbose Tracing for all the Default Tracing Available (just like running this: -GetAdvisor -GetApmConnector -GetBID -GetConfigService -GetDAS -GetFailover -GetManaged -GetNASM -GetNative -GetScript -GetUI)
.\Start-ScomETLTrace.ps1 -VerboseTrace
Get Debug Tracing for all the Default Tracing Available (just like running this: -GetAdvisor -GetApmConnector -GetBID -GetConfigService -GetDAS -GetFailover -GetManaged -GetNASM -GetNative -GetScript -GetUI)
.\Start-ScomETLTrace.ps1 -DebugTrace
Get Verbose Tracing for all the Default Tracing Available and Network Tracing (just like running this: -GetAdvisor -GetApmConnector -GetBID -GetConfigService -GetDAS -GetFailover -GetManaged -GetNASM -GetNative -GetScript -GetUI)
.\Start-ScomETLTrace.ps1 -VerboseTrace -NetworkTrace
Get Verbose Tracing for all the Default Tracing Available and OpsMgrModuleLogging for Linux Related Issues
.\Start-ScomETLTrace.ps1 -VerboseTrace -OpsMgrModuleLogging

Leave some feedback if this helped you! :v:

Enforce TLS 1.2 & TLS 1.3 in SCOM - The PowerShell Way!

May 25, 2023

:book: Introduction

This PowerShell script will allow you to enforce TLS 1.2 & TLS 1.3 in your SCOM Environment to help you to secure your environment. (A big thank you to Kevin Holman for the original creation of his TLS 1.2 enforcement script, which this script originated from.) It will attempt to auto download the prerequisites if they are not present in the local directory (or if you set the parameter DirectoryForPrerequisites to another path it will check there). The script from a high level will do the following:

Creates a log file to Program Data (C:\ProgramData\SCOM_TLS_1.2_-_<Month>-<Day>-<Year>.log).
Locate or Download the prerequisites for TLS 1.2 Enforcement.
Checks the SCOM Role (Management Server, Web Console, ACS Collector).
Checks the version of System Center Operations Manager to confirm supportability of TLS enforcement.
Checks the .NET version to confirm you are on a valid version.
Checks the SQL version (on both the Operations Manager and Data Warehouse Database Instances) to confirm your version of SQL supports TLS enforcement.
Checks and/or installs the (prerequisite software) MSOLEDB driver (or SQL Client).
Checks and/or installs the (prerequisite software) ODBC driver.
Checks and/or modifies the registry to enforce TLS 1.2 (If your using Window Server 2022 (or newer) or Windows 11 (or newer) it will attempt to enforce TLS 1.2 and TLS 1.3).
Ask to reboot the machine to finalize the configuration.

:classical_building: Argument List

Parameter	Alias	Type	Description
AssumeYes	yes	Switch	The script will not ask any questions. Good for unattended runs.
DirectoryForPrerequisites	dfp	String	The directory to save / load the prerequisites from. Default is the current directory.
ForceDownloadPrerequisites	fdp	Switch	Force download the prerequisites to the directory specified in DirectoryForPrerequisites.
SkipDotNetCheck	sdnc	Switch	Skip the .NET Check step.
SkipDownloadPrerequisites	sdp	Switch	Skip downloading the prerequisite files to current directory.
SkipModifyRegistry	smr	String	Skip any registry modifications.
SkipRoleCheck	src	Switch	Skip the SCOM Role Check step.
SkipSQLQueries	ssq	Switch	Skip any check for SQL version compatibility.
SkipSQLSoftwarePrerequisites	sssp	Switch	Skip the ODBC, MSOLEDBSQL, and/or Microsoft SQL Server 2012 Native Client.
SkipVersionCheck	svc	Switch	Skip SCOM Version Check step.

:notebook: Note

You may edit line 1904 in the script to change what happens when the script is run without any arguments or parameters, this also allows you to change what happens when the script is run from the Powershell ISE.

:arrow_down: How to get it

You can get a copy of the script here:
Invoke-EnforceSCOMTLS1.2.ps1 :arrow_left: Direct Download Link
or
Personal File Server - Invoke-EnforceSCOMTLS1.2.ps1 :arrow_left: Alternative Download Link
or
Personal File Server - Invoke-EnforceSCOMTLS1.2.txt :arrow_left: Text Format Alternative Download Link

:page_with_curl: How to use it

Example 1

Normal run:
.\Invoke-EnforceSCOMTLS1.2.ps1
Example 2

Set the prerequisites folder:
.\Invoke-EnforceSCOMTLS1.2.ps1 -DirectoryForPrerequisites "C:\Temp"
Example 3

Assume yes to all questions asked by script:
.\Invoke-EnforceSCOMTLS1.2.ps1 -AssumeYes

Check TLS Configuration

You can run the following PowerShell script to gather your current TLS configuration:

:arrow_down: How to get it

Get-TLSRegistryKeys.ps1 :arrow_left: Direct Download Link
or
Personal File Server - Get-TLSRegistryKeys.ps1 :arrow_left: Alternative Download Link
or
Personal File Server - Get-TLSRegistryKeys.txt :arrow_left: Text Format Alternative Download Link

Leave some feedback if this helped you! :v:

How to Decommission SCOM Gateway

April 25, 2023

The Steps Required

Move any agents (Agents or Agentless) assigned to the Gateway to another server.
Uninstall the SCOM Gateway through programs and features.
Delete the Gateway with the Gateway Approval Tool.

The Microsoft.EnterpriseManagement.GatewayApprovalTool.exe tool is needed only on the management server, and it only has to be run once.

To copy Microsoft.EnterpriseManagement.GatewayApprovalTool.exe to management servers From a target management server, open the Operations Manager installation media \SupportTools\ (amd64 or x86) directory.

Copy the Microsoft.EnterpriseManagement.GatewayApprovalTool.exe from the installation media to the Operations Manager installation directory.

The command to delete a SCOM Gateway:
```
 Microsoft.EnterpriseManagement.GatewayApprovalTool.exe /ManagementServerName=<managementserverFQDN> /GatewayName=<GatewayFQDN> /Action=Delete
```

Kevin Holman has a good guide for Management Server’s here (which some items apply for Gateway’s as well): https://kevinholman.com/2016/02/03/checklist-removing-migrating-old-management-servers-to-new-ones/

Powershell Script to move Agentless-Managed Machines

#Author: Blake Drumm ([email protected])
#Date: August 8th, 2023

#------------------------------- 
# Variables Section 
#------------------------------- 
$FromGatewayServer = 'GW01-2019.contoso-2019.com' # Management Server or Gateway Server to move FROM
$ToGatewayServer = 'MS01-2019.contoso-2019.com' # Management Server or Gateway Server to move TO 
#-------------------------------

$MSList = Get-SCOMManagementServer 
$FromGatewayServer = $MSList | Where-Object {$_.Name -match $FromGatewayServer} 
$ToGatewayServer =  $MSList | Where-Object {$_.Name -match $ToGatewayServer}

Get-SCOMAgentlessManagedComputer | Where-Object {$_.ProxyAgentPrincipalName -match $FromGatewayServer.DisplayName} | Set-SCOMAgentlessManagedComputer -ManagedByManagementServer $ToGatewayServer -PassThru 

Powershell Script to move Agents

You can find the below script (and others) here:
https://github.com/blakedrumm/SCOM-Scripts-and-SQL/tree/master/Powershell/Agents%20Failover

Replace the following variables before running:

<MoveFrom_MS>
<MoveToPrimary_MS>
<MoveToSecondary_MS>

Be aware, the machines will need to be remotely manageable before you can run the below script. More information here: https://kevinholman.com/2010/02/20/how-to-get-your-agents-back-to-remotely-manageable-in-scom/

# ===============================
# Author: Blake Drumm ([email protected])
# Created: September 30th, 2022
# Modified: September 30th, 2022
# Script location: https://github.com/blakedrumm/SCOM-Scripts-and-SQL/blob/master/Powershell/Agents%20Failover/Set-AgentFailover.ps1
# ===============================

Import-Module OperationsManager

#===================================================================
#region Script Variables

#We will look for all Agents Managed by this Management Server.
$movefromManagementServer = Get-SCOMManagementServer -Name "<MoveFrom_MS>"

#Primary Management Server
$movetoPrimaryMgmtServer = Get-SCOMManagementServer -Name "<MoveToPrimary_MS>"

#Secondary Management Server
$movetoFailoverMgmtServer = Get-SCOMManagementServer -Name '<MoveToSecondary_MS>'

#Gather the System Center Agent Class so we can get the Agents:
$scomAgent = Get-SCOMClass | Where-Object{ $_.name -eq "Microsoft.SystemCenter.Agent" } | Get-SCOMClassInstance

#endregion Variables
#===================================================================

#===================================================================
#region MainScript
$i = 0
foreach ($agent in $scomAgent)
{
	$i++
	$i = $i
	
	#Check the name of the current
	$scomAgentDetails = Get-SCOMAgent -ManagementServer $movefromManagementServer | Where { $_.DisplayName -match $agent.DisplayName }
	if ($scomAgentDetails)
	{
		#Remove Failover Management Server
		Write-Output "($i/$($scomAgent.count)) $($agent.DisplayName)`n`t`tRemoving Failover"
		$scomAgentDetails | Set-SCOMParentManagementServer -FailoverServer $null | Out-Null
		#Set Primary Management Server
		Write-Output "`t`tCurrent Primary: $($movefromManagementServer.DisplayName)`n`t`tUpdating Primary to: $($movetoPrimaryMgmtServer.DisplayName)"
		$scomAgentDetails | Set-SCOMParentManagementServer -PrimaryServer $movetoPrimaryMgmtServer | Out-Null
		if ($movetoFailoverMgmtServer -and $movetoFailoverMgmtServer -ne '<MoveToSecondary_MS>')
		{
			#Set Secondary Management Server
			Write-Output "               $($agent.DisplayName) Failover: $($movetoFailoverMgmtServer.DisplayName)`n`n"
			$scomAgentDetails | Set-SCOMParentManagementServer -FailoverServer $movetoFailoverMgmtServer | Out-Null
		}
	}
	else
	{
		Write-Verbose "Unable to locate any data."
	}
}
Write-Output "Script completed!"
#endregion MainScript
#===================================================================

How to change SQL Server Reporting Services to use Kerberos instead of NTLM

April 19, 2023

I had a customer ask if SCOM Reporting requires NTLM or not. So when I started digging for information on this, I found this blog post showing how to change SSRS to use Kerberos instead of NTLM: How to change SCOM reporting to use Kerberos instead of NTLM - Cloud management at your fingertips (mscloud.be)

I found that the article is missing some screenshots and seems to no longer be maintained. So, I thought it would be a good time to publish something for this.

:book: What is Kerberos and NTLM

If you are wondering, what is NTLM? What is Kerberos? How do these help or hurt? This article goes into detail to explain the differences:
Difference between Kerberos and NTLM

:exclamation: SCOM Reporting Installation Quirk

Something you need to take into account is that the SCOM installation for Reporting will overwrite the rsreportserver.config file and reverse any changes you may perform (which will set authentication back to NTLM). Which means you will need to apply the steps to change to Kerberos again AFTER SCOM Reporting Installation.

:page_with_curl: How to change SSRS Authentication

:memo: Manually change SSRS Authentication

To change the report server authentication settings, edit the value in the rsreportserver.config file:
```
C:\Program Files\Microsoft SQL Server Reporting Services\SSRS\ReportServer\rsreportserver.config
```
Replace RSWindowsNTLM with RSWindowsNegotiate in the config file.

My config file: https://files.blakedrumm.com/rsreportserver.config

:zap: Automatically change SSRS Authentication with PowerShell

This script will allow you to automatically set the Authentication for SSRS to Windows Negotiate instead of NTLM:

$RS = "root\Microsoft\SqlServer\ReportServer\" + ((Get-CimInstance -Namespace 'root\Microsoft\SqlServer\ReportServer' -ClassName __Namespace).CimInstanceProperties).Value | Select-Object -First 1
$RSV = $RS + "\" + (Get-CimInstance -Namespace $RS -ClassName __Namespace -ErrorAction Stop | Select-Object -First 1).Name + "\Admin"
$RSInfo = Get-CimInstance -Namespace $RSV -ClassName MSReportServer_ConfigurationSetting -ErrorAction Stop
(Get-Content ($RSInfo).PathName).Replace("<RSWindowsNTLM","<RSWindowsNegotiate") | Out-File ($RSInfo).PathName -Encoding UTF8

:page_with_curl: Set SSRS SPN’s

Using RSWindowsNegotiate will result in a Kerberos authentication error if you configured the Report Server service to run under a domain user account and you did not register a Service Principal Name (SPN) for the account. For more information on SPN’s for SSRS see:
Register a Service Principal Name (SPN) for a Report Server - SQL Server Reporting Services (SSRS) | Microsoft Learn

:memo: Check SSRS SPN’s

The following command allows you to check the SPN’s for the SSRS Server, you will need to replace the username with the service account running SSRS:
```
setspn -l <domain>\<domain-user-account>
```

:memo: Manually set SSRS SPN’s

The following command allows you to set the SPN’s for the SSRS Server:

setspn -s http/<computer-name>.<domain-name> <domain>\<domain-user-account>

:zap: Automatically set SSRS SPN’s

The following script allows you to automatically set the SSRS SPN’s:

$RS = "root\Microsoft\SqlServer\ReportServer\" + ((Get-CimInstance -Namespace 'root\Microsoft\SqlServer\ReportServer' -ClassName __Namespace).CimInstanceProperties).Value | Select-Object -First 1
$RSV = $RS + "\" + (Get-CimInstance -Namespace $RS -ClassName __Namespace -ErrorAction Stop | Select-Object -First 1).Name + "\Admin"
$RSInfo = Get-CimInstance -Namespace $RSV -ClassName MSReportServer_ConfigurationSetting -ErrorAction Stop

# Get Computer FQDN
$DNSComputerName = $env:COMPUTERNAME + '.' + (Get-CimInstance Win32_ComputerSystem).Domain

# Set SSRS SPN
$setspn = setspn -s "http/$DNSComputerName" "$($RSInfo.WindowsServiceIdentityActual)"

if ($setspn -match "^Duplicate SPN found, aborting operation!$")
{
    Write-Output "SPN is already set or duplicate SPN found."
}

# Check SSRS SPN
setspn -l $RSInfo.WindowsServiceIdentityActual

:closed_lock_with_key: Set the Kerberos AES Encryption Support

Enable “This account supports Kerberos AES 128 bit encryption.” and “This account supports Kerberos AES 256 bit encryption.”

:memo: Manually set the Kerberos AES Encryption Support

Open a run box (Windows Key + R).
Type in dsa.msc and press Enter.
Select the domain, right click and select Find.
Locate the Service Account for your Data Reader Account and Data Access Service Account.
Open the properties and go to the Account Tab.
Scroll down in the Account options: section and locate “This account supports Kerberos AES 128 bit encryption.” and “This account supports Kerberos AES 256 bit encryption.”

:zap: Set the Kerberos AES Encryption Support via PowerShell

Enable Kerberos AES Encryption for OMDAS (Data Access Service Account) and OMRead (Data Reader account) via PowerShell, you only have to modify the bottom line in the below script to your UserNames:

<#
.SYNOPSIS
This PowerShell script is designed to manage Kerberos AES encryption settings for specified user accounts in Active Directory.

.DESCRIPTION
The Set-KerberosAESEncryption function enables or disables Kerberos Advanced Encryption Standard (AES) 128-bit and 256-bit encryption for given Active Directory user accounts. It provides options to either enable or disable these encryption settings based on the provided parameters.

The script uses the System.DirectoryServices.AccountManagement namespace for accessing and modifying the properties of Active Directory user accounts. It updates the 'msDS-SupportedEncryptionTypes' property to set or unset AES 128 and AES 256 encryption types.

.PARAMETERS
- UserNames: An array of user account names (strings) in Active Directory for which the encryption settings will be modified.
- EnableEncryption: A switch parameter. When used, the script sets the Kerberos AES 128 and 256-bit encryption for the specified user accounts.
- DisableEncryption: A switch parameter. When used, the script unsets the Kerberos AES 128 and 256-bit encryption for the specified user accounts.

.OUTPUT
The script outputs a table with the following columns for each user account processed:
- UserName: The name of the user account.
- PreviousEncryptionTypes: The encryption types (AES 128, AES 256, both, or Not Set) before the script execution.
- UpdatedEncryptionTypes: The encryption types after the script execution.
- UpdateApplied: Indicates whether an update was applied ('Yes' or 'No').

.EXAMPLE
To enable Kerberos AES encryption for users 'OMDAS' and 'OMRead':
Set-KerberosAESEncryption -UserNames @("OMDAS", "OMRead") -EnableEncryption

.EXAMPLE
To disable Kerberos AES encryption for users 'OMDAS' and 'OMRead':
Set-KerberosAESEncryption -UserNames @("OMDAS", "OMRead") -DisableEncryption

.NOTES
Ensure you have appropriate permissions to modify user properties in Active Directory before running this script. It's recommended to test the script in a non-production environment first.

  Author: Blake Drumm ([email protected])
  Date Created: November 21st, 2023

#>
# Add required assembly references
Add-Type -AssemblyName System.DirectoryServices.AccountManagement

# Function to manage Kerberos AES encryption settings
function Set-KerberosAESEncryption {
    param (
        [string[]]$UserNames,
        [switch]$EnableEncryption,
        [switch]$DisableEncryption
    )

    if (-NOT $EnableEncryption -and -NOT $DisableEncryption)
    {
        Write-Warning "Missing required parameter: -EnableEncryption OR -DisableEncryption"
        break
    }

    # Define encryption type values
    $AES128 = 0x08
    $AES256 = 0x10

    # Combine AES 128 and AES 256 bit encryption support
    $newEncryptionTypes = $AES128 -bor $AES256

    # Function to convert encryption type value to readable format
    function ConvertTo-EncryptionTypeName {
        param ($encryptionTypeValue)
        switch ($encryptionTypeValue) {
            0       { "Not Set" }
            0x08   { "AES 128" }
            0x10   { "AES 256" }
            24     { "AES 128, AES 256" }
            Default { "Unknown" }
        }
    }

    # Initialize an array to store output data
    $outputData = @()

    foreach ($userName in $UserNames) {
        try {
            # Create a principal context for the domain
            $context = New-Object System.DirectoryServices.AccountManagement.PrincipalContext([System.DirectoryServices.AccountManagement.ContextType]::Domain)

            # Find the user
            $user = [System.DirectoryServices.AccountManagement.UserPrincipal]::FindByIdentity($context, $userName)

            if ($user -ne $null) {
                # Access the underlying DirectoryEntry
                $de = $user.GetUnderlyingObject()

                # Get current encryption types
                $currentEncryptionTypes = $de.Properties["msDS-SupportedEncryptionTypes"].Value
                $currentEncryptionTypes = if ($currentEncryptionTypes -ne $null) { $currentEncryptionTypes -as [int] } else { 0 }

                # Determine if update is needed
                $changeNeeded = ($EnableEncryption -and $currentEncryptionTypes -ne $newEncryptionTypes) -or
                                ($DisableEncryption -and $currentEncryptionTypes -ne 0)

                # Update encryption types if needed
                if ($changeNeeded) {
                    $de.Properties["msDS-SupportedEncryptionTypes"].Value = if ($EnableEncryption) { $newEncryptionTypes } else { $null }
                    $de.CommitChanges()
                }

                # Add data to the output array
                $updatedEncryptionTypes = $de.Properties["msDS-SupportedEncryptionTypes"].Value
                $updatedEncryptionTypes = if ($updatedEncryptionTypes -ne $null) { $updatedEncryptionTypes -as [int] } else { 0 }

                $outputData += New-Object PSObject -Property @{
                    UserName = $userName
                    PreviousEncryptionTypes = ConvertTo-EncryptionTypeName -encryptionTypeValue $currentEncryptionTypes
                    UpdatedEncryptionTypes = ConvertTo-EncryptionTypeName -encryptionTypeValue $updatedEncryptionTypes
                    UpdateApplied = if ($changeNeeded) { "Yes" } else { "No" }
                }
            } else {
                Write-Warning "User $userName not found."
            }
        } catch {
            Write-Host "Error updating user $userName`: $_"
        }
    }

    # Output the results in a single table
    $outputData | Format-Table -AutoSize
}

# Example usage of the function
Set-KerberosAESEncryption -UserNames "OMDAS", "OMRead" -EnableEncryption

:memo: Restart SSRS Service

The following script allows you to restart the SSRS Service:

  $RS = "root\Microsoft\SqlServer\ReportServer\" + ((Get-CimInstance -Namespace 'root\Microsoft\SqlServer\ReportServer' -ClassName __Namespace).CimInstanceProperties).Value | Select-Object -First 1
  $RSV = $RS + "\" + (Get-CimInstance -Namespace $RS -ClassName __Namespace -ErrorAction Stop | Select-Object -First 1).Name + "\Admin"
  $RSInfo = Get-CimInstance -Namespace $RSV -ClassName MSReportServer_ConfigurationSetting -ErrorAction Stop

  # Restart SSRS Service
  Restart-Service ($RSInfo).ServiceName

All Operations Manager Event ID's

March 28, 2023

	Event	Source	Log Name	LongID	Description

Link to download excel file: https://files.blakedrumm.com/All%20SCOM%20Event%20Ids.xlsx

SCOM Certificate Checker Script

February 27, 2023

:book: Introduction

This tool will allow you to check your SCOM Certificate. It is very efficient and has been improved upon over time. You may edit line 751 to allow you to change what happens when you run from Powershell ISE. Copying and pasting the script to Powershell ISE after you run MOMCertImport on a certificate is the most common way to run the script, which requires no arguments or modifications. Just run the script and you will see where the issue may be.

:memo: Authors

Tyson Paul (https://monitoringguys.com/)
Lincoln Atkinson (https://latkin.org/blog/)
Mike Kallhoff
Blake Drumm (https://blakedrumm.com/)

:page_with_curl: Where to get it

Test-SCOMCertificate.ps1 :arrow_left: Direct Download Link
or
Personal File Server - Test-SCOMCertificate.ps1 :arrow_left: Alternative Download Link
or
Personal File Server - Test-SCOMCertificate.txt :arrow_left: Text Format Alternative Download Link

:classical_building: Argument List

Argument List	Description
-All	Check All Certificates in Local Machine Store.
-Servers	Each Server you want to Check SCOM Certificates on.
-SerialNumber	Check a specific Certificate serial number in the Local Machine Personal Store. Not reversed.
-OutputFile	Where to Output the File (txt, log, etc) for Script Execution.

:question: Examples

Example 1

Check the certificate you have currently configured for SCOM on the local machine:
  PS C:\> .\Test-SCOMCertificate.ps1
Example 2

Check for a specific Certificate Serial number in the Local Machine Personal Certificate store:
  PS C:\> .\Test-SCOMCertificate.ps1 -SerialNumber 1f00000008c694dac94bcfdc4a000000000008
Example 3

Check all certificates on the local machine:
  PS C:\> .\Test-SCOMCertificate.ps1 -All
Example 4

Check All Certificates on 4 Servers and outputting the results to C:\Temp\Output.txt:
  PS C:\> .\Test-SCOMCertificate.ps1 -Servers ManagementServer1, ManagementServer2.contoso.com, Gateway.contoso.com, Agent1.contoso.com -All -OutputFile C:\Temp\Output.txt

Example of Failure

SCOM SDK Crashing - Cannot Generate SSPI Context

January 24, 2023

:book: Introduction

I had a case recently for a customer that is having issues when opening the SCOM Console. Ultimately this was due to the SCOM SDK Service crashing, with the following Event in the Operations Manager Event Log:

Log Name:      Operations Manager
Source:        OpsMgr Management Configuration
Date:          1/24/2023 12:11:06 PM
Event ID:      29112
Task Category: None
Level:         Error
User:          N/A
Computer:      MS01.contoso.com
Description:
OpsMgr Management Configuration Service failed to execute bootstrap work item 'ConfigurationDataProviderInitializeWorkItem' due to the following exception



System.Data.SqlClient.SqlException (0x80131904): The target principal name is incorrect.  Cannot generate SSPI context.
   at System.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, DbConnectionPool pool, String accessToken, Boolean applyTransientFaultHandling, SqlAuthenticationProviderManager sqlAuthProviderManager)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
   at System.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
   at System.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ConnectionManagementOperation.Execute()
   at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
   at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.CmdbDataProvider.Initialize()
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.ConfigurationDataProviderInitializeWorkItem.ExecuteWorkItem()
   at Microsoft.EnterpriseManagement.ManagementConfiguration.Interop.ConfigServiceEngineWorkItem.Execute()
ClientConnectionId:c0z7eb24-124d-46ed-xe78-36q2ba9f7949

:page_with_curl: How to fix #1

In order to resolve this issue for my customer, first we verified if RC4 was disabled (RC4 was disabled) (More Information here: Decrypting the Selection of Supported Kerberos Encryption Types - Microsoft Tech Community). Then we had to verify the user account running the Operations Manager SQL Service has AES Attributes enabled. Navigate to the user object in Active Directory and verify that the Account options have the following:

Check This account supports Kerberos AES 128 bit encryption.
Check This account supports Kerberos AES 256 bit encryption.

We were no longer having an issue with the SDK Service crashing after this change.

Relevant Article: https://learn.microsoft.com/system-center/scom/install-with-rc4-disabled#configure-the-encryption-types-allowed-for-kerberos

:page_with_curl: How to fix #2

You may also resolve the above issue by fixing SPN information for the MSSQLSvc (thank you Joana da Rocha Carvalho!):

nslookup the SCOM SQL Database Instance(s)
Update the SPN information for the SQL Service Account with the following command:
```
setspn -S MSSQLSvc/SERVER.contoso.com:1433 emea\SQLSVCaccount
```
Check SPN for SQL Service Account:
```
setspn -L emea\SQLSVCaccount
```

SCOM Agent Installation Error - Microsoft ESENT Keys are required

January 5, 2023

:book: Introduction

I had a case recently for a customer that is having issues when installing an SCOM Agent (MOMAgent.msi) with an warning which stated:

Microsoft ESENT Keys are required to install this application.
Please see the release notes for more information.

:page_with_curl: How to fix

In order to resolve this issue for my customer, we had to run the installer as administrator. Which we accomplished by opening a Powershell window as administrator and navigating to the directory of the SCOM Agent installer (MOMAgent.msi).

Create Custom PowerShell CPU Monitor

December 17, 2022

:book: Introduction

Recently I had a case and I needed to assist my customer with a CPU Monitor that targets all Windows Computers. The customer is using the PowerShell Authoring Community MP by Cookdown. I will document my steps to get this working as intended.

:red_circle: Prerequisites

Import Cookdown PowerShell Monitoring - Community Management Pack (https://www.cookdown.com/scom-essentials/powershell-authoring/)

:heavy_check_mark: Example MP

You can get a copy of the Management Pack I created in this guide, here:
Custom Monitor for CPU Usage :arrow_left: Direct Download Link

:notebook: Note

This guide will require you to edit the Management Pack directly, be aware that you need to be careful when editing any Management Pack to ensure you do not accidently change the wrong thing.

:page_with_curl: Start of Guide

Step 1. Create a new Unit Monitor

You will need to create a new Unit Monitor.

Open the SCOM Console and navigate to the Authoring Tab -> Monitors -> Right Click Monitors -> Hover over Create a Monitor and select Unit Monitor…:
Open the Scripting folder -> PowerShell Based -> PowerShell Script Two State Monitor (Community)
Select the appropriate Management Pack to save the new Monitor.
It is HIGHLY recommended to save to a new MP instead of any existing MPs!
Select Next >
Type in an appropriate name / description, I used: Custom CPU Monitor
Change the Monitor Target to Windows Computer
Click Next >
Run every 15 minutes by default, this is a good interval to start with. Click Next >
Type in the File Name you want to use, I used: my_custom_cpu_monitoring_script.ps1

Copy and Paste the below script into the Script section and click Next >:

$api = New-Object -ComObject "MOM.ScriptAPI";
$PropertyBag = $api.CreatePropertyBag();
[int]$Result = [int]((Get-Counter "\Processor(_Total)\% Processor Time").CounterSamples.CookedValue);
$PropertyBag.AddValue("CPUUsage", $Result);
$PropertyBag

Copy and paste the below, when everything below has been copied, click Next >:
- Parameter Name column: Property[@Name="CPUUsage"]
- Operator: Greater than or equal to
- Value: 50
Copy and paste the below, when everything below has been copied, click Next >:
- Parameter Name column: Property[@Name="CPUUsage"]
- Operator: Less than
- Value: 50
Optional: Change Unhealthy from Warning to Critical.

Click the Generate alerts for this monitor checkbox and change the Alert Description text to this:

The CPU has reached at or above 50% usage.

      Current CPU usage is: $Data/Context/Property[@Name="CPUUsage"]$%

Done creating the unit monitor, now onto the more advanced stuff!

Step 2. Modify the Management Pack

We will need to modify the Management Pack in order to allow the expressions to evaluate correctly. The Monitor is setup to use string values instead of integers, which will cause problems when we try to evaluate the health of the monitor.

Go to Administration Tab -> Management Packs -> Installed Management Packs -> Search for the Management Pack where your Monitor is saved. Select and export the Management Pack to any location.
Navigate to the exported Management Pack xml file, open the MP XML with Notepad.
Find and replace:
<XPathQuery>Property[@Name="CPUUsage"]</XPathQuery>
- Replace any occurrences with the following:
  <XPathQuery Type="Integer">Property[@Name="CPUUsage"]</XPathQuery>
- The above change allows us to interpret the output as integer, instead of string.
Find and replace:
<Value Type="String">50</Value>
- Replace any occurrences with the following:
  <Value Type="Integer">50</Value>
- The above change allows us to interpret the output as integer, instead of string.
Modify Line 5 (the version of the Management Pack) in the xml file from:
<Version>1.0.0.0</Version>
to
<Version>1.0.0.1</Version>
Save the Management Pack XML and import the Management Pack back into your environment.

Leave some feedback if this helped you! :v:

Reconfigure System Center Operations Manager for Database Move Tool

November 23, 2022

:book: Introduction

Welcome to the official page for the Reconfigure System Center Operations Manager for Database Move Tool. This tool is compatible with all versions of Operations Manager and is designed to help you manage SQL connections used by Operations Manager.

This script automates the steps outlined here: https://learn.microsoft.com/system-center/scom/manage-move-opsdb

:spiral_notepad: Note

The script does not edit the Reporting Services Data Warehouse Data Source -> Connection String
(shown here (change to https if you are using SSL Certificates): http://<reportingServerURL>/Reports/manage/catalogitem/properties/Data%20Warehouse%20Main)
You will need to update the Connection String value to the correct data, here is an example of mine:
data source=SQL01-2022;initial catalog=OperationsManagerDW;Integrated Security=SSPI

Features

Update / Verify the System Center Operations Manager SQL
- SQL Instance Configuration for CLR and SQL Service Broker.
- Database Tables for data related to SQL Connections in SCOM.
Update / Verify the Registry data
- Local SCOM Management Servers.
- Remote SCOM Management Servers.
Update / Verify the Configuration File
- Local SCOM Management Servers.
- Remote SCOM Management Servers.
Language Changer combo-box which adds automatic support for 8 Languages:
- English
- Spanish (Español)
- Arabic (عربي)
- Turkish (Türkçe)
- Chinese (中国人)
- Portuguese (Português)
- French (Français)
- German (Deutsch)
  A HUGE THANK YOU to the following individuals for assisting with translation in the different languages:
- Rafael Lujan (Spanish)
- Fayez Al Saadi (Arabic)
- Murat Coskun (Turkish)
- Wendi Cai (Chinese)
- Wagner da Costa Carvalho (Portuguese)
- Joana da Rocha Carvalho (French)
- Stefan Wuchenauer (German)

Requirements

System Center Operations Manager installation
Powershell 5

:arrow_down_small: Latest Version

Check the changelog for the latest version: Latest Changes

:page_with_curl: How to Use

https://aka.ms/SCOM-DB-Move-Tool

You have multiple ways to download the SCOM Reconfigure DB Move GUI Tool:

Download and install the MSI: MSI Download
Download and run the EXE: EXE Downloads
Download or Copy the Powershell Script to Powershell ISE: Powershell Script
Download or Copy the Powershell Script to Powershell ISE: Text Format Alternative Download Link

The script by default will attempt to gather the current database connection from the local registry. If it is unable to locate the registry keys the Database Connection box will be empty. If it is empty you will need to manually type the values in here. The Values to Set section is required for the script to run and you will need to manually populate these fields. The Management Servers section is also required for you to be able to set which Management Servers to update the Database information on.

This script will log actions to the Application Event Log. Look for the Event Source: SCOMDBMoveTool

:page_facing_up: More Information

You will get prompted each time you run the script to accept the license agreement, unless you select do not ask me again, when you select this it will save a file to your ProgramData Directory:

C:\ProgramData\SCOM-DBMoveTool-AgreedToLicense.log

Attribution for the icon: Database icons created by manshagraphics - Flaticon

Remove Data from the SCOM Database Instantly - The PowerShell Way!

October 2, 2022

:book: Introduction

I have had many cases where I’ve had to run the following SQL Queries by Kevin Holman: Deleting and Purging data from the SCOM Database (kevinholman.com) (A big thank you to Kevin Holman for his guide!)

After the 6th or 7th time running the tsql queries I decided that I should automate this process in Powershell, and make it very easy to do this automatically for as many servers as needed. This script will query the Operations Manager database and run all the steps in Kevin Holman’s queries. The script will ask questions at each step to verify the action is correct. You have to answer Y or N before the script will proceed.

:exclamation: Important!

It is recommended to only run the script when requested by a Microsoft Support Engineer. This script can be dangerous if used incorrectly.

If you absolutely need to run the script, make sure you have full backups of your SCOM SQL Instances / Databases!

:arrow_down: How to get it

You can get a copy of the script here:
Remove-SCOMBaseManagedEntity.ps1 :arrow_left: Direct Download Link
or
Personal File Server - Remove-SCOMBaseManagedEntity.ps1 :arrow_left: Alternative Download Link
or
Personal File Server - Remove-SCOMBaseManagedEntity.txt :arrow_left: Text Format Alternative Download Link

:notebook: Note

You may edit line 769 in the script to change what happens when the script is run without any arguments or parameters, this also allows you to change what happens when the script is run from the Powershell ISE.

:classical_building: Argument List

Parameter	Alias	ValueFromPipeline	Type	Description
AssumeYes	yes		Switch	Optionally assume yes to any question asked by this script.
Database			String	The name of the OperationsManager Database for SCOM.
DontStop	ds		Switch	Optionally force the script to not stop when an error occurs connecting to the Management Server.
Id			Array	You may provide any Base Managed Entity Id’s to be deleted specifically from the Operations Manager Database.
InputFile			String	A file you would like to have the script input as a list of Agents. The file should have a new line for each server.
ManagementServer	ms		String	SCOM Management Server that we will remotely connect to. If running on a Management Server, there is no need to provide this parameter.
Name			Array	The Base Managed Entity Display Name of the object you are wanting to delete from the Operations Manager Database.
Servers		True	Array	Each Server (comma separated) you want to Remove related BME ID’s related to the Display Name in the OperationsManager Database. This will also remove from Agent Managed.
SqlServer			String	SQL Server/Instance, Port that hosts OperationsManager Database for SCOM.

:page_with_curl: How to use it

Example 1

Remove SCOM BME Related Data from the OperationsManager DB, for Agent1 in the Management Group:
Get-SCOMAgent -Name Agent1.contoso.com | %{.\Remove-SCOMBaseManagedEntity.ps1 -Servers $_.DisplayName}
Example 2

Remove SCOM BME Related Data for 2 Agent machines:
.\Remove-SCOMBaseManagedEntity.ps1 -Servers IIS-Server.contoso.com, WindowsServer.contoso.com
Example 3

Remove SCOM BME Related Data for 2 Agent machines automatically with no prompts / questions:
.\Remove-SCOMBaseManagedEntity.ps1 -Servers IIS-Server.contoso.com, WindowsServer.contoso.com -AssumeYes
Example 4

Remove SCOM BME IDs from the Operations Manager Database:
.\Remove-SCOMBaseManagedEntity.ps1 -Id C1E9B41B-0A35-C069-16EB-00AC43BB9C47, CB29ECDE-BCE8-2213-D5DD-0353116EDA6B
Example 5

If you need to input a file that contains Agents (CSV or TXT):
.\Remove-SCOMBaseManagedEntity.ps1 -InputFile C:\Temp\List_Of_Servers.txt

Leave some feedback if this helped you! :v:

Data Reader account provided is not same as that in the management group

September 19, 2022

_{Thank you to Luis Décio for the original creation of the KB!}

During installation of SCOM Reporting services, you may see the following error: Data Reader account provided is not same as that in the management group.

How to fix

Reconfigure the “Data Warehouse Report Deployment Account” Profile

Set the Data Warehouse Report Deployment Account to the Class “Collection Server” and “Data Warehouse Synchronization Server”

Reconfigure the “Data Warehouse Account” Profile

Data Warehouse Action Account to the Class “Collection Server”, “Data Warehouse Synchronization Server”, “Data Set” and “Operations Manager APM Data Transfer Service”

After configuring the above, the SCOM Reporting component installed successfully with the Data Warehouse Report Deployment Account as expected.

Automate with Powershell

You can use Powershell to automate fixing the DW RunAs Profiles. I would suggest that you remove all RunAs accounts from both RunAs profiles (“Data Warehouse Account” and “Data Warehouse Report Deployment Account”) prior to running the below Powershell script.

Blog post from Udish Mudiar: https://udishtech.com/associate-scom-data-warehouse-profile-using-powershell/

Here is my own take on Udish’s Powershell script:
https://github.com/blakedrumm/SCOM-Scripts-and-SQL/blob/master/Powershell/Quick%20Fixes/Resolve-DWRunAsProfileAssociation.ps1

# Original Author: Udish Mudiar
# Original Blog Post: https://udishtech.com/associate-scom-data-warehouse-profile-using-powershell/
# =========================================================================================================================
# Modified by: Blake Drumm ([email protected])
# Last Modified: March 8th, 2024
# Blog Post: https://blakedrumm.com/blog/data-reader-account-provided-is-not-same-as-that-in-the-management-group/

function Invoke-TimeStamp
{
	$TimeStamp = (Get-Date).DateTime
	return "$TimeStamp - "
}
Write-Host "`n`n------------------------------------------------------------" -ForegroundColor Green
#Associate Run As Account association in Data Warehouse and Report Deployment Run As Profile.
Write-Output "$(Invoke-TimeStamp)Script started"

Import-Module OperationsManager

#Get the run as profiles
$DWActionAccountProfile = Get-SCOMRunAsProfile -DisplayName "Data Warehouse Account"
$ReportDeploymentProfile = Get-SCOMRunAsProfile -DisplayName "Data Warehouse Report Deployment Account"

#Get the run as accounts
$DWActionAccount = Get-SCOMrunAsAccount -Name "Data Warehouse Action Account"
$DWReportDeploymentAccount = Get-SCOMrunAsAccount -Name "Data Warehouse Report Deployment Account"

#Get all the required classes
$CollectionServerClass = Get-SCOMClass -DisplayName "Collection Server"
$DataSetClass = Get-SCOMClass -DisplayName "Data Set"
$APMClass = Get-SCOMClass -DisplayName "Operations Manager APM Data Transfer Service"
$DWSyncClass = Get-SCOMClass -DisplayName "Data Warehouse Synchronization Server"

#Setting the association
Write-Output "$(Invoke-TimeStamp)Setting the Run As Account Association for Data Warehouse Account Profile"
$error.Clear()
try
{
	if ($APMClass)
	{
		Set-SCOMRunAsProfile -ErrorAction Stop -Action "Add" -Profile $DWActionAccountProfile -Account $DWActionAccount -Class $CollectionServerClass, $DataSetClass, $APMClass, $DWSyncClass
	}
	else
	{
		Set-SCOMRunAsProfile -ErrorAction Stop -Action "Add" -Profile $DWActionAccountProfile -Account $DWActionAccount -Class $CollectionServerClass, $DataSetClass, $DWSyncClass
	}
	Write-Output "$(Invoke-TimeStamp)Completed Successfully!"
}
catch
{
	Write-Output "$(Invoke-TimeStamp)Unable to set the RunAs accounts, try removing all accounts from inside the RunAs Profile (`"Data Warehouse Account`"), and run the script again."
	Write-Warning "$(Invoke-TimeStamp)$error"
}
Write-Output "$(Invoke-TimeStamp)Setting the Run As Account Association for Data Warehouse Report Deployment Account Profile"
$error.Clear()
try
{
	Set-SCOMRunAsProfile -ErrorAction Stop -Action "Add" -Profile $ReportDeploymentProfile -Account $DWReportDeploymentAccount -Class $CollectionServerClass, $DWSyncClass
	Write-Output "$(Invoke-TimeStamp)Completed Successfully!"
}
catch
{
	Write-Output "$(Invoke-TimeStamp)Unable to set the RunAs accounts, try removing all accounts from inside the RunAs Profile (`"Data Warehouse Report Deployment Account`"), and run the script again."
	Write-Warning "$(Invoke-TimeStamp)$error"
}

Write-Output "$(Invoke-TimeStamp)Script ended"
Write-Host '------------------------------------------------------------' -ForegroundColor Green

SCOM Uptime Availability Report

August 17, 2022

In order to properly report on the uptime of a machine, you will need to target the Health Service Watcher object class. This class will allow you to return the availability of the machine itself, and not all of the rollup monitors underneath that may affect the report accuracy.

Open SCOM Console, Click on the Reporting tab
Click on Microsoft Generic Report Library
Open the Availability Report
Add Object:
Click on Options… -> Click on Add -> Type Health Service Watcher in the search field -> Add Health Service Watcher:
Click OK on the Options Window:
Type in the machines you want to discover (or you can just press search and it will return all related items):
Modify the time range and run the report.
Resulting output:

:notebook: Note

Quick tip, you can modify the parameters for the report at any time with the Show or Hide Parameter Area button:

Dynamic Agent Health Service Watcher Group

If you would like to make it easier to run the report again, you can create a group for the Health Service Watchers. Just create a group in Authoring -> Groups, with the following Dynamic Member Rule:

Membership Result:

Event 26319 - Assembly Trust Issues with SQL 2017+

August 5, 2022

_{Thank you to Lorne Sepaugh for the original creation of the KB!}

Symptom

I had a customer today that had an issue with being unable to open any SCOM Console without receiving the following error:

Date: 6/8/2022 1:57:24 AM
Application: Operations Manager
Application Version: 10.19.10505.0
Severity: Error
Message: 

An error occurred in the Microsoft .NET Framework while trying to load assembly id 65537. The server may be running out of resources, or the assembly may not be trusted. Run the query again, or check documentation to see how to solve the assembly trust issues. For more information about this error: 
System.IO.FileLoadException: Could not load file or assembly 'microsoft.enterprisemanagement.sql.userdefineddatatype, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. An error relating to security occurred. (Exception from HRESULT: 0x8013150A)
System.IO.FileLoadException: 
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean forIntrospection)
   at System.Reflection.RuntimeAssembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.Load(String assemblyString)

This error is due to a change in the SQL 2017 security approach for CLR, as stated in the SQL Server 2017 docs:

There are two assemblies used by SCOM that are marked as UNSAFE and not allowed to run by default in one of these scenarios - as such we need to mark them as safe and trusted on each server instance. The assemblies are:

Microsoft.EnterpriseManagement.SQL.DataAccessLayer
Microsoft.EnterpriseManagement.SQL.UserDefinedDataType

How to Resolve

Prerequisites

*If using availability groups, this is to be completed after the databases are added

Ensure that CLR is enabled on all SQL Server instances with this script:

Sp_configure 'show advanced options', 1; 
GO 
RECONFIGURE; 
GO 
Sp_configure 'clr enabled', 1; 
GO 
RECONFIGURE; 
GO

Make sure that you have SQL admin access, or a DBA on hand
(Optional) Stop all the SCOM services on each management server - don’t forget to restart when finished
Make sure you have a database backup

Step 1: Verify CLR Strict Security State

Run this query in SQL Management Studio:

SELECT * FROM sys.configurations WHERE name LIKE 'clr strict security'; 

You should get a return like this, “value_in_use” and “value” should be 1:

This table describes what the values mean:

Value	Description
0	Disabled - Provided for backwards compatibility. Disabled value is not recommended
1	Enabled - Causes the Database Engine to ignore the PERMISSION_SET information on the assemblies, and always interpret them as UNSAFE. Enabled is the default value for SQL Server 2017 (14.x)

Note

By default, CLR strict security will be OFF after upgrading to SQL Server 2017

If the value is 0 - check this doc for more info on how to set it to 1 - https://docs.microsoft.com/sql/database-engine/configure-windows/clr-strict-security?view=sql-server-2017

Step 2: Create the Trusted Assemblies

To create the Trusted Assemblies, run the below TSQL Query on each SQL 2017+ instance(s) hosting the Operations Manager Database:

USE master;
GO

-- First Trusted Assembly
DECLARE @clrName1 nvarchar(4000) = 'Microsoft.EnterpriseManagement.Sql.DataAccessLayer'
PRINT N'Trusted Assembly: ' + CAST(@clrName1 AS nvarchar(120))
DECLARE @hash1 varbinary(64) = 0xEC312664052DE020D0F9631110AFB4DCDF14F477293E1C5DE8C42D3265F543C92FCF8BC1648FC28E9A0731B3E491BCF1D4A8EB838ED9F0B24AE19057BDDBF6EC;

-- Drop trusted assembly if exists
IF EXISTS (select * from sys.trusted_assemblies where description = @clrName1)
BEGIN
PRINT N' - Dropping Trusted Assembly'
EXEC SYS.sp_drop_trusted_assembly @hash1
END

--Add to trusted assembly
PRINT N' - Adding Trusted Assembly'
EXEC sys.sp_add_trusted_assembly @hash = @hash1,
                                 @description = @clrName1;

PRINT N' '
-- Second Trusted Assembly
DECLARE @clrName2 nvarchar(4000) = 'Microsoft.EnterpriseManagement.Sql.UserDefinedDataType'
PRINT N'Trusted Assembly: ' + CAST(@clrName2 AS nvarchar(120))
DECLARE @hash2 varbinary(64) = 0xFAC2A8ECA2BE6AD46FBB6EDFB53321240F4D98D199A5A28B4EB3BAD412BEC849B99018D9207CEA045D186CF67B8D06507EA33BFBF9A7A132DC0BB1D756F4F491;

-- Drop trusted assembly if exists
IF EXISTS (select * from sys.trusted_assemblies where description = @clrName2)
BEGIN
PRINT N' - Dropping Trusted Assembly'
EXEC SYS.sp_drop_trusted_assembly @hash2
END

--Add to trusted assembly
PRINT N' - Adding Trusted Assembly'
EXEC sys.sp_add_trusted_assembly @hash = @hash2,
                                 @description = @clrName2;

You can verify the currently trusted assemblies with the following query:

USE OperationsManager;
GO
SELECT * FROM sys.assemblies
SELECT * FROM sys.trusted_assemblies

Once done on all SQL Server instance(s) that host the Operations Manager Database, restart the SCOM Console on the management servers and everything should load correctly.

Resolve SCOM Agent Deployment Error: 80070643

July 29, 2022

:book: Introduction

I had a case where my customer is experiencing an error on their SCOM Console when attempting to resolve an agent Pending Upgrade in Pending Management. The Agent needed to be upgraded from the SCOM 2019 Agent to the SCOM 2022 Agent. We reviewed the Log file: Agent1AgentInstall.Log located in the following directory: C:\Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\AgentLogs.

Discovery Wizard Error:

The Agent Management Operation Agent Install failed for remote computer Agent1.contoso.com. 

Install account: CONTOSO\SCOMAdmin 

Error Code: 80070643 

Error Description: Fatal error during installation. 

Microsoft Installer Error Description: 

For more information, see Windows Installer log file "C:\Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\AgentLogs\Agent1AgentInstall.log" on the Management Server.

:page_with_curl: How to fix

The MSI error highlighted below was the main cause for the installation failure:

Action start 16:18:53: _SuppressComputerReboot.
MSI (s) (AC:F0) [16:18:53:021]: Skipping action: SetIS_NETFRAMEWORK_472_OR_LATER_INSTALLED (condition is false)
MSI (s) (AC:F0) [16:18:53:021]: Doing action: LaunchConditions
Action ended 16:18:53: _SuppressComputerReboot. Return value 1.
Action start 16:18:53: LaunchConditions.
MSI (s) (AC:F0) [16:18:53:021]: Product: Microsoft Monitoring Agent – The .NET Framework 4.7.2 is required to install this application.

The .NET Framework 4.7.2 is required to install this application.
Action ended 16:18:53: LaunchConditions. Return value 3.
Action ended 16:18:53: INSTALL. Return value 3.

We also attempted a manual install and this will also show you the error:

After installing .NET Framework 4.7.2 as required for the SCOM 2022 Agent, the installation succeeded.
SCOM 2022 Agent Requirements

SCOM Unix/Linux Discovery Errors + How To Fix Them

June 22, 2022

Errors you may see

Error Example 1

DiscoveryResult.ErrorData type. Please file bug report - Parameter Name: s
When discovering a Unix/Linux machine, the wizard shows the machine as unmanageable in the discovery results with below error:

Unexpected DiscoveryResult.ErrorData type.  Please file bug report. 
ErrorData: System.ArgumentNullException 
Value cannot be null. 
Parameter name: s 
at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout) 
at System.Activities.WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs, TimeSpan timeout, WorkflowInstanceExtensionManager extensions) 
at Microsoft.SystemCenter.CrossPlatform.ClientActions.DefaultDiscovery.InvokeWorkflow(IManagedObject managementActionPoint, DiscoveryTargetEndpoint criteria, IInstallableAgents installableAgents) 

How to Fix

Sometimes this can happen because WinHTTP proxy settings have been configured on the management servers in the Unix/Linux Resource Pool, and the agent which we are trying to discover is not included in the Bypass List

Open a CMD prompt as Administrator on the management servers in the Unix/Linux Resource Pool and run the following command

netsh winhttp show proxy

If there is a WinHTTP proxy server configured, add the FQDN for the server which we are trying to discover in the Bypass List by running the following command

netsh winhttp set proxy proxy-server="<proxyserver:port>" bypass-list="*.ourdomain.com;*.yourdomain.com*;<serverFQDN>" 

Once the Bypass List has been configured, check if discovery of the agent is now successful

Note

You can disable WinHTTP Proxy by running the following command, this will remove a proxy server and configure “Direct Access”:
netsh winhttp reset proxy

Error Example 2

DiscoveryResult.ErrorData type. Please file bug report - Parameter name: lhs
When discovering a Linux machine, the wizard shows the machine as unmanageable in the discovery results with below error:

Discovery not successful
Message: Unspecified failure
Details: Unexpected DiscoveryResult.ErrorData type. Please file bug report.
ErrorData: System.ArgumentNullException
Value cannot be null.
Parameter name: lhs
at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout)
at System.Activities.WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs, TimeSpan timeout, WorkflowInstanceExtensionManager extensions)
at Microsoft.SystemCenter.CrossPlatform.ClientActions.DefaultDiscovery.InvokeWorkflow(IManagedObject managementActionPoint, DiscoveryTargetEndpoint criteria, IInstallableAgents installableAgents)

How to fix

Sometimes this can happen because of omsagent shell files in the installed kits folder.

Navigate to the following directory in file explorer:
C:\Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits

If there are omsagent files listed here, move them to a temporary directory outside of the SCOM files.

Once they have been moved from the DownloadedKits folder, retry discovery and discovery should now succeed (or you should get a different error which would indicate additional troubleshooting is needed such as sudoers, connectivity, etc.)

Error while installing SCOM 2019 Reporting

June 9, 2022

Issue Description

The issue I have seen is the following error message in the SCOM Reporting Services installer:

“Unable to connect to the Data Access service for this management server. Ensure the Data Access service is running and that the service, the management group, and setup are all the same version.”

The problem stems from an SCOM 2019 Update Rollup that was previously applied to the Management Server and this causes the SCOM Reporting Services to fail due to the Reporting Services installer expecting the RTM version to be present.

The regression was introduced in Update Rollup 1 for System Center Operations Manager 2019 (KB4533415)

The “Operations Manager Products” view in the Admin console did not update the Version column for the installed component version. This column now reflects the updated version of all the listed components.

Solution

The following may resolve the above error for you:

Start by connecting to the Operations Manager database via SQL Server Management Studio. (Create a backup of your Databases prior to any direct edits)

Get the current version of the Management Server you are connecting SSRS with note the version (we will use this later to revert the changes to the DB)

Get Version

 -- SCOM 2019 RTM
 -- 10.19.10050.0

 -- SCOM 2019 UR1
 -- 10.19.10311.0

 -- SCOM 2019 UR1 - Hotfix for Alert Management
 -- 10.19.10349.0

 -- SCOM 2019 UR2
 -- 10.19.10407.0

 -- SCOM 2019 UR2 - Hotfix for Event Log Channel
 -- 10.19.10475.0

 -- SCOM 2019 UR3
 -- 10.19.10505.0

 -- SCOM 2019 UR3 - Hotfix for Web Console
 -- 10.19.10550.0

 -- SCOM 2019 UR3 - Hotfix Oct 2021
 -- 10.19.10552.0

 select
     PrincipalName,
     Version
 from MTV_HealthService
 where
     IsManagementServer = 1 and 
     PrincipalName = 'MS01-2019.contoso.com'

We will run the following query to update the Management Server to the RTM version of SCOM 2019 (10.19.10050.0):
Update Version to RTM
```
update MTV_HealthService
set Version = '10.19.10050.0'
-- SCOM 2019 RTM
where PrincipalName = 'MS01-2019.contoso.com'
```
Install SCOM Reporting Services! :sunglasses: :thumbsup:
After SCOM Reporting Services installs, you will need to revert the changes to the SCOM SQL Operations Manager database, run the following SQL Query to return the Management Server version back to the version returned in Step 2:
Revert Change
```
update MTV_HealthService
set Version = '10.19.10552.0'
-- SCOM 2019 UR3 - Hotfix Oct 2021
where PrincipalName = 'MS01-2019.contoso.com'
```

Rebuild Performance Counters

June 6, 2022

How to Rebuild

Rebuild the Performance counters with the following Powershell Script, you can copy and paste the below script to a Powershell ISE Window running as Administrator:

Push-Location $PWD
$FirstPath = 'C:\Windows\System32'
$SecondPath = 'C:\Windows\SysWOW64'
cd $FirstPath
Write-Output '---------------------------------------------------------'
Write-Output "Recreating Performance Counters in: $FirstPath"
Write-Output ' - Running: lodctr /R'
lodctr /R
cd $SecondPath
Write-Output "`nRecreating Performance Counters in: $SecondPath"
Write-Output ' - Running: lodctr /R'
lodctr /R
Write-Output '---------------------------------------------------------'
Write-Output 'Resyncing the Performance Counters with Windows Management Instrumentation (WMI)'
Write-Output ' - Running: C:\Windows\System32\wbem\WinMgmt.exe /RESYNCPERF'
C:\Windows\System32\wbem\WinMgmt.exe /RESYNCPERF
Write-Output '---------------------------------------------------------'
Write-Output 'Restarting Service: Performance Logs & Alerts (pla)'
$error.Clear()
try
{
	Get-Service -Name "pla" | Restart-Service -ErrorAction Stop | Out-Null
}
catch
{
	Write-Warning "A Failure has occurred: `n$error"
}
$error.Clear()
Write-Output 'Restarting Service: Windows Management Instrumentation (winmgmt)'
try
{
	Get-Service -Name "winmgmt" | Restart-Service -Force -ErrorAction Stop | Out-Null
}
catch
{
	Write-Warning "A Failure has occurred: `n$error"
}
Pop-Location

Custom SQL Management Pack Invoke-SqlCmd Exception

May 2, 2022

Introduction

The customer noticed the issue was occurring inside of one of their custom Management Packs (MP) for SCOM SQL Blocking monitoring. They deployed this custom MP to two SCOM Management Groups. The Agents where we were seeing the issue were dual (multi) homed SCOM Agents that were SQL Servers (due to the way the MP was designed).

Script Section

The command inside of the custom Management Pack was Invoke-SqlCmd, which is apart of the built in SQL Powershell Cmdlets:

...
$conn = "MSSQL-2019"
$ScriptName = "Custom.SQLBlocking.Timed.Monitor.DataSource.ps1"
[int]$WaitTimeMS = $WaitTime * 1000
$blocked_session = $null
$sql1 = "use master
                  SELECT r.session_id,
                  r.blocking_session_id,
                  s.login_name,
                  s.login_time,
                  s.program_name,
                  s.host_name,
                  s.memory_usage as Memory,
                  DB_NAME(r.database_id) AS DatabaseName,
                  r.wait_time,
                  r.command,
                  r.status,
                  r.cpu_time,
                  t.text as Query_Text
                  FROM sys.dm_exec_requests r
                  CROSS APPLY sys.dm_exec_sql_text(sql_handle) t
                  INNER JOIN sys.dm_exec_sessions s ON r.session_id = s.session_id
                  WHERE r.blocking_session_id <> 0 and wait_time > $WaitTimeMS"
          
$error.Clear
Try {
$blocked_session = Invoke-Sqlcmd -Query $sql1  -ServerInstance $conn
}
Catch {$momapi.LogScriptEvent($ScriptName, 9994, 0, $error)}
...

Invoke-SqlCmd in the above context should be able to successfully run with Local System (or a User Account) as the Action Account.

Catching the error

We noticed the following error after the script above ran as a part of the Management Pack:

Custom.SQLBlocking.Timed.Monitor.DataSource.ps1 : ManagedBatchParser.ParserException
    at ManagedBatchParser.Parser.Parse()
    at Microsoft.SqlServer.Management.PowerShell.ExecutionProcessor.ExecuteTSql(String sqlCommand)

Resolution

You will need to set the time intervals between the rules / monitors to run at a different interval than the other Management Group that is reporting to this same Agent / Group to cause it to be multi-homed.

You can remove the other Management Group and allow only 1 Management Server in the Agent Management control panel.

Verify Assemblies are loaded with GACUtil

April 22, 2022

What is GAC?

Each computer where the Common Language Runtime is installed has a machine-wide code cache called the Global Assembly Cache. The Global Assembly Cache stores assemblies specifically designated to be shared by several applications on the computer. More information can be found here: https://docs.microsoft.com/dotnet/framework/app-domains/gac

Prerequisites

Download and install the .NET 4.8 Framework Developer Pack on the affected machine: https://dotnet.microsoft.com/download/dotnet-framework/net48

Check the currently installed Assemblies

Open a Powershell or Command Prompt as Administrator
Navigate to the installation directory:
cd "C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.8 Tools"
Run this GACUtil Command to List the Assemblies installed:
Powershell:
```
 .\gacutil /L
```
Command Prompt:
```
 gacutil /L
```

System Center Operations Manager - Data Warehouse Grooming Tool

March 22, 2022

:book: Introduction

This tool can be used to modify the System Center Operations Manager Data Warehouse Grooming retention days, allows you to see grooming history, you can manually run grooming, and you may also export the current configuration so you can keep a backup of your settings. You have the option of resetting the values to Defaults for the typical data sets in the Data Warehouse.

:arrow_down_small: Latest Version

Check the changelog for the latest version: Latest Changes

:page_with_curl: How to Use

https://aka.ms/SCOM-DW-Grooming-Tool

You have multiple ways to download the SCOM DW Grooming GUI Tool:

Download and install the MSI: MSI Download
Download and run the EXE: EXE Downloads
Download or Copy the Powershell Script to Powershell ISE: Powershell Script
Download or Copy the Powershell Script to Powershell ISE: Text Format Alternative Download Link

You will need to provide the Data Warehouse DB Server Name or Address, and the Data Warehouse Database Name. The script may auto detect these variables from the local registry on the machine you are running the script from. You can run on any machine that has network connectivity to the SCOM DW SQL Server. To get started, you will need to press the Get Current Settings button. This will allow the script to gather the information from the Data Warehouse database server. Once you make the changes you can save the change with Set.

This script will log some actions to the Application Event Log. Look for the Event Source: SCOMDWTool

:page_facing_up: More Information

You will get prompted each time you run the script to accept the license agreement, unless you select do not ask me again, when you select this it will save a file to your ProgramData Directory: C:\ProgramData\SCOM-DataWarehouseGUI-AgreedToLicense.log

If you have any questions or concerns, please leave a comment and I will do my best to assist!

How to create a Certificate Template for Operations Manager

February 28, 2022

Steps to Create Certificate Template

Step 1

Open the Certificate Authority Tool:

Step 2

Expand the tree on the left and Right Click on Certificate Templates and select Manage:

Step 3

Right Click on the IPSec (Offline request) template display name, and select Duplicate Template:

Step 4

Confirm the Compatibility tab:

Step 5

Confirm you have modified the following in the General tab:

Template Display Name:
System Center Operations Manager
Validity period:
5 years (change this per your security policy)
Check Publish certificate in Active Directory

Step 6

Confirm you have modified the following in the Request Handling tab:

Check Allow private key to be exportable (this is required for Server Authentication)

Step 7

Confirm you have modified the following in the Cryptography tab:

Verify Microsoft RSA SChannel Cryptographic Provider is Checked
Under Providers Check Microsoft Enhanced Cryptographic Provider v1.0 and move it to below Microsoft RSA SChannel Cryptographic Provider
Verify Minimum key size is set to 2048 or 1024 (2048 adds CPU overhead)

Step 8

Confirm that None is selected for the Key Attestation tab:

Step 9

Confirm you have modified the following in the Extensions tab:

Click on Application Policies and Edit
- Remove IP security IKE intermediate
- Add:
  - Client Authentication
  - Server Authentication
    - Click OK
Click on Key Usage and Edit
- Confirm you have checked:
  - Make this extension critical
  - Click OK

Step 10

Confirm you have modified the following in the Security tab:

You can add multiple types of objects here: Users, Computers, Service Accounts, Groups, or Built-in security principals.
For simplicity I will keep the defaults and only add the following permissions on Authenticated Users:
- Read
- Enroll
- Click OK to confirm / create the Certificate Template

Step 11

Close the certificate templates Console.

In the Certificate Authority tool, right click on Certificate Templates
- Hover over New -> Select Certificate Template to Issue
- Select the Certificate Template Display Name you created in Step 5:
  - Click OK

Step 12

Verify you are seeing the Certificate Template on your Management Server.

Open Local Machine Certificate Store: certlm.msc
Open Personal -> Certificates
Right Click Certificates and hover over All Tasks -> Select Request New Certificate…
- Skip through the first screen
  - Click Next
- Verify that you have selected Active Directory Enrollment Policy
  - Click Next
- Select the checkbox next to the Certificate Template you created and click on the Warning sign below the Certificate Template, it says click here to configure settings.
- Example of how to configure the certificate to be used by a Management Server
  - Click OK
- Click on Enroll

Step 13

In order to use certificates with System Center Operations Manager you will need to generate / perform run the MOMCertImport tool on atleast one of the Management Servers, and any servers that will communicate via Certificates (DMZ servers, Workgroup Machines, etc.).

MOMCertImport.exe is located in your System Center Operations Manager Installation Media inside of SupportTools\AMD64.
- Right Click on MOMCertImport.exe and select Run as administrator
  - Select the certificate you generated via the System Center Operations Manager Certificate Template.

Step 14

Restart the Microsoft Monitoring Agent with the following Powershell Command:

Restart-Service HealthService

After restarting the Microsoft Monitoring Agent (HealthService). You will wait until you see the following Event (Event ID: 20053) in the Operations Manager Event Log confirming that the certificate has been loaded:

Log Name: Operations Manager
Source: OpsMgr Connector
Date: 2/28/2022 10:35:36 AM
Event ID: 20053
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: MS01-2019.contoso.com
Description:
The OpsMgr Connector has loaded the specified authentication certificate successfully.

:bangbang: Important

You may experience issues when a certificate Re-enrolls automatically. Operations Manager needs the certificate to be imported with MOMCertImport.exe prior to being able to be used by SCOM. Unfortunately, there is not an automated method for SCOM Certificate Management.

:notebook: Note

You may experience issues with connectivity between the remote machine and the Management Server(s), verify you have checked these things:

Ensure all SPN’s are correctly registered for Management Servers, Operations Manager & Data Warehouse Databases, and services that are utilizing them.

Event ID 20071 and 21016 on Gateway point to Firewall, SPN, or Certificate issue in most cases.

Run the Gateway Approval Tool using the SDK (Data Access Service) account OR an account with high permission level (SysAdmin privileges) to Operations Manager SQL DB.

Verify you selected the appropriate certificate when you run MOMCertImport, check the certificate properties.

You may also check the following registry path: HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft OperationsManager\3.0\Machine Settings

Check the key: ChannelCertificateSerialNumber this is the serial number of the certificate you imported, reversed.

Verify that there are not any other certificates in the Local Machine Personal Store that have matching Subject Names.

Operations Manager only uses the first CN name in the Subject of the Certificate.

Cryptography API Key Storage Provider (KSP) is not supported for Operations Manager certificates.

SCOM Unix/Linux RunAs Account View - System.Xml.XmlException

February 1, 2022

I received a case today for a customer who is having issues when attempting to open the RunAs Account view for Unix/Linux RunAs Accounts in the SCOM Console (Administration Tab-> Run As Configuration -> Unix/Linux Accounts). We were unable to return any Unix/Linux RunAs Accounts in the SCOM Console, even though we can create new ones, they are not populating in the list of RunAs Accounts for Unix/Linux.

We noticed an exception that will pop-up intermittently (while Console is idle or active):

Pop-Up Exception

   __Date:__ 2/1/2022 1:29:19 PM 
   __Application:__ Operations Manager 
   __Application Version:__ 10.19.10505.0 
   __Severity:__ Error 
   __Message:__ 
   System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1. 
   at System.Xml.XmlTextReaderImpl.Throw(Exception e) 
   at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace() 
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent() 
   at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace) 
   at System.Xml.XmlDocument.Load(XmlReader reader) 
   at System.Xml.XmlDocument.LoadXml(String xml)  
   at Microsoft.SystemCenter.CrossPlatform.ClientLibrary.CredentialManagement.Core.ScxRunAsAccountHelper. DeserializeToScxRunAsAccount(ScxCredentialRef credentialRef) 
   at System.Linq.Enumerable.WhereSelectListIterator\`2.MoveNext() 
   at System.Collections.Generic.List\`1..ctor(IEnumerable\`1 collection) 
   at System.Linq.Enumerable.ToList\[TSource](IEnumerable\`1 source)  
   at Microsoft.SystemCenter.CrossPlatform.ClientLibrary.CredentialManagement.Core.ScxRunAsAccountHelper. EnumerateScxRunAsAccount(IManagementGroupConnection managementGroupConnection) 
   at Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.Administration.ScxRunAsAccountInfoFactory.EnumerateScxRunAsAccount() 
   at Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.Administration.ScxRunAsAccountHelper.<GetScxRunAsAccountInstances>b__6(Object sender, ConsoleJobEventArgs e) 
   at Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ConsoleJobExceptionHandler.ExecuteJob(IComponent component, EventHandler`1 job, Object sender, ConsoleJobEventArgs args)

Full Exception

DetailID = 4
Count: 1
Type: System.Xml.XmlException
Message: Data at the root level is invalid. Line 1, position 1.
Stack:
[HelperMethodFrame]
System.Xml.XmlTextReaderImpl.Throw(System.Exception)
System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
System.Xml.XmlTextReaderImpl.ParseDocumentContent()
System.Xml.XmlLoader.Load(System.Xml.XmlDocument, System.Xml.XmlReader, Boolean)
System.Xml.XmlDocument.Load(System.Xml.XmlReader)
System.Xml.XmlDocument.LoadXml(System.String)
Microsoft.SystemCenter.CrossPlatform.ClientLibrary.CredentialManagement.Core.ScxRunAsAccountHelper.DeserializeToScxRunAsAccount(Microsoft.SystemCenter.CrossPlatform.ClientLibrary.Common.SDKAbstraction.ScxCredentialRef)
System.Linq.Enumerable+WhereSelectListIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()
System.Collections.Generic.List`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>)
System.Linq.Enumerable.ToList[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>)
Microsoft.SystemCenter.CrossPlatform.ClientLibrary.CredentialManagement.Core.ScxRunAsAccountHelper.EnumerateScxRunAsAccount(Microsoft.SystemCenter.CrossPlatform.ClientLibrary.Common.SDKAbstraction.IManagementGroupConnection)
Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.Administration.ScxRunAsAccountInfoFactory.EnumerateScxRunAsAccount()
Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.Administration.ScxRunAsAccountHelper.<GetScxRunAsAccountInstances>b__6(System.Object, Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ConsoleJobExceptionHandler.ExecuteJob(System.ComponentModel.IComponent, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs>, System.Object, Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs)
Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobsService.RunJob(Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobDescription)
Microsoft.EnterpriseManagement.ConsoleFramework.WindowJobsService.RunAsyncJobInThisThread(Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobDescription)
Microsoft.EnterpriseManagement.ConsoleFramework.WindowJobsService.RunJob(Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobDescription)
Microsoft.EnterpriseManagement.ConsoleFramework.WindowJobsService.RunJob(System.ComponentModel.IComponent, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs>, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobErrorEventArgs>, System.Object[])
Microsoft.EnterpriseManagement.ConsoleFramework.WindowJobsService.RunJob(System.ComponentModel.IComponent, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs>, System.Object[])
Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobs.RunJob(System.ComponentModel.IComponent, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs>, System.Object[])
Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.Administration.RunAsAccountQuery.DoQuery(System.String)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.Query`1[[System.__Canon, mscorlib]].DoQuery(System.String, System.Nullable`1<System.DateTime>)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.Query`1[[System.__Canon, mscorlib]].FullUpdateQuery(Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.CacheSession, Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.IndexTable ByRef, Boolean, System.DateTime)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.Query`1[[System.__Canon, mscorlib]].InternalSyncQuery(Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.CacheSession, Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.IndexTable, Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.UpdateReason, Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.UpdateType)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.Query`1[[System.__Canon, mscorlib]].InternalQuery(Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.CacheSession, Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.UpdateReason)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.Query`1[[System.__Canon, mscorlib]].TryDoQuery(Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.UpdateReason, Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.CacheSession)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ConsoleJobExceptionHandler.ExecuteJob(System.ComponentModel.IComponent, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs>, System.Object, Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs)
Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobsService.RunJob(Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobDescription)
Microsoft.EnterpriseManagement.ConsoleFramework.WindowJobsService.RunAsyncJobInThisThread(Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobDescription)
Microsoft.EnterpriseManagement.ConsoleFramework.WindowJobsService.RunJob(Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobDescription)
Microsoft.EnterpriseManagement.ConsoleFramework.WindowJobsService.RunJob(System.ComponentModel.IComponent, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs>, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobErrorEventArgs>, System.Object[])
Microsoft.EnterpriseManagement.ConsoleFramework.WindowJobsService.RunJob(System.ComponentModel.IComponent, System.EventHandler`1<Microsoft.EnterpriseManagement.ConsoleFramework.ConsoleJobEventArgs>, System.Object[])
Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.Query`1[[System.__Canon, mscorlib]].DoQuery(Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.UpdateReason, Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.CacheSession)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.QueryBase.Update(Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.UpdateReason, Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.CacheSession)
Microsoft.EnterpriseManagement.Mom.Internal.UI.Cache.DataCache.Polling()
System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
System.Threading.ThreadHelper.ThreadStart()
[GCFrame]
[DebuggerU2MCatchHandlerFrame]

How to Resolve

First we ran the following Powershell command output to gather RunAs Accounts related to Unix/Linux:

Get-SCOMRunAsAccount | Where {$_.AccountType -like "SCX*"}

We compared the above output to the accounts we can choose in the RunAs Profile for Unix/Linux Action Account. We noticed an entry that was not in the Get-SCOMRunAsAccount command run above:

I asked the customer if he is comfortable with removing the RunAs Account and re-adding the Account. He said this was fine, so we proceeded to remove the RunAs Account like this:

Get-SCOMRunAsAccount | Where {$_.Name -eq "Test"} | Remove-SCOMRunAsAccount

Conclusion

Removing the Orphaned RunAs account allowed the Unix/Linux RunAs Account view to populate as intended. You can only remove these Orphaned RunAs accounts with the Powershell Commands: Get-SCOMRunAsAccount and Remove-SCOMRunAsAccount

How to resolve SCOM Notifications Stopped & No New Alerts

January 28, 2022

First thing I noticed was that the SCOM Management Servers had an SCOM Agent Installed on it. We verifing by navigating to the following location, you can see the Agent Management Groups:

Apparently the Operations Manager Management Server received a package from SCCM, and this attempted to automatically install the SCOM Agent (Microsoft Monitoring Agent). And during this process the installer overwrote some registry keys inside of the following location:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\

Precaution

Take care when directly modifying the Registry via regedit.exe. If you insist on making changes, always backup the registry first. There is always the possibility you can cause more damage than you are fixing.

I had the customer Delete the Agent Management Groups key and we matched / created the Registry Values for Server Management Groups compared to my lab environment. We needed to create the Management Group Name Key in my situation.

After doing this we cleared the cache on the SCOM Management Servers by running the following PowerShell script:
SCOM-Scripts-and-SQL/Clear-SCOMCache.ps1 at master · blakedrumm/SCOM-Script-and-SQL

We waited for a few minutes after clearing the SCOM Management Server cache. The Management Servers were coming back online, but they were still not receiving new Alerts, and there were not any Notification Emails.

While reviewing the Operations Manager Event Logs, we found that there were 2115 Errors indicating an issue with the insertion of Discovery and other related data:

Event 1

  __Log Name:__      Operations Manager 
  __Source:__        HealthService 
  __Date:__          1/20/2022 3:42:02 PM 
  __Event ID:__      2115 
  __Task Category:__ None 
  __Level:__         Warning 
  __Keywords:__      Classic 
  __User:__          N/A 
  __Computer:__      ManagementServer1.contoso.com 
  __Description:__ 
  A Bind Data Source in Management Group ManagementGroup1 has posted items to the workflow, but has not received a response in 480 seconds.  This indicates a performance or functional problem with the workflow. 
  __Workflow Id :__ Microsoft.SystemCenter.CollectEventData 
  __Instance    :__ ManagementServer1.contoso.com 
  __Instance Id :__ {AEC38E5Z-67A9-0406-20DB-ACC33BB9C4A4}

Event 2

  __Log Name:__      Operations Manager 
  __Source:__        HealthService 
  __Date:__          1/20/2022 3:42:02 PM 
  __Event ID:__      2115 
  __Task Category:__ None 
  __Level:__         Warning 
  __Keywords:__      Classic 
  __User:__          N/A 
  __Computer:__      ManagementServer1.contoso.com 
  __Description:__ 
  A Bind Data Source in Management Group ManagementGroup1 has posted items to the workflow, but has not received a response in 480 seconds.  This indicates a performance or functional problem with the workflow. 
  __Workflow Id :__ Microsoft.SystemCenter.CollectPerformanceData 
  __Instance    :__ ManagementServer1.contoso.com 
  __Instance Id :__ {AEC38E5Z-67A9-0406-20DB-ACC33BB9C4A4}

Event 3

  __Log Name:__      Operations Manager 
  __Source:__        HealthService 
  __Date:__          1/20/2022 3:42:02 PM 
  __Event ID:__      2115 
  __Task Category:__ None 
  __Level:__         Warning 
  __Keywords:__      Classic 
  __User:__          N/A 
  __Computer:__      ManagementServer1.contoso.com 
  __Description:__ 
  A Bind Data Source in Management Group ManagementGroup1 has posted items to the workflow, but has not received a response in 480 seconds.  This indicates a performance or functional problem with the workflow. 
  __Workflow Id :__ Microsoft.SystemCenter.CollectPublishedEntityState 
  __Instance    :__ ManagementServer1.contoso.com 
  __Instance Id :__ {AEC38E5Z-67A9-0406-20DB-ACC33BB9C4A4}

Event 4

  __Log Name:__      Operations Manager 
  __Source:__        HealthService 
  __Date:__          1/20/2022 3:42:03 PM 
  __Event ID:__      2115 
  __Task Category:__ None 
  __Level:__         Warning 
  __Keywords:__      Classic 
  __User:__          N/A 
  __Computer:__      ManagementServer1.contoso.com 
  __Description:__ 
  A Bind Data Source in Management Group ManagementGroup1 has posted items to the workflow, but has not received a response in 480 seconds.  This indicates a performance or functional problem with the workflow. 
  __Workflow Id :__ Microsoft.SystemCenter.CollectSignatureData 
  __Instance    :__ ManagementServer1.contoso.com 
  __Instance Id :__ {AEC38E5Z-67A9-0406-20DB-ACC33BB9C4A4}

Event 5

  __Log Name:__      Operations Manager 
  __Source:__        HealthService 
  __Date:__          1/20/2022 3:42:30 PM 
  __Event ID:__      2115 
  __Task Category:__ None 
  __Level:__         Warning 
  __Keywords:__      Classic 
  __User:__          N/A 
  __Computer:__      ManagementServer1.contoso.com 
  __Description:__ 
  A Bind Data Source in Management Group ManagementGroup1 has posted items to the workflow, but has not received a response in 480 seconds.  This indicates a performance or functional problem with the workflow. 
  __Workflow Id :__ Microsoft.SystemCenter.CollectDiscoveryData 
  __Instance    :__ ManagementServer1.contoso.com 
  __Instance Id :__ {AEC38E5Z-67A9-0406-20DB-ACC33BB9C4A4}

We reviewed the current SQL Logs and found that there were authentication failures that indicated the Computer Account for the SCOM Management Server didnt have permission to the Database. This lead us to check the Default Action Account in Run As Profiles. We modified the Default Action Account for the Management Servers to be assigned an Windows Action Account instead of Local System. This resolved the issue and now Notifications and Alerts are being being sent normally.

UNIX/Linux System Center Operations Manager Agents Troubleshooting Tips

January 26, 2022

_{This post was last updated on September 27th, 2023}

Build list containing the SCOM Linux Agent versions

The following url contains the versions of the SCX Agent: https://docs.microsoft.com/system-center/scom/release-build-versions

Verify the versions for all prerequisite software

You can run the following command on a monitored and not monitored server to compare the software installed, or verify with the official list https://docs.microsoft.com/system-center/scom/plan-supported-crossplat-os:

rpm -qa | egrep "^glibc|^openssl|^pam|^scx|^omi"

Agent Version	Version	Management Group Version	Release Date
scx-1.5.1-242.e16.x86_64	7.5.1068.0	SCOM 2012 R2 UR12	01/27/2017

Working Example:

Non-working Example:

Clearing the System Security Services Daemon (SSSD) Cache

The System Security Services Daemon (SSSD) provides access to identity and authentication providers. Basically rather than relying on locally configured authentication, SSSD is used to lookup its local cache. The entries within this cache may come from different remote identity providers, such as an LDAP directory, Kerberos, or Active Directory for example.

SSSD caches the results of users and credentials from these remote locations so that if the identity provider goes offline, the user credentials are still available and users can still login. This helps to improve performance and facilitates scalability with a single user that can login over many systems, rather than using local accounts everywhere.

The cached results can potentially be problematic if the stored records become stale and are no longer in sync with the identity provider, so it is important to know how to flush the SSSD cache to fix various problems and update the cache.

:notebook: Note

It’s recommend to only clear the sssd cache if the identity provider servers performing the authentication within the domain are available, otherwise users will not be able to log in once the sssd cache has been flushed.

Stop SSSD Service

Service sssd stop;

Clear SSSD Cache

rm -rf /var/lib/sss/db/*;

Start SSSD Service

service sssd start

SSSD should now start up correctly with an empty cache, any user login will now first go directly to the defined identity provider for authentication, and then be cached locally afterwards.

Tail the Logs

Enable Verbose SCX Logs

Use the below command to enable verbose logging for the SCX Provider:

scxadmin -log-set all verbose

Enable Verbose logging for OMI Server

Use the below command to modify the omiserver.conf file. Change loglevel to one of the following valid options, verbose being highest: ERROR, WARNING, INFO, DEBUG, VERBOSE

sudo vim /etc/opt/omi/conf/omiserver.conf

If loglevel does not exist in the file you can add the following lines to the file and change the loglevel value

##
## loglevel -- set the loggiing options for OMI server
##   Valid options are: ERROR, WARNING, INFO, DEBUG, VERBOSE (debug build)
##   If commented out, then default value is: WARNING
##
loglevel = INFO

After saving and quitting vim, restart the service with below command for logging to be enabled

sudo /opt/omi/bin/service_control restart

Secure Log

You can run the following command to show current log data pertaining to authentication and authorization privileges:

tail -f /var/log/secure

Messages Log

You can run the following command to show all the global system messages, including the messages that are logged during system startup:

tail -f /var/log/messages

OMI

Server Log

tail -f /var/opt/omi/log/omiserver.log

Agent Log

tail -f /var/opt/microsoft/scx/log/omiagent.root.root.log

SCX

Agent Log

tail -f /var/opt/microsoft/scx/log/scx.log

Verify OpenSSL s_client

The OpenSSL s_client command is a helpful test client for troubleshooting remote SSL or TLS connections:

openssl s_client -connect server.domain.com:1270
openssl s_client -connect server.domain.com:1270 -tls1
openssl s_client -connect server.domain.com:1270 -ssl3

Check Linux Ciphers

openssl ciphers -V

Get MB / GB size of file

Run the following command to gather the MB / GB size of a file:

du -sh /var/opt/microsoft/scx/log/scx.log

WinRM Enumerate SCX Agent

From the Management Server(s) in the Unix/Linux Resource Pool, verify that the following command resolves correctly:

Basic Authentication

winrm enumerate http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_Agent?__cimnamespace=root/scx -username:<username> -password:<password> -r:https://<LINUXSERVERFQDN>:1270/wsman -auth:basic -skipCAcheck -skipCNcheck -skipRevocationcheck -encoding:utf-8

Kerberos Authentication

winrm e http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_Agent?__cimnamespace=root/scx -r:https://<LINUXSERVERFQDN>:1270 -u:<[email protected]> -p:<password> -auth:Kerberos -skipcacheck -skipcncheck -encoding:utf-8

:notebook: Note 1

Verify that you have enabled Kerberos via the Management Server Registry on each Management Server in the resource pool you are using to monitor the Unix/Linux agents: https://docs.microsoft.com/system-center/scom/manage-linux-kerberos-auth

Example 1

Issue

You may experience an error that contains the following when running the above Commands:

WSManFault  
    Message = The server certificate on the destination computer (<LINUXSERVERFQDN>:1270) has the following errors:
Encountered an internal error in the SSL library.
Error number:  -2147012721 0x80072F8F
A security error occurred

or this error via the Discovery Wizard:

Agent verification failed. Error detail: The server certificate on the destination computer (<LINUXSERVERFQDN>:1270) has the following errors:
Encountered an internal error in the SSL library.
It is possible that:
   1. The destination certificate is signed by another certificate authority not trusted by the management server.
   2. The destination has an invalid certificate, e.g., its common name (CN) does not match the fully qualified domain name (FQDN) used for the connection.  The FQDN used for the connection is: <LINUXSERVERFQDN>.
   3. The servers in the resource pool have not been configured to trust certificates signed by other servers in the pool.

Resolution

You could potentially import (Merge) the below known working ciphers by copying the text to a new file on your server called example.reg, right click and Merge the file into your registry:

   Windows Registry Editor Version 5.00

   [HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002]
       "Functions"="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P521,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P521,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P521,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P521,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P521,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P521,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P521,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P521,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P521,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256"

An alternative to importing the above registry is to download and run IISCrypto and select Best Practices.

:notebook: Note

Be sure to verify if the Linux agent supports the same ciphers as the Management Server / Gateway. List of Ciphers Supported for Linux / Unix SCOM Agents: https://docs.microsoft.com/system-center/scom/manage-security-crossplat-config-sslcipher#cipher-suite-support-matrix

Enumerate OMI Information on UNIX/Linux Machine

Enumerate the SCX_OperatingSystem via omicli:

/opt/omi/bin/omicli ei root/scx SCX_OperatingSystem

Enumerate the SCX_ProcessorStatisticalInformation via omicli:

/opt/omi/bin/omicli ei root/scx SCX_ProcessorStatisticalInformation

Some OMI WMI Namespaces on the Linux Agent:

SCX_Agent
SCX_DiskDrive
SCX_DiskDriveStatisticalInformation
SCX_EthernetPortStatistics
SCX_FileSystem
SCX_FileSystemStatisticalInformation
SCX_IPProtocolEndpoint
SCX_LogFile
SCX_MemoryStatisticalInformation
SCX_OperatingSystem
SCX_ProcessorStatisticalInformation
SCX_StatisticalInformation
SCX_UnixProcess
SCX_UnixProcessStatisticalInformation

Certificate Troubleshooting

UNIX/Linux Certificate

Check Certificate
openssl x509 -in /etc/opt/omi/ssl/omi.pem -text -noout
Generate Certificate with custom name
/opt/microsoft/scx/bin/tools/scxsslconfig -f -d contoso.com -h redhat
Generate Certificate
/opt/microsoft/scx/bin/tools/scxsslconfig -f
Remove Old Certificate
rm /etc/opt/omi/ssl/omikey.pem --force

Configure the Xplat certificates (export/import) for each management server in the resource pool

The below steps show how to configure Xplat certificates easily for Management Servers with Powershell. You can automate this process for as many management servers as you need!

Windows Management Server Commands

Export Certificate On MS1 (Admin Powershell Prompt)
& "C:\Program Files\Microsoft System Center\Operations Manager\Server\scxcertconfig.exe" -export \\MS2\c$\MS1.cer
Export Certificate On MS2 (Admin Powershell Prompt)
& "C:\Program Files\Microsoft System Center\Operations Manager\Server\scxcertconfig.exe" -export \\MS1\c$\MS2.cer
Import Certificate On MS1 (Admin Powershell Prompt)
& "C:\Program Files\Microsoft System Center\Operations Manager\Server\scxcertconfig.exe" -import \\MS1\c$\MS2.cer
Import Certificate On MS2 (Admin Powershell Prompt)
& "C:\Program Files\Microsoft System Center\Operations Manager\Server\scxcertconfig.exe" -import \\MS2\c$\MS1.cer

Location for Certificates

SCOM 2012 Linux Agent:
/etc/opt/microsoft/scx/ssl/scx.pem
/etc/opt/microsoft/scx/ssl/scx-host-<HostName>.pem
SCOM 2016 Linux Agent and newer:
/etc/opt/omi/ssl/omi.pem
/etc/opt/omi/ssl/omi-host-<HostName>.pem

Linux Agent Certificate Hostname Detection steps during initial Installation

The following steps are what happens (from a high level) during initial installation of the Linux / Unix Agent to generate a Certificate for the Agent.

Try hostname -f (this will fail on some Linux systems)
Attempt to obtain the domain name from /etc/resolve.conf
Attempt to obtain long hostname with nslookup command

Purge SCX Agent Installation

On the Unix/Linux Machine locate the SCOM Linux Agent installation file, and run the following command to uninstall and purge the SCOM Linux Agent installation:

sh ./scx-<version>.<type>.<version>.<arch>.sh --purge --force

Linux Regular Expression (Regex)

The SCOM console uses .NET Regex and the SCX Agent uses POSIX Regex.

Firewalld Service

Check that the SCOM Linux Agent (omiengine) Port is listening, which would mean the SCX Linux Agent service is listening:

netstat -tulpn | grep -i 1270

To open port 1270 (any ip address) for System Center Operations Manager (SCOM) on a Red Hat server using Firewalld, you can execute the following commands:
1. Open the Port: To open port 1270, you would typically use the --add-port option to specify the port and the protocol (either TCP or UDP, depending on your requirement).
```
 sudo firewall-cmd --zone=public --add-port=1270/tcp --permanent
```
  The --zone=public flag specifies that you’re modifying the public zone, but you could replace public with the name of another zone if that’s more appropriate for your setup. The --permanent flag makes sure the rule survives reboots.
2. Reload Firewalld: After adding the port, you’ll need to reload Firewalld to apply the changes:
```
 sudo firewall-cmd --reload
```
3. Check the Rules: To make sure the port has been opened, you can list the rules for the public zone as follows:
```
 sudo firewall-cmd --zone=public --list-ports
```
  This should display the ports that are open, and 1270 should be among them if the previous commands were successful.
These commands are based on the assumption that you are using the firewall-cmd utility, which is the default frontend for Firewalld on Red Hat.
To open port 1270 (specific ip address) for System Center Operations Manager (SCOM) on a Red Hat server using Firewalld, you can execute the following commands:

Here’s how you can open port 1270 for a specific IP address (e.g., 192.168.1.10):
```
 sudo firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.10" port port=1270 protocol=tcp accept' --permanent
```
This adds a “rich rule” to the public zone, specifying that only traffic from the IP address 192.168.1.10 is allowed to access port 1270 over TCP. Replace 192.168.1.10 with the specific IP address that needs access to your SCOM service on port 1270.

After adding the rule, don’t forget to reload the firewall to apply the changes:
```
 sudo firewall-cmd --reload
```
To check if the rule has been successfully added, you can list all the rich rules for the public zone like so:
```
 sudo firewall-cmd --zone=public --list-rich-rules
```
This should show your newly added rule among the list.

Note: Be sure to check any security policies or guidelines in your organization before opening ports on a server.

SCOM ACS Collector Troubleshooting Tips

January 24, 2022

A customer of mine was having problems with Audit Collection Services in SCOM 2019 UR3. I figured this information may prove helpful someday to someone. Enjoy!

SQL Queries

Here are some ACS SQL Queries you can utilize:
https://github.com/blakedrumm/SCOM-Scripts-and-SQL/tree/master/SQL%20Queries/OperationsManagerAC

Verify ACS collector performance

Open Performance Monitor, right click on Start Menu and select run.
In run box type the following:
```
perfmon.msc
```
Once Opened go to Monitoring Tools Folder, then right click on Performance Monitor and go to Properties:
Select the Data tab:
Clear the currently selected counters by removing each counter listed with the Remove Button, then select Add:
Locate the ACS Collector in available counters, select it, and click on the Add button:

This is how it should look when it is added:
Click OK to confirm the property settings.
Click on the following button to change the view type, change to Report:
View your performance data for ACS:

Gather Partition Table History

You can run the following tsql query against the OperationsManagerAC

-- Status:
-- 0: active, set by collector
-- 1: inactive, mark for closing during collector startup & indexing, set manually
-- 2: archived, ready for deletion, set from outside the collector
-- 100 - 108: closed, indexing in progress
-- 109: indexing complete
select * from dtpartition order by partitionstarttime

Enable ACS Collector Logging

TraceFlags

TraceFlags consists of three groups that are OR’d (added) together. They are Verbosity Level, Component, and Log Output Method. Verbosity Level refers to the number and type of events that are logged to the trace log when trace logging is enabled, while Component can only be used if the Verbosity Level is set to Level Debug and is used to enable/disable logging for various internal components. Log Output Method is used to specify the location that tracing information is written to.

Verbosity Level settings

Value	Description
0x00000000	No logging.
0x00000001	Level Error. Logs only errors.
0x00000002	Level Warning. Logs errors and warnings.
0x00000003	Level Informational. Logs errors, warnings, and information.
0x00000004	Level Debug. Logs everything and is very verbose.

Component settings

Value	Description
0x00000010	General
0x00000020	Networking
0x00000080	Event tracking

Log Output Method settings

Value	Description
0x00020000	Log to debug out
0x00040000	Log to the console, if available (only when running the collector from the command line)
0x00080000	Log to file in `%SystemRoot%\Temp` directory

%SystemRoot% = C:\Windows\

Trace log files are located in the %SystemRoot%\Temp directory. For the Collector, the log files are written to %SystemRoot%\Temp\AdtServer.log and %SystemRoot%\Temp\AdtSrvDll.log. When a log file fills to capacity, the system overwrites the oldest entries with the most recent entries. Log files have a default size of 1 MB and begin to overwrite the oldest entries once they are full.

To enable logging, open RegEdit navigate to the following location:
HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\AdtAgent\Parameters
Create the TraceFlags DWORD (32-bit) Value and set it to Hexadecimal 80003 unless otherwise advised. Trace logging begins immediately; no restart of the Collector is needed. Additionally, Change permission must be granted to %SystemRoot%\Temp for the NetworkService account.

Note

For example, to log errors, warnings, and information messages to a file change the traceflags registry value to 0x00080003. This is a combination of 0x00080000 [file] and 0x00000003 [errors +warnings+informational].
To disable logging, delete the TraceFlags value or set it to 0.

Change the scheduled time the SQL Scripts run that are in the ACS Installation Folder

In order for you to change the time the jobs will run the partitioning, grooming and reindexing. You need to edit the dtConfig table and change the “table switch offset in seconds since midnight UTC” value to another time in UTC (Google UTC Converter):

Get current configuration:

select * from dtConfig

Get current SQL UTC time:

SELECT GETUTCDATE() AS 'Current UTC Time' 

Get 10 minutes ahead of current UTC time:

SELECT DATEADD(MINUTE, 10, GETUTCDATE())

Get seconds from midnight for dtConfig “table switch offset in seconds since midnight UTC” ahead in seconds, in this example we add 10 minutes to the current UTC time:

Declare @d DateTime
Select @d = DATEADD(MINUTE, 10, GETUTCDATE())
Select (DatePart(HOUR, @d) * 3600) + (DatePart(MINUTE, @d) * 60) + DatePart(SECOND, @d)

Update configuration to midnight UTC, which is 7:00 PM (EST):

UPDATE [dbo].[dtConfig]
   SET [Value] = 0 -- Default: 25200 (2:00 AM (EST))
   WHERE [Id] = 4
GO

Be mindful that the scripts are usually run overnight when there is not alot of activity in the ACS Database. So take precautions if changing this when alot of logons/logoffs are happening in your environment.

Set filter for ACS Data

On the ACS server, Go to the registry: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\AdtServer\Parameters
Right click the key Parameters, select Permissions
Select the row “Network service”
Click edit
Set “Apply to” as “This key and subkeys”
Select Full control in the column Allow, click OK
Run CMD as an Administrator

Run the below command:

cd C:\Windows\System32\Security\AdtServer

Run the below command to check current query we being used by SCOM ACS. (There should be a no filter clause to exclude the collected event):
```
adtadmin –getquery
```

Run the following command to add filter for the collected events:

adtadmin /setquery /query:"SELECT * FROM AdtsEvent WHERE NOT (EventID=528 or EventID=540 or EventID=680)”

Note

Change the event IDs to the events you don’t want collected. See the below example for a basic filter. Since every environment has its own concerns, please double check which events can be ignored to reduce data collection amount.
adtadmin /setquery /query:"SELECT * FROM AdtsEvent WHERE NOT (((EventId=528 AND String01='5') OR (EventId=576 AND (String01='SeChangeNotifyPrivilege' OR HeaderDomain='NT Authority')) OR (EventId=538 OR EventId=566 OR EventId=672 OR EventId=680)))"

Install ACS from Command Line

In order to install the ACS Collector from the command line you are required to provide an XML, this an example of the XML that is required:

<?xml version="1.0" encoding="utf-8" ?>

<InstallationOptions>

  <AcsCollector>

    <!-- Interval (in seconds) between heartbeats. -->
   <HeartBeatInterval>60</HeartBeatInterval>

    <!-- Number of failed heart beats before collector
             disconnects the forwarder. -->
    <HeartBeatFailedCount>3</HeartBeatFailedCount>

    <!-- Time (in seconds) to wait before disconnecting forwarders
             which have not yet sent the initial data packet on a
             new connection. -->
    <ConnectNoDataThreshold>10</ConnectNoDataThreshold>

    <!-- The TCP port on which the collector should listen
             for forwarder connections. -->
    <AdtAgentPort>51909</AdtAgentPort>

    <!-- Maximum length of database queue. This value will be
             rounded up to a power of two if necessary. -->
    <MaximumQueueLength>0x40000</MaximumQueueLength>

    <!-- Lower bound (in seconds) of randomly chosen backoff
             time sent to agent if the collector is overloaded. -->
    <BackOffMinTime>120</BackOffMinTime>

    <!-- Upper bound (in seconds) of randomly chosen backoff
             time sent to agent if the collector is overloaded. -->
    <BackOffMaxTime>480</BackOffMaxTime>

    <!-- (percentage of max queue length) -->
   <BackOffThreshold>50</BackOffThreshold>

    <!-- (percentage of max queue length) -->
   <DisconnectThreshold>75</DisconnectThreshold>

    <!-- Asset value new agents get assigned.
             -1 means they get the value of their group instead. -->
    <DefaultAssetValue>-1</DefaultAssetValue>

    <!-- Group ID of the group new agents get assigned to. -->
   <DefaultGroup>0</DefaultGroup>

    <!-- Number of simultaneous database connections to use
             for event insertion. -->
    <DbEventConnections>8</DbEventConnections>

    <!-- Number of simultaneous database connections to use
             for string insertion. -->
    <DbStringConnections>8</DbStringConnections>

    <!-- Minimum asset value required in order to report 
             application log events regarding an agent. -->
    <ReportThreshold>1</ReportThreshold>

    <!-- Minimum asset value required in order to store events
             from an agent. -->
    <StoreThreshold>1</StoreThreshold>

    <!-- (seconds) -->
   <CheckPointInterval>198</CheckPointInterval>

    <!-- (seconds) -->
   <PurgingInterval>198</PurgingInterval>

    <!-- If running in grooming mode, the collector only kicks
             off a grooming cycle if the database queue has at most
             MaxPurgingQueue entries. -->
    <MaxPurgingQueue>500</MaxPurgingQueue>

    <!-- 0 = Event timestamps will be reported in UTC.
         1 = Event timestamps will be reported in local time. -->
    <ConvertToLocalTime>1</ConvertToLocalTime>

    <!-- Size (in bytes) of memory to use for caching principal data. -->
   <PrincipalCacheSize>0x8000</PrincipalCacheSize>

    <!-- Size (in bytes) of memory to use for caching string data. -->
   <StringCacheSize>0x20000</StringCacheSize>

    <!-- Time, in seconds, to backoff unwanted agents -->
   <BackOffUnwanted>21600</BackOffUnwanted>

    <!-- Maximum time (in minutes) a connection can exist before it
             gets disconnected for rekeying purposes. -->
    <TlsRekeyInterval>600</TlsRekeyInterval>

    <!-- -->
   <AcceptSockets>4</AcceptSockets>

    <!-- Time, in seconds, before a new connection gets
             disconnected if the client doesn't send any data -->
    <MaxAcceptIdle>15</MaxAcceptIdle>

    <!-- Time, in seconds, before a connection attempt times out -->
   <MaxConnectIdle>15</MaxConnectIdle>

    <!-- Interval (in seconds) between recalculation of queue
             statistics. -->
    <MinUpdateInterval>21</MinUpdateInterval>

    <!-- Name of ODBC data source -->
   <DataSource>acs</DataSource>

    <!-- 0 = SQL Server runs on a different machine
         1 = SQL Server runs on collector machine -->
    <DbServerLocal>1</DbServerLocal>

    <!-- Name of machine running SQL Server -->
   <DbServerName>acs-sql</DbServerName>

    <!-- Name of SQL Server instance to use. Usually blank. -->
   <DbServerInstance></DbServerInstance>

    <!-- Name of collector database -->
   <DbName>OperationsManagerAC</DbName>

    <!-- 0 = Specified data and log directories will be used.
         1 = SQL Server's data and log directories will be used. -->
    <UseDefaultPaths>1</UseDefaultPaths>

    <!-- Path (on DB server) where data file should be created
             Necessary only if UseDefaultPaths is set to 0 -->
    <DataFilePath>d:\acsdata</DataFilePath>

    <!-- Initial data file size in MB -->
   <DbDataFileInitialSize>1024</DbDataFileInitialSize>

    <!-- Maximum data file size in MB, 0 means unlimited -->
   <DbDataFileMaximumSize>65536</DbDataFileMaximumSize>

    <!-- File growth amount in MB -->
   <DbDataFileIncrementSize>1024</DbDataFileIncrementSize>

    <!-- Path (on DB server) where log file should be created.
             Necessary only if UseDefaultPaths is set to 0 -->
    <LogFilePath>e:\acslog</LogFilePath>

    <!-- Initial log file size in MB -->
   <DbLogFileInitialSize>128</DbLogFileInitialSize>

    <!-- Maximum log file size in MB, 0 means unlimited -->
   <DbLogFileMaximumSize>4096</DbLogFileMaximumSize>

    <!-- File growth amount in MB -->
   <DbLogFileIncrementSize>128</DbLogFileIncrementSize>

    <!-- (hours) time to keep events -->
   <EventRetentionPeriod>24</EventRetentionPeriod>

    <!-- Number of partitions to use -->
   <PartitionCount>1</PartitionCount>

    <!-- Time when partition switching should occur.
             Expressed in seconds since midnight in UTC -->
    <PartitionSwitchOffset>0</PartitionSwitchOffset>

    <!-- Type of authentication collector should use
             when connecting to the database:
             0 = Use SQL authentication (not recommended)
             1 = Use Windows authentication (recommended) -->
    <UseWindowsAuth>1</UseWindowsAuth>

   <!-- only necessary when using SQL authentication -->
   <DbUser>SampleAdtServerDbUser</DbUser>

    <!-- only necessary when using SQL authentication -->
   <DbPassword>SampleAdtServerDbPwd</DbPassword>

  </AcsCollector>

</InstallationOptions>

Maintenance Schedule Failing Intermittently

January 11, 2022

The customer had setup Maintenance Scheduled Jobs that were to run Daily against a group of servers. This Maintenance Schedule would work as intended for a while (up to a week), but would stop working intermittently. The “Next Run” date/time for the Maintenance Mode Scheduled Job would be empty and to fix this we would need to change, “The schedule is effective beginning” date, to today. Or recreate the maintenance schedule job.

In order to resolve this issue we had to run the SCOM Console as the SDK (System Center Data Access Service) Account and recreate the Maintenance Scheduled Jobs. This allowed the Maintenance Mode Scheduled jobs to run without any issues.

Conclusion

So the issue was due to the users’ personal Active Directory account being locked out when setting up the Maintenance Schedule. To resolve we just need to run as the Data Access Service (SDK) Account.

Resolve Post Install error with Unix / Linux Install for SCOM Agent

January 10, 2022

Introduction

My customer was running Solaris 10 (SunOS 5.10), they were having issues when attempting to install the SCOM Agent (scx-1.6.8-1.solaris.10.sparc.sh). So we dug further into things to verify why the installer was failing on the PostInstall step.

Discovery error is shown when attempting to discover from the SCOM Console via the Discovery Wizard:

Task invocation failed with error code -2130771918. Error message was: The SCXCertWriteAction module encountered a DoProcess exception. The workflow "Microsoft.Unix.Agent.GetCert.Task" has been unloaded.
 
Module: SCXCertWriteAction
Location: DoProcess
Exception type: ScxCertLibException
 
Exception message: Unable to create certificate context
; {ASN1 bad tag value met.
}
 
Additional data: Sudo path: /etc/opt/microsoft/scx/conf/sudodir/
Management group: SCOM2019
Workflow name: Microsoft.Unix.Agent.GetCert.Task
Object name: UNIX/Linux Resource Pool
Object ID: {6F1686BA-C068-4C9F-F1ED-29CDD800528A}

What we did

I asked the customer to copy the SCOM Linux / Unix Agent install files (C:\Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits) to the Solaris Machine so we can attempt a manual installation.

Extract the Solaris Installer file:

sh scx-1.6.8-1.solaris.10.sparc.sh --extract

We looked at the output after running the install manually:

[solaris10:root@/home/ops_monitoring/scxbundle.24766] pkgadd -d scx-1.6.8-1.solaris.10.sparc.pkg
 
The following packages are available:
  1  MSFTscx     Microsoft System Center 2012 Operations Manager for UNIX/Linux agent
                 (sparc) 1.6.8-1
 
Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]:
 
Processing package instance <MSFTscx> from </home/ops_monitoring/scxbundle.24766/scx-1.6.8-1.solaris.10.sparc.pkg>
 
Microsoft System Center 2012 Operations Manager for UNIX/Linux agent(sparc) 1.6.8-1
http://www.microsoft.com
## Processing package information.
## Processing system information.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.
 
This package contains scripts which will be executed with super-user
permission during the process of installing this package.
 
Do you want to continue with the installation of <MSFTscx> [y,n,?] y
 
Installing Microsoft System Center 2012 Operations Manager for UNIX/Linux agent as <MSFTscx>
 
## Executing preinstall script.
Waiting for service stop: svc:/application/management/omid ...
## Installing part 1 of 1.
/etc/opt/microsoft/scx/pf_file.sh
/etc/opt/omi/conf/omiregister/root-scx/SCXProvider-omi.reg
/etc/opt/omi/conf/omiregister/root-scx/SCXProvider-req.reg
/etc/opt/omi/conf/omiregister/root-scx/SCXProvider-root.reg
/opt/microsoft/scx/bin/omi_preexec
/opt/microsoft/scx/bin/scxlogfilereader
/opt/microsoft/scx/bin/setup.sh
/opt/microsoft/scx/bin/tools/.scxadmin
/opt/microsoft/scx/bin/tools/.scxsslconfig
/opt/microsoft/scx/bin/tools/scxadmin
/opt/microsoft/scx/bin/tools/scxsslconfig
/opt/microsoft/scx/bin/tools/setup.sh
/opt/microsoft/scx/bin/uninstall
/opt/microsoft/scx/lib/libSCXCoreProviderModule.so
/opt/omi/lib/libSCXCoreProviderModule.so <symbolic link>
[ verifying class <none> ]
/etc/opt/microsoft/scx/conf/installinfo.txt
/etc/opt/microsoft/scx/conf/scxconfig.conf
/etc/opt/microsoft/scx/conf/scxlog.conf
/etc/opt/microsoft/scx/conf/scxrunas.conf
[ verifying class <config> ]
## Executing postinstall script.
/var/sadm/pkg/MSFTscx/install/postinstall: ENV=/usr/sislocal/profile: is not an identifier
pkgadd: ERROR: postinstall script did not complete successfully
 
Installation of <MSFTscx> failed.

The piece of the message above that stood out to me was this line:

/var/sadm/pkg/MSFTscx/install/postinstall: ENV=/usr/sislocal/profile: is not an identifier

I asked the customer to take a look at their Environmental Variables in /etc/profile and verify if there are any custom lines in there. We noticed there were custom lines, we removed these lines and attempted the installation again, it succeeded!

Set and Check User Rights Assignment via PowerShell

January 5, 2022

_{This post was last updated on November 5th, 2024}

I stumbled across this gem (weloytty/Grant-LogonAsService.ps1) that allows you to grant Logon as a Service Right for a User. I modified the script you can now run the Powershell script against multiple machines, users, and user rights.

Set User Rights

:arrow_down: How to get it

Set-UserRights.ps1 :arrow_left: Direct Download Link
or
Personal File Server - Set-UserRights.ps1 :arrow_left: Alternative Download Link
or
Personal File Server - Set-UserRights.txt :arrow_left: Text Format Alternative Download Link

All of the User Rights that can be set:

Privilege	PrivilegeName
SeAssignPrimaryTokenPrivilege	Replace a process level token
SeAuditPrivilege	Generate security audits
SeBackupPrivilege	Back up files and directories
SeBatchLogonRight	Log on as a batch job
SeChangeNotifyPrivilege	Bypass traverse checking
SeCreateGlobalPrivilege	Create global objects
SeCreatePagefilePrivilege	Create a pagefile
SeCreatePermanentPrivilege	Create permanent shared objects
SeCreateSymbolicLinkPrivilege	Create symbolic links
SeCreateTokenPrivilege	Create a token object
SeDebugPrivilege	Debug programs
SeDelegateSessionUserImpersonatePrivilege	Obtain an impersonation token for another user in the same session
SeDenyBatchLogonRight	Deny log on as a batch job
SeDenyInteractiveLogonRight	Deny log on locally
SeDenyNetworkLogonRight	Deny access to this computer from the network
SeDenyRemoteInteractiveLogonRight	Deny log on through Remote Desktop Services
SeDenyServiceLogonRight	Deny log on as a service
SeEnableDelegationPrivilege	Enable computer and user accounts to be trusted for delegation
SeImpersonatePrivilege	Impersonate a client after authentication
SeIncreaseBasePriorityPrivilege	Increase scheduling priority
SeIncreaseQuotaPrivilege	Adjust memory quotas for a process
SeIncreaseWorkingSetPrivilege	Increase a process working set
SeInteractiveLogonRight	Allow log on locally
SeLoadDriverPrivilege	Load and unload device drivers
SeLockMemoryPrivilege	Lock pages in memory
SeMachineAccountPrivilege	Add workstations to domain
SeManageVolumePrivilege	Perform volume maintenance tasks
SeNetworkLogonRight	Access this computer from the network
SeProfileSingleProcessPrivilege	Profile single process
SeRelabelPrivilege	Modify an object label
SeRemoteInteractiveLogonRight	Allow log on through Remote Desktop Services
SeRemoteShutdownPrivilege	Force shutdown from a remote system
SeRestorePrivilege	Restore files and directories
SeSecurityPrivilege	Manage auditing and security log
SeServiceLogonRight	Log on as a service
SeShutdownPrivilege	Shut down the system
SeSyncAgentPrivilege	Synchronize directory service data
SeSystemEnvironmentPrivilege	Modify firmware environment values
SeSystemProfilePrivilege	Profile system performance
SeSystemtimePrivilege	Change the system time
SeTakeOwnershipPrivilege	Take ownership of files or other objects
SeTcbPrivilege	Act as part of the operating system
SeTimeZonePrivilege	Change the time zone
SeTrustedCredManAccessPrivilege	Access Credential Manager as a trusted caller
SeUndockPrivilege	Remove computer from docking station

:notebook: Note

You may edit line 564 in the script to change what happens when the script is run without any arguments or parameters, this also allows you to change what happens when the script is run from the PowerShell ISE.

Here are a few examples:

Add Users

Single Users

Example 1

Add User Right “Allow log on locally” for current user:
.\Set-UserRights.ps1 -AddRight -UserRight SeInteractiveLogonRight
Example 2

Add User Right “Log on as a service” for CONTOSO\User:
.\Set-UserRights.ps1 -AddRight -Username CONTOSO\User -UserRight SeServiceLogonRight
Example 3

Add User Right “Log on as a batch job” for CONTOSO\User:
.\Set-UserRights.ps1 -AddRight -Username CONTOSO\User -UserRight SeBatchLogonRight
Example 4

Add User Right “Log on as a batch job” for user SID S-1-5-11:
.\Set-UserRights.ps1 -AddRight -Username S-1-5-11 -UserRight SeBatchLogonRight
Add Multiple Users / Rights / Computers

Example 5

Add User Right “Log on as a service” and “Log on as a batch job” for CONTOSO\User1 and CONTOSO\User2 and run on, local machine and SQL.contoso.com:
.\Set-UserRights.ps1 -AddRight -UserRight SeServiceLogonRight, SeBatchLogonRight -ComputerName "$env:COMPUTERNAME", "SQL.contoso.com" -UserName "CONTOSO\User1", "CONTOSO\User2"

Remove Users

Single Users

Example 1

Remove User Right “Allow log on locally” for current user:
.\Set-UserRights.ps1 -RemoveRight -UserRight SeInteractiveLogonRight
Example 2

Remove User Right “Log on as a service” for CONTOSO\User:
.\Set-UserRights.ps1 -RemoveRight -Username CONTOSO\User -UserRight SeServiceLogonRight
Example 3

Remove User Right “Log on as a batch job” for CONTOSO\User:
.\Set-UserRights.ps1 -RemoveRight -Username CONTOSO\User -UserRight SeBatchLogonRight
Example 4

Remove User Right “Log on as a batch job” for user SID S-1-5-11:
.\Set-UserRights.ps1 -RemoveRight -Username S-1-5-11 -UserRight SeBatchLogonRight
Remove Multiple Users / Rights / Computers

Example 5

Remove User Right “Log on as a service” and “Log on as a batch job” for CONTOSO\User1 and CONTOSO\User2 and run on, local machine and SQL.contoso.com:
.\Set-UserRights.ps1 -RemoveRight -UserRight SeServiceLogonRight, SeBatchLogonRight -ComputerName "$env:COMPUTERNAME", "SQL.contoso.com" -UserName "CONTOSO\User1", "CONTOSO\User2"

Check User Rights

:arrow_down: How to get it

Get-UserRights.ps1 :arrow_left: Direct Download Link
or
Personal File Server - Get-UserRights.ps1 :arrow_left: Alternative Download Link
or
Personal File Server - Get-UserRights.txt :arrow_left: Text Format Alternative Download Link

In order to check the Local User Rights, you will need to run the above (Get-UserRights), you may copy and paste the above script in your PowerShell ISE and press play.

Note

You may edit line 493 in the script to change what happens when the script is run without any arguments or parameters, this also allows you to change what happens when the script is run from the PowerShell ISE.

Here are a few examples:

Local Computer

Get Local User Account Rights and output to text in console:

.\Get-UserRights.ps1
# or
.\Get-UserRights.ps1 -UserName DOMAIN\Username

Remote Computer

Get Remote SQL Server User Account Rights:

.\Get-UserRights.ps1 -ComputerName SQL.contoso.com

Get Local Machine and SQL Server User Account Rights:

.\Get-UserRights.ps1 -ComputerName "$env:COMPUTERNAME", "SQL.contoso.com"

Output Types

Output Local User Rights on Local Machine as CSV in ‘C:\Temp’:

.\Get-UserRights.ps1 -FileOutputPath C:\Temp -FileOutputType CSV

Output to Text in ‘C:\Temp’:

.\Get-UserRights.ps1 -FileOutputPath C:\Temp -FileOutputType Text
# or
.\Get-UserRights.ps1 -FileOutputPath C:\Temp

PassThru object to allow manipulation / filtering:

.\Get-UserRights.ps1 -ComputerName SQL.contoso.com -UserName DOMAIN\SQLUser -PassThru | Where-Object {$_.Privilege -match 'SeInteractiveLogonRight'}
# or
.\Get-UserRights.ps1 -PassThru | Where-Object {$_.Privilege -match 'SeServiceLogonRight'}

Leave some feedback if this helped you! :v:

Create your own offline Certificate Request for SCOM Off-Domain Server

December 29, 2021

In the below example we are assuming your machine is named IIS-2019.

Create a new file on your machine and name it:

IIS-2019-CertReq.inf

Edit the file to include something similar to the following (NOTE: Be aware Operations Manager only uses the first CN name in the Subject):

[NewRequest]
Subject="CN=IIS-2019.contoso.com,OU=Servers,O=Support Team,L=Charlotte,S=North Carolina,C=US"
Exportable=TRUE ; Private key is exportable - Required for Server Authentication
KeyLength=2048
KeySpec=1 ; Key Exchange – Required for encryption
KeyUsage=0xf0 ; Digital Signature, Key Encipherment
MachineKeySet=TRUE
ProviderName = "Microsoft RSA SChannel Cryptographic Provider"
ProviderType = 12
KeyAlgorithm = RSA

; Optionally include the Certificate Template
; [RequestAttributes]
; CertificateTemplate="SystemCenterOperationsManager"

[EnhancedKeyUsageExtension]
OID=1.3.6.1.5.5.7.3.1 ; Server Authentication
OID=1.3.6.1.5.5.7.3.2  ; Client Authentication

Open an Administrator Command Prompt and navigate to where you saved the above file.
Run the following:

Certreq -New -f IIS-2019-CertReq.inf IIS-2019-CertRequest.req

:notebook: Note

The server where you run the above Certreq command will be where the Certificate Private Key will be stored.

Upload the above (IIS-2019-CertRequest.req) file to your Certificate Authority.
…
Once you receive back your signed certificate, import the Certificate into the Local Computer Personal Certificate Store:

certlm.msc

Run this Powershell script to check the certificate you imported:
https://github.com/blakedrumm/SCOM-Scripts-and-SQL/blob/master/Powershell/Test-SCOMCertificate.ps1

Run it like this:
```
.\Test-SCOMCertificate.ps1 -All
```
You can also copy/paste the script to an Powershell ISE (Running as Administrator), you just need to edit line 770 to include the arguments you want to run.

On a side note. If you run the SCOM Certificate Checker script above and it shows an output that looks like this:

You may also notice that the Private Key for the Certificate is missing:

It is possible you may need to run the following command in an Administrator Command Prompt to restore the Keyspec and Private Key (replace the numbers & letters after my with the serial number of your Certificate):

certutil -repairstore my 1f00000008c694dac94bcfdc4a000000000008

After you run the certutil command above, you will notice the Certificate is now showing a Private Key (notice the key icon):

You should now see this when you run the SCOM Certificate Checker Powershell Script:

Now you just need to import the Certificate with MOMCertImport (located on the SCOM Installation Media):

Right Click and Run the Program as Administrator, select the certificate you imported:

Lastly, you will need to restart the Microsoft Monitoring Agent (HealthService). You can do this via Powershell with this command:

Restart-Service HealthService

After restarting the Microsoft Monitoring Agent (HealthService). You will wait until you see the following Event ID in the Operations Manager Event Log (20053) confirming that the certificate has been loaded:

Log Name: Operations Manager
Source: OpsMgr Connector
Date: 2/28/2022 10:35:36 AM
Event ID: 20053
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: MS02-2019.contoso.com
Description:
The OpsMgr Connector has loaded the specified authentication certificate successfully.

Don’t forget the Management Server
The Management Server also needs to have a certificate requested for itself, and imported into the Personal Store in the Local Machine Certificates. You also need to run MOMCertImport.exe on the Management Server for the Management Server certificate. Otherwise the communication between the Management Server and the Agent or Gateway Server will not work via certificates.

Resolve Operations Manager Event ID 33333

August 16, 2021

:book: Introduction

I had a case recently where my customer was having issues with the following Event ID 33333 on all the Management Servers in the Management Group.

This is the Exact Event Information:

Log Name:      Operations Manager
Source:        DataAccessLayer
Date:          7/7/2021 2:19:52 PM
Event ID:      33333
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      ManagementServer1.contoso.com
Description:
Data Access Layer rejected retry on SqlError:
 Request: p_StateChangeEventProcess -- (BaseManagedEntityId=2ee6ce9a-6ec1-4a16-335b-700136600z60), (EventOriginId=b48055d3-18f2-40f7-a0e7-a0f8bd1b44a3), (MonitorId=f1baeb56-8cce-f8c7-79ae-d69796c9d926), (NewHealthState=3), (OldHealthState=1), (TimeGenerated=7/7/2021 6:19:43 PM), (Context=), (RETURN_VALUE=0)
 Class: 14
 Number: 2627
 Message: Violation of PRIMARY KEY constraint 'PK_StateChangeEvent'. Cannot insert duplicate key in object 'dbo.StateChangeEvent'. The duplicate key value is (b48055d3-18f2-40f7-a0e7-a0f8bd1b44a3).

:page_with_curl: How to fix

This Powershell Script will allow you to detect for Event ID 33333 being generated and get a count of unique Servers referenced across your OperationsManager Event Log:

$events = (Get-EventLog -LogName 'Operations Manager' -Source 'DataAccessLayer' -ErrorAction SilentlyContinue | Where-Object { $_.EventID -eq 33333 })
# If Event 33333 found in the OperationsManager Event Log, do the below
if (($events.Message -like "*Violation of PRIMARY KEY constraint 'PK_StateChangeEvent'. Cannot insert duplicate key in object 'dbo.StateChangeEvent'. The duplicate key value is*") -and ($events.Message -like "*f1baeb56-8cce-f8c7-79ae-d69796c9d926*"))
	{
		$message = $events | %{ ($_ | Select-Object -Property Message -ExpandProperty Message) }
		$matches = $message -split "," | select-string "MonitorId=(.*)"
		$match = $matches.matches.groups[1].value.TrimEnd(")")
		$i++
		$i = $i
		"Found $($message.count) issues with the Event ID 33333 (Monitor Id: $match), see the following article:`n   https://kevinholman.com/2017/05/29/stop-healthservice-restarts-in-scom-2016/"
	}

Run the above script on each of your Management Servers in your Management Group.

I found that this correlates to this article written by Kevin Holman: https://kevinholman.com/2017/05/29/stop-healthservice-restarts-in-scom-2016/

Following the steps in the article above, will resolve the issues you are having with Event ID: 33333 Violation of PRIMARY KEY constraint 'PK_StateChangeEvent'. Cannot insert duplicate key in object 'dbo.StateChangeEvent'. .

SCOM Clear Cache Script

July 22, 2021

_{Article last updated on April 12th, 2023}

:arrow_down: How to get it

Clear-SCOMCache.ps1 :arrow_left: Direct Download Link
or
Personal File Server - Clear-SCOMCache.ps1 :arrow_left: Alternative Download Link
or
Personal File Server - Clear-SCOMCache.txt :arrow_left: Text Format Alternative Download Link

The script without any modifications or parameters clears the Operations Manager cache only on the local server, nothing else.

:classical_building: Argument List

Parameter	Description
-All	Optionally clear all caches that SCOM could potentially use that doesnt require a reboot. Flushing DNS, Purging Kerberos Tickets, Resetting NetBIOS over TCPIP Statistics. (Combine with -Reboot for a full clear cache)
-Reboot	Optionally reset winsock catalog, stop the SCOM Services, clear SCOM Cache, then reboot the server. This will always perform on the local server last.
-Servers	Optionally each Server you want to clear SCOM Cache on. Can be an Agent, Management Server, or SCOM Gateway. This will always perform on the local server last.
-Shutdown	Optionally shutdown the server after clearing the SCOM cache. This will always perform on the local server last.
-Sleep	Time in seconds to sleep between each server.

:question: Examples

Clear all Gray SCOM Agents

#Get the SystemCenter Agent Class
$agent = Get-SCOMClass | where-object{$_.name -eq "microsoft.systemcenter.agent"}
#Get the grey agents
$objects = Get-SCOMMonitoringObject -class:$agent | where {$_.IsAvailable -eq $false}
.\Clear-SCOMCache.ps1 -Servers $objects

Clear SCOM cache on every Management Server in Management Group

Get-SCOMManagementServer | .\Clear-SCOMCache.ps1

Clear SCOM cache on every Agent in the Management Group

Get-SCOMAgent | .\Clear-SCOMCache.ps1

Clear SCOM cache and reboot the Servers specified

.\Clear-SCOMCache.ps1 -Servers AgentServer.contoso.com, ManagementServer.contoso.com -Reboot

Clear SCOM cache and shutdown the Servers specified

.\Clear-SCOMCache.ps1 -Servers AgentServer.contoso.com, ManagementServer.contoso.com -Shutdown

System Center Operations Manager - Data Collector

July 6, 2021

:book: Introduction

This tool was designed to assist with troubleshooting complex System Center Operations Manager issues.

:arrow_down_small: Quick Download Link

https://aka.ms/SCOM-DataCollector

:page_facing_up: Personal Webpage

Data Collector

https://files.blakedrumm.com/SCOM-DataCollector.zip

Report Builder

https://files.blakedrumm.com/ReportBuilder.zip

:link: Github Link

Data Collector

https://github.com/blakedrumm/SCOM-Scripts-and-SQL/releases/latest/download/SCOM-DataCollector.zip

Report Builder

https://github.com/blakedrumm/SCOM-Scripts-and-SQL/releases/latest/download/ReportBuilder.zip

:red_circle: Requirements

System Center Operations Manager - Management Server
Administrator Privileges
Powershell 4

:page_with_curl: Instructions

Download the zip file and extract zip file to a directory (ex. C:\Data Collector). You have 2 options for running this script.

Right Click the SCOM Data Collector script and select Run with Powershell.
Open an Powershell shell as Adminstrator and change to the directory the SCOM Data Collector Powershell Script is located, such as:
```
cd C:\Data Collector
.\DataCollector-v4.0.0.ps1
```

:o: Optional

You have the ability to run this script as any user you would like when you start the script without any switches. The default is the System Center Data Access Service account.

Run this script on a Operations Manager Management Server to gather their SQL server names and DB names from the local registry. Otherwise user will need to manually input names. It will attempt to query the SQL Server Instance remotely, and will create CSV output files in the Output folder located in the SCOM Data Collector script directory.The SCOM Data Collector has the ability to query multiple databases in the SCOM SQL Instance (master, OperationsManager, OperationsManagerDW), having a high level of rights to SQL is preferred for a full gather.

After the script has completed you will see that the Output folder that is temporary created during script execution is removed. The Data Collector will automatically create a zip file in the same directory as the SCOM Data Collector Powershell Script named something similar to this:
SDC_Results_04_04_1975.zip

This script has the ability to gather the following information:

Data from Management Server(s), Operations Manager SQL Server(s). You can also gather from Servers(s) specified with -Servers.
Event Logs – Application, System, OperationsManager
SCOM Version Installed
Update Rollup Information for SCOM Upgrades
SQL Queries that collect information about many aspects of your environment (too many queries to go into detail here, here are some of the queries it uses: https://github.com/blakedrumm/SCOM-Scripts-and-SQL/tree/master/SQL%20Queries)
Windows Updates installed on Management Servers / SQL Server
Service Principal Name (SPN) Information for SCOM Management Servers
Local Administrators Group on each Management Server and any other servers you specify
Local User Account Rights on each Management Server and any other servers you specify
Database Information / DB Version
SCOM RunAs Account Information
Check TLS 1.2 Readiness
TLS Settings on each Management Server and SQL Server
MSInfo32
Sealed / Unsealed MPs
Clock Synchronization
Latency Check (Ping Test)
Rules / Monitors in your SCOM Environment
Get Run As Accounts from SCOM Management Group
Test SCOM Ports
Best Practice Analyzer to verify you are following SCOM Best Practices (only a few items being checked currently)
Gathers Group Policy settings on each Management Server and SQL Server
Gathers installed Software on each Management Server
Management Group Overall Health and verify Configuration matches across Management Servers in Management Group
Check SCOM Certificates for Validity / Usability
SCOM Install / Update Logs
IP Address of each Management Server
Gather SCOM Configuration from registry and configuration file
this list is not complete…

:question: Examples

:o: Optional

If you know you have (read) Query rights against the DB(s) and Administrator permissions on the Management Servers, run any Switch (-Command) with -AssumeYes (-Yes). Otherwise you will need to provide an account that has permissions at runtime.

Available Switches

Every Switch Available:

.\DataCollector.ps1 -All -ManagementServers "<array>" -Servers "<array>" -AdditionalEventLogs "<array>" -GetRulesAndMonitors -GetRunAsAccounts -CheckTLS -CheckCertificates -GetEventLogs -ExportMPs -GPResult -SQLLogs -CheckPorts -GetLocalSecurity -GetInstalledSoftware -GetSPN -AssumeYes -GetConfiguration -CheckGroupPolicy -GetInstallLogs -SkipSQLQueries -SQLOnly -SQLOnlyOpsDB -SQLOnlyDW -BuildPipeline -CaseNumber "<string>" -ExportSCXCertificates -ExportMSCertificates -GenerateHTML -GetNotificationSubscriptions -GetUserRoles -LeastAmount -MSInfo32 -NoSQLPermission -PingAll -SCXAgents "<array>" -SCXUsername "<string>" -SCXMaintenanceUsername "<string>" -SCXMonitoringUsername "<string>" -SkipBestPracticeAnalyzer -SkipConnectivityTests -SkipGeneralInformation -SQLLogs

All Switches

This will allow you to run every switch available currently:

.\DataCollector.ps1 -All
.\DataCollector.ps1 -All -Servers Agent1
.\DataCollector.ps1 -All -Servers Agent1, Agent2, Agent3
.\DataCollector.ps1 -All -Servers Agent1 -ManagementServer MS01-2019.contoso.com, MS02-2019.contoso.com
.\DataCollector.ps1 -All -Yes

To see the built in menu, run the script with no arguments or switches:

.\DataCollector.ps1

You can also right click the .ps1 file and Run with Powershell.

Certificates

To Check the Certificate(s) Installed on the Management Server(s) in the Management Group, and an Server:

.\DataCollector.ps1 -CheckCertificates -Servers AppServer1.contoso.com

To Check the Certificate(s) Installed on the Management Server(s) in the Management Group:

.\DataCollector.ps1 -CheckCertificates

Gather only SQL Queries

To gather only the SQL Queries run the following:

.\DataCollector.ps1 -SQLOnly

If you know the account running the Data Collector has permissions against the SCOM Databases, run this:

.\DataCollector.ps1 -SQLOnly -Yes

Event Logs

To gather Event Logs from 3 Agents and the Management Server(s) in the Current Management Group:

.\DataCollector.ps1 -GetEventLogs -Servers Agent1.contoso.com, Agent2.contoso.com, Agent3.contoso.com

To just gather the Event Logs from the Management Server(s) in the Management Group:

.\DataCollector.ps1 -GetEventLogs

Management Packs

To Export Installed Management Packs:

.\DataCollector.ps1 -ExportMPs

RunAs Accounts

To Export RunAs Accounts from the Management Server:

.\DataCollector.ps1 -GetRunAsAccounts

Check TLS 1.2 Readiness

To Run the TLS 1.2 Hardening Readiness Checks on every Management Server and SQL SCOM DB Server(s) in the Management Group:

.\DataCollector.ps1 -CheckTLS

Blake Drumm - Technical Blog

Sending Emails with Azure Communication Services (ACS) — A C# Console Application with Custom Headers

💡 Introduction

💾 Download the Code

📬 What the Application Does

🏗️ High-Level Architecture

⚙️ Prerequisites

1. Install .NET

2. Create an Azure Communication Services Resource

3. Configure ACS Email

4. Install the Required SDK

🔑 Configure the Connection String

Windows (Command Prompt)

PowerShell

▶️ Build the Application

▶️ Run the Application

🖥️ Interactive Mode

📧 Email Template and Message Content

🏷️ Custom Email Headers

🔎 Tracking and Correlation

⚙️ How It Works Internally (Deep Dive)

Step 1 — Initialize the Email Client

Step 2 — Build the Email Message

Step 3 — Submit the Send Request

Step 4 — Poll the Operation

Step 5 — Log the Results

🧪 When This Tool Is Useful

⚠️ Common Issues When Testing ACS Email

401 Unauthorized

Sender Not Verified

Email Never Arrives

🧠 Final Thoughts

Deep-Dive Email Delivery Analysis with KQL - Azure Communication Services (ACS)

💡 Introduction

📋 Prerequisites

📩 The Problem with Raw ACS Email Logs

🔍 What This Query Does

⚙️ Data Sources Used

1️⃣ ACSEmailSendMailOperational

2️⃣ ACSEmailStatusUpdateOperational

📜 The Complete Query

⏱️ Delivery Latency Measurement

📦 Message Characteristics and Size Handling

💥 Bounce Classification

📜 SMTP and Enhanced SMTP Code Translation

🌐 Recipient and Server Intelligence

🎯 Common Usage Patterns

Filter by Specific Sender

Find All Bounces to a Specific Domain

Identify Slow Deliveries (>10 seconds)

Messages with Large Size or Attachments

All Hard Bounces in Last 24 Hours

📊 Aggregation Examples for Dashboards

Delivery Success Rate by Recipient Domain

Average Delivery Time by Hour

Top Failure Reasons

Bounce Rate by Sender

Message Size Distribution

⚠️ Limitations & Known Issues

Subject Lines Are Not Available

IsHardBounce Casing

Time Range Performance

Correlation Timing

🧪 When to Use This Query

🚀 Next Steps

📚 Related Resources

🧠 Final Thoughts

Custom IAM Roles Are Not Supported - Azure Communication Services (ACS)

💡 Introduction

🔐 The Issue

🔍 The Root Cause

✅ The Resolution

✔ Required Built-In Role

⚠️ Important Notes About ACS RBAC

🧭 Design Guidance

🧠 Final Thoughts

Outbound Calling with a Lightweight Console Tool - Azure Communication Services (ACS)

:bulb: Introduction

:telephone_receiver: Why Build a Tool for ACS Outbound Calling?

:mag: Understanding ACS CallAutomation Behavior

:mag: The Problem — Orphaned `.status` Files