Episode 4 — The data-collection pipeline

Series: AZ-104 · Monitor and Maintain Azure Resources — micro-learning. Where we are: having met logs and KQL in Episode 3, we now trace HOW data actually reaches a workspace — the activity log, resource logs, diagnostic settings, the Azure Monitor Agent and DCRs — before we build alerts on it in Episode 5.

Why it matters

Azure Monitor is only as good as the data you route into it. Some data is collected by default and some is not collected until you configure it — and knowing which is which is the single most tested idea of this skill. The mental anchor is control plane vs data plane: the activity log records management operations for free, while resource logs record what happens inside a resource and require a diagnostic setting. Layer on the Azure Monitor Agent (AMA) for guest-OS data, all steered by data collection rules (DCRs), and you have the full pipeline.

Learning objectives

• By the end, you can explain the difference between control-plane (activity log) and data-plane (resource logs) data and say which is collected by default.
• By the end, you can configure a diagnostic setting to route logs and metrics to a workspace, storage account, or event hub, and recall its key limits.
• By the end, you can describe what the Azure Monitor Agent collects and why it needs DCRs.
• By the end, you can explain what a DCR, a DCR association, and a transformation do — and use transformations to cut cost.

The anchor: control plane vs data plane

Everything in this episode hangs on one distinction.

Mnemonic: control the box vs use the box. Managing the box (create/resize/delete) is the activity log; using what's inside the box (read a secret, run a query) is a resource log.

The activity log (control plane, free by default)

The activity log is system-generated and immutable — you can't change or delete entries — and it is collected by default with no configuration.

Key facts to lock in:

• Entries are available for analysis and alerting within 3 to 20 minutes of the event.
• Retention is 90 days, then Azure deletes them. You are not charged during this window, regardless of volume.
• It typically does not capture read operations — only changes (create/update/delete) and initiated actions.
• It is the only place that stores resource-creator information (who created a resource, and when).
• To keep data past 90 days you must export via a diagnostic setting (retention up to 12 years).

Scopes

• Subscription level — the default; captures events created directly by resource providers.
• Tenant / management-group level — captures only Azure Resource Manager events in that hierarchy. In the portal, use Monitor > Activity log, change the Activity pull-down to Directory Activity (tenant), or Management groups > group > Activity log. A diagnostic setting on the highest-level management group covers all groups beneath it (watch for duplicate events).

Portal filters

Resource, Resource type, Operation, Event initiated by, Event category.

Retrieve it programmatically

az monitor activity-log list --subscription "$subscriptionId" --offset "14d"

Get-AzActivityLog -StartTime (Get-Date).AddDays(-14) -EndTime (Get-Date)

Export to a workspace → the AzureActivity table

Exporting to a Log Analytics workspace lets you correlate with other logs, build log alerts, use Power BI, and retain beyond 90 days. There are no data-ingestion charges for activity logs; a retention charge applies only past the default 90 days. Data lands in the AzureActivity table, which powers the Activity log insights workbook.

AzureActivity
| where CategoryValue == "Administrative"

Case gotcha: AzureActivity fields can vary in case — compare with =~ or tolower().

Diagnostic settings (the router)

A diagnostic setting sends three source types to destinations. Create one setting per resource you want to collect from.

Destinations

Constraints you must memorize:

• The destination must already exist before you create the setting.
• Regional resources need a Storage/Event Hubs destination in the same region.
• With VNets enabled, allow trusted Microsoft services to bypass the firewall.
• One of each destination type per setting (want two workspaces? create two settings).
• Up to five diagnostic settings per resource.
• You can't send a resource's own logs back to itself (a storage account/event hub can't be the destination for its own logs) — an infinite loop.

Category groups

Instead of individual categories, pick a category group:

• allLogs — all categories for the resource.
• audit — logs recording customer interactions with data/settings.

If you use a category group you can't also select individual categories. Not every service offers them.

Create one

$KV = Get-AzKeyVault -ResourceGroupName <rg> -VaultName <vault>
$Law = Get-AzOperationalInsightsWorkspace -ResourceGroupName <rg> -Name <workspace>
$metric = New-AzDiagnosticSettingMetricSettingsObject -Enabled $true -Category AllMetrics
$log = New-AzDiagnosticSettingLogSettingsObject -Enabled $true -CategoryGroup allLogs
New-AzDiagnosticSetting -Name 'KeyVault-Diagnostics' -ResourceId $KV.ResourceId -WorkspaceId $Law.ResourceId -Log $log -Metric $metric -Verbose

az monitor diagnostic-settings create \
--name KeyVault-Diagnostics \
--resource /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/mykeyvault \
--logs    '[{"category": "AuditEvent","enabled": true}]' \
--metrics '[{"category": "AllMetrics","enabled": true}]' \
--storage-account <storage-account-id> \
--workspace <workspace-id> \
--event-hub-rule <event-hub-authorization-rule-id> \
--event-hub <event hub name> \
--export-to-resource-specific true

Exam trap: New-AzDiagnosticSetting can't configure the activity log. For the activity log via CLI use az monitor diagnostic-settings subscription create/update (a subscription-scope setting).

Latency and cost

• After creating a setting, data should flow within 90 minutes. No data after 24 hours = no logs generated or a routing misconfig (try disable/re-enable).
• Control cost: collect only needed categories; skip platform metrics unless you need them in the workspace (they're already in Metrics Explorer for free); use transformations to filter within a category.

Azure Monitor Agent (AMA) — guest-OS data

The activity log and resource logs describe Azure's view of a resource. To see inside the operating system of a VM, you need an agent. Without an agent you can collect only host-machine data — no access to the guest OS, its logs, or running processes.

• AMA collects guest-OS logs and performance data from Azure VMs and hybrid VMs (other clouds / on-premises, connected via Azure Arc).
• It delivers to Azure Monitor, and feeds Microsoft Sentinel and Microsoft Defender for Cloud.
• It is the supported agent, replacing the legacy Log Analytics agent and diagnostic extensions.
• No cost for the agent — you pay only for ingestion and storage.
• Install via VM extension, Azure Policy, or by enabling VM insights.
• What it collects is defined entirely by DCRs. (You'll see AMA-driven guest logs again in Episode 6.)

Data collection rules (DCRs) — the ETL brain

A DCR is an ETL-like configuration stored as a first-class Azure resource (find them under Monitor > Data Collection Rules). A DCR specifies:

• What data to collect and send.
• The schema of incoming data.
• Transformations to apply before storage.
• The destination.

DCR associations (DCRAs)

A DCRA links a resource to a DCR. The relationship is many-to-many: one DCR can serve many resources, and a single resource can have up to 30 DCRs.

Transformations

Transformations are KQL queries that run against each record before it is stored. Use them to:

• Filter unneeded data to reduce ingestion cost.
• Remove sensitive data so it's never persisted.
• Reshape data to match the destination schema.
• Enrich or fan out to multiple destinations.

DCRs are a zone-redundant service, created in one region and backed up to the paired region within the same geography.

The whole pipeline, at a glance

Numbers & names to memorize

Apply it

Scenario 1. A security team wants an immutable, long-term archive of all key vault secret-read operations for compliance, plus real-time streaming into a third-party SIEM. What do you configure, and to which destinations?

Scenario 2. A VM's CPU metrics look fine in Metrics Explorer, but the team can't see the Windows Event Logs from inside the VM. Nothing is misconfigured on the host. What is missing and how do you add it while keeping ingestion cost down?

Q3 (single-select). Which data is collected by default, with no configuration?

• A. Resource logs
• B. Guest-OS performance counters
• C. The activity log
• D. Windows Event Logs

Q4 (multi-select — choose two). Which statements about diagnostic settings are true?

• A. A resource can have up to 5 diagnostic settings.
• B. A single setting can target two Log Analytics workspaces at once.
• C. The destination must already exist before you create the setting.
• D. New-AzDiagnosticSetting configures the activity log export.

Answers

• Scenario 1: Create a diagnostic setting on the key vault. Enable the appropriate resource-log category (e.g. audit) and route to two destinations: an Azure Storage account with immutable storage (compliance archive) and an Azure Event Hubs namespace (stream to the SIEM). Both must pre-exist and be in the vault's region for regional resources.
• Scenario 2: There is no agent in the guest OS, so only host data is available. Deploy the Azure Monitor Agent and associate a DCR that collects Windows Event Logs; add a transformation in the DCR to filter to only the needed events, cutting ingestion cost.
• Q3: C. The activity log is system-generated and collected by default; resource logs and guest-OS data are not.
• Q4: A and C. One destination of each type per setting (so not two workspaces — B is false), and New-AzDiagnosticSetting can't do the activity log (D is false).

Recap

• Control plane = activity log (management ops, free, 90 days, immutable, resource-creator info); data plane = resource logs (inside the resource, need a diagnostic setting).
• Platform metrics and the activity log are automatic; resource logs are not.
• Diagnostic settings route the three sources to LAW / Storage / Event Hubs / partner; limits: destination must exist, same region for regional resources, 1 of each type per setting, 5 settings per resource, no self-loop.
• AMA brings guest-OS data (Azure + Arc-connected hybrid); the agent is free, and its behavior is defined by DCRs.
• DCRs are ETL configs with transformations (filter to cut cost, redact, reshape, enrich); DCRAs are many-to-many, up to 30 DCRs per resource; the service is zone-redundant.
• Export the activity log to a workspace to keep it past 90 days and query AzureActivity — no ingestion charge.

Next up

Episode 5 — now that the data is flowing, we turn it into action: alerts, action groups and alert processing rules, including alerts on the activity log itself.