Troubleshooting ARC -Agent (Connected Machine Agent) Connection Issues

This article has not been completed yet. However, it may already contain helpful Information and therefore it has been published at this stage.

Prerequisites:

A server with an Arc Agent installed

Service Tags:

(that need to be checked / included.)

AzureAD (previously AzureActiveDirectory)
AzureTrafficManager
AzureResourceManager
AzureArcInfrastructure
AzureStorage (previously Storage)
WindowsAdminCenter

For a list of IP addresses for each service tag/region, see the JSON file below:

Download Link: Azure IP Ranges and Service Tags – Public Cloud.

Microsoft publishes weekly updates containing each Azure Service and the IP ranges it uses

I took the effort and built a fully automated solution for this.
Please find below the PowerShell - Script (Tags.ps1):

$rawhtml = Invoke-RestMethod -Uri 'https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519' -UseBasicParsing
# Dowload https://www.microsoft.com/download/details.aspx?id=56519

# Source file location
$source = [Regex]::Match($rawhtml, 'https://download.microsoft.com[^"]*').Value
# Destination to save the file
$destination = 'c:\temp\ServiceTags_Public.json'
#Download the file
Invoke-WebRequest -Uri $source -OutFile $destination

$lookups='AzureArcInfrastructure','AzureAD','AzureTrafficManager','AzureResourceManager','AzureStorage','WindowsAdminCenter'

$ServiceTags = Get-Content -path c:\temp\ServiceTags_Public.json -Raw | ConvertFrom-Json

$region = $ServiceTags.values | Select-Object -expand properties | Select-Object @{Label="Region";Expression={($_.region)}} -Unique | Sort-Object -Property Region | Out-GridView -OutputMode Single -Title 'Select an Azure Region'

# $ServiceTags.values | Select-Object -expand properties | Select-Object region, systemService, addressPrefixes | Where-Object { $_.region -eq $region.Region -or $_.region -eq "" }

$result= @()
foreach ($lookup in $lookups){
$tagresult = $ServiceTags.values | Select-Object -expand properties | Select-Object region, systemService, addressPrefixes | Where-Object { $_.region -eq $region.Region -or $_.region -eq "" } | Where-Object { $_.systemService -eq $lookup}
$result += $tagresult
}
$result
# $result | Select-Object region,systemService
# $result | Select-Object -ExpandProperty addressPrefixes
Write-Output "---------------------------------------" 
foreach ($object in $result)  {
        Write-Output "Service:" ($object.systemService).ToString()
        Write-Output "Region: " ($object.region).ToUpper()
        Write-Output "IP-Count: " $object.addressPrefixes.Count
        Write-Output "IPs: " $object.addressPrefixes
        Write-Output "---------------------------------------" 
}

# Refernence:
# https://learn.microsoft.com/en-us/azure/azure-arc/servers/network-requirements?tabs=azure-cloud

Expected Result:

Selecting the right Azure Region (extracted from the JSON - File)

Result - A list of IPs for each Service Tag

URLs to check:

(extracted from the official Microsoft documentation and the further references listed at the end of the blog post)

Static:

aka.ms
download.microsoft.com
packages.microsoft.com
management.azure.com
login.windows.net
login.microsoftonline.com
pas.windows.net
guestnotificationservice.azure.com
dc.services.visualstudio.com
www.office.com

Depending on the region:

*.guestconfiguration.azure.com
*.guestnotificationservice.azure.com
*.his.arc.azure.com
*.blob.core.windows.net
azgn*.servicebus.windows.net
*.servicebus.windows.net

Unfortunately, I was not able to verify / find out all the correct addresse

Microsoft's description is not 100% exact, even if it seems so when reading the following statement.

To translate the *.servicebus.windows.net wildcard into specific endpoints, use the command \GET https://guestnotificationservice.azure.com/urls/allowlist?api-version=2020-01-01&location=<location>. Within this command, the region must be specified for the <location> placeholder.

This provides the following addresses:

If you use the following URL you will also get a usable result for the *.his.arc.azure.com addresses:

https://gbl.his.arc.azure.com/discovery?location=westeurope&api-version=1.1-preview

So the abbreviation for West Europe is "weu".

For the following addresses I could not collect any URLs.

# *.guestnotificationservice.azure.com
 # Test-NetConnection "weu.guestnotificationservice.azure.com" -Port 443
 # Test-NetConnection "westeurope.guestnotificationservice.azure.com" -Port 443
 # Test-NetConnection "weuguestnotificationservice.azure.com" -Port 443
 # Test-NetConnection "westeuropeguestnotificationservice.azure.com" -Port 443

# *.guestconfiguration.azure.com
# Test-NetConnection "weuguestconfiguration.azure.com" -Port 443
# Test-NetConnection "weu.guestconfiguration.azure.com" -Port 443
# Test-NetConnection "westeurope.guestconfiguration.azure.com" -Port 443
# Test-NetConnection "westeuropeguestconfiguration.azure.com" -Port 443

I included all these points in a script that handles them fully automatically:

If there were any issues, they would show up at that point.

Modified Script - Connection.ps1:

$rawhtml = Invoke-RestMethod -Uri 'https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519' -UseBasicParsing
# Dowload https://www.microsoft.com/download/details.aspx?id=56519

# Source file location
$source = [Regex]::Match($rawhtml, 'https://download.microsoft.com[^"]*').Value
# Destination to save the file
$destination = 'c:\temp\ServiceTags_Public.json'
#Download the file
Invoke-WebRequest -Uri $source -OutFile $destination

$lookups='AzureArcInfrastructure','AzureAD','AzureTrafficManager','AzureResourceManager','AzureStorage','WindowsAdminCenter'

$ServiceTags = Get-Content -path c:\temp\ServiceTags_Public.json -Raw | ConvertFrom-Json

$region = ($ServiceTags.values | Select-Object -expand properties | Select-Object @{Label="Region";Expression={($_.region)}} -Unique | Sort-Object -Property Region | Out-GridView -OutputMode Single -Title 'Select an Azure Region').Region

$urls = @('aka.ms','download.microsoft.com','packages.microsoft.com','management.azure.com','login.windows.net','login.microsoftonline.com','pas.windows.net','guestnotificationservice.azure.com','dc.services.visualstudio.com','www.office.com','agentserviceapi.guestconfiguration.azure.com')

# $ServiceTags.values | Select-Object -expand properties | Select-Object region, systemService, addressPrefixes | Where-Object { $_.region -eq $region.Region -or $_.region -eq "" }
# ((Invoke-RestMethod -Uri "https://gbl.his.arc.azure.com/discovery?location=$region&api-version=1.1-preview").split('.')[0]).substring(8)
# (Invoke-RestMethod -Uri "https://gbl.his.arc.azure.com/discovery?location=$region&api-version=1.1-preview")

$urls += (Invoke-RestMethod -Uri ("https://gbl.his.arc.azure.com/discovery?location=$region&api-version=1.1-preview")).substring(8)
$urls += (Invoke-RestMethod "https://guestnotificationservice.azure.com/urls/allowlist?api-version=2020-01-01&location=$region")

foreach ($url in $urls) {
     if (((Test-NetConnection $url -Port 443 -WarningAction:SilentlyContinue).TCPTESTSucceeded) -ne $true) {Write-Host "$url is not reachable"}
     $url
}

Verify the connection using the agent:

azcmagent check --location westeurope

Result:

Check for Problems using the AMA - Troubleshooting - Tool

Open a PowerShell or Windows Terminal as an Administrator

Change directory to:

 cd "C:\Packages\Plugins\Microsoft.AzureMonitorWindowsAgent\<AMA Agent Version Number>\Troubleshooter"

Type AgentTroubleshooter.exe --ama and press the ENTER

Result:

If nothing helps and all the measures listed here do not provide any further information or progress, I recommend uninstalling the agent completely and then performing a clean reinstallation.