Troubleshooting ARC -Agent (Connected Machine Agent) Connection Issues

This article has not been completed yet. However, it may already contain helpful Information and therefore it has been published at this stage.

Prerequisites:

  • A server with an  Arc Agent installed

Service Tags:

(that need to be checked / included.)

  • AzureAD (previously AzureActiveDirectory)
  • AzureTrafficManager
  • AzureResourceManager
  • AzureArcInfrastructure
  • AzureStorage (previously Storage)
  • WindowsAdminCenter

For a list of IP addresses for each service tag/region, see the JSON file below:

Download Link:  Azure IP Ranges and Service Tags – Public Cloud.

Microsoft publishes weekly updates containing each Azure Service and the IP ranges it uses

I took the effort and built a fully automated solution for this.
Please find below the PowerShell - Script (Tags.ps1):

$rawhtml = Invoke-RestMethod -Uri 'https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519' -UseBasicParsing
# Dowload https://www.microsoft.com/download/details.aspx?id=56519

# Source file location
$source = [Regex]::Match($rawhtml, 'https://download.microsoft.com[^"]*').Value
# Destination to save the file
$destination = 'c:\temp\ServiceTags_Public.json'
#Download the file
Invoke-WebRequest -Uri $source -OutFile $destination

$lookups='AzureArcInfrastructure','AzureAD','AzureTrafficManager','AzureResourceManager','AzureStorage','WindowsAdminCenter'

$ServiceTags = Get-Content -path c:\temp\ServiceTags_Public.json -Raw | ConvertFrom-Json

$region = $ServiceTags.values | Select-Object -expand properties | Select-Object @{Label="Region";Expression={($_.region)}} -Unique | Sort-Object -Property Region | Out-GridView -OutputMode Single -Title 'Select an Azure Region'

# $ServiceTags.values | Select-Object -expand properties | Select-Object region, systemService, addressPrefixes | Where-Object { $_.region -eq $region.Region -or $_.region -eq "" }

$result= @()
foreach ($lookup in $lookups){
$tagresult = $ServiceTags.values | Select-Object -expand properties | Select-Object region, systemService, addressPrefixes | Where-Object { $_.region -eq $region.Region -or $_.region -eq "" } | Where-Object { $_.systemService -eq $lookup}
$result += $tagresult
}
$result
# $result | Select-Object region,systemService
# $result | Select-Object -ExpandProperty addressPrefixes
Write-Output "---------------------------------------" 
foreach ($object in $result)  {
        Write-Output "Service:" ($object.systemService).ToString()
        Write-Output "Region: " ($object.region).ToUpper()
        Write-Output "IP-Count: " $object.addressPrefixes.Count
        Write-Output "IPs: " $object.addressPrefixes
        Write-Output "---------------------------------------" 
}

# Refernence:
# https://learn.microsoft.com/en-us/azure/azure-arc/servers/network-requirements?tabs=azure-cloud

Expected Result:

Donwloading the JSON - File
Selecting the right Azure Region (extracted from the JSON - File)
Result - A list of IPs for each Service Tag

URLs to check:

(extracted from the official Microsoft documentation and the further references listed at the end of the blog post)

Static:

  • aka.ms
  • download.microsoft.com
  • packages.microsoft.com
  • management.azure.com
  • login.windows.net
  • login.microsoftonline.com
  • pas.windows.net
  • guestnotificationservice.azure.com
  • dc.services.visualstudio.com
  • www.office.com

Depending on the region:

  • *.guestconfiguration.azure.com
  • *.guestnotificationservice.azure.com
  • *.his.arc.azure.com
  • *.blob.core.windows.net
  • azgn*.servicebus.windows.net
  • *.servicebus.windows.net

Unfortunately, I was not able to verify / find out all the correct addresse

Microsoft's description is not 100% exact, even if it seems so when reading the following statement.

To translate the *.servicebus.windows.net wildcard into specific endpoints, use the command \GET https://guestnotificationservice.azure.com/urls/allowlist?api-version=2020-01-01&location=<location>. Within this command, the region must be specified for the <location> placeholder.

This provides the following addresses:

If you use the following URL you will also get a usable result for the *.his.arc.azure.com addresses:

https://gbl.his.arc.azure.com/discovery?location=westeurope&api-version=1.1-preview

So the abbreviation for West Europe is "weu".

For the following addresses I could not collect any URLs.

# *.guestnotificationservice.azure.com
 # Test-NetConnection "weu.guestnotificationservice.azure.com" -Port 443
 # Test-NetConnection "westeurope.guestnotificationservice.azure.com" -Port 443
 # Test-NetConnection "weuguestnotificationservice.azure.com" -Port 443
 # Test-NetConnection "westeuropeguestnotificationservice.azure.com" -Port 443

# *.guestconfiguration.azure.com
# Test-NetConnection "weuguestconfiguration.azure.com" -Port 443
# Test-NetConnection "weu.guestconfiguration.azure.com" -Port 443
# Test-NetConnection "westeurope.guestconfiguration.azure.com" -Port 443
# Test-NetConnection "westeuropeguestconfiguration.azure.com" -Port 443

I included all these points in a script that handles them fully automatically:

Donwloading the JSON - File
Selecting the right Azure Region (extracted from the JSON - File)
Testing Connection
List of tested URLs

If there were any issues, they would show up at that point.

Modified Script - Connection.ps1:

$rawhtml = Invoke-RestMethod -Uri 'https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519' -UseBasicParsing
# Dowload https://www.microsoft.com/download/details.aspx?id=56519

# Source file location
$source = [Regex]::Match($rawhtml, 'https://download.microsoft.com[^"]*').Value
# Destination to save the file
$destination = 'c:\temp\ServiceTags_Public.json'
#Download the file
Invoke-WebRequest -Uri $source -OutFile $destination

$lookups='AzureArcInfrastructure','AzureAD','AzureTrafficManager','AzureResourceManager','AzureStorage','WindowsAdminCenter'

$ServiceTags = Get-Content -path c:\temp\ServiceTags_Public.json -Raw | ConvertFrom-Json

$region = ($ServiceTags.values | Select-Object -expand properties | Select-Object @{Label="Region";Expression={($_.region)}} -Unique | Sort-Object -Property Region | Out-GridView -OutputMode Single -Title 'Select an Azure Region').Region

$urls = @('aka.ms','download.microsoft.com','packages.microsoft.com','management.azure.com','login.windows.net','login.microsoftonline.com','pas.windows.net','guestnotificationservice.azure.com','dc.services.visualstudio.com','www.office.com','agentserviceapi.guestconfiguration.azure.com')

# $ServiceTags.values | Select-Object -expand properties | Select-Object region, systemService, addressPrefixes | Where-Object { $_.region -eq $region.Region -or $_.region -eq "" }
# ((Invoke-RestMethod -Uri "https://gbl.his.arc.azure.com/discovery?location=$region&api-version=1.1-preview").split('.')[0]).substring(8)
# (Invoke-RestMethod -Uri "https://gbl.his.arc.azure.com/discovery?location=$region&api-version=1.1-preview")

$urls += (Invoke-RestMethod -Uri ("https://gbl.his.arc.azure.com/discovery?location=$region&api-version=1.1-preview")).substring(8)
$urls += (Invoke-RestMethod "https://guestnotificationservice.azure.com/urls/allowlist?api-version=2020-01-01&location=$region")

foreach ($url in $urls) {
     if (((Test-NetConnection $url -Port 443 -WarningAction:SilentlyContinue).TCPTESTSucceeded) -ne $true) {Write-Host "$url is not reachable"}
     $url
}

Verify the connection using the agent:

azcmagent check --location westeurope

Result:

Agent - Connection Test

Check for Problems using the AMA - Troubleshooting - Tool

Open a PowerShell or Windows Terminal as an Administrator

Change directory to:

 cd "C:\Packages\Plugins\Microsoft.AzureMonitorWindowsAgent\<AMA Agent Version Number>\Troubleshooter"

Type AgentTroubleshooter.exe --ama and press the ENTER

Result:

If nothing helps and all the measures listed here do not provide any further information or progress, I recommend uninstalling the agent completely and then performing a clean reinstallation.

References:

Check if network connectivity for Azure Arc Connected Machine Agent is blocked
Is there a way to download Microsoft Azure datacenter IP ranges in Java?
My question is similar to Is there a way to automatically and programmatically download the latest IP ranges used by Microsoft Azure? except that I’m trying to use Java to do it. I’m proceeding to
Connected Machine agent network requirements - Azure Arc
Learn about the networking requirements for using the Connected Machine agent for Azure Arc-enabled servers.
Azure Arc and Defender for Endpoint Ports & URLs
During the onboarding / rollout of Defender for Endpoint and Azure Arc Agent, the network plays a significant role. Communication via the Internet is usually restricted by segmented networks and secured by firewalls and proxies. To prevent errors or communication problems, the required ports & URLs …
SOLVED: Missing Windows Server Agent Heartbeat – Check Azure Arc Status