Troubleshooting Load Balancer Health Check Failures

1. Overview

This guide explains how to identify and resolve health check failures when using a Cloud Load Balancer.

2. Operating System Checks

2.1 Verify That the Application or Service Is Running

Linux

Check the service status:

systemctl status <service-name>

Restart the service if required:

systemctl restart <service-name>

Verify that the application is listening on the required port:

ss -tulnp | grep <port>

Windows

Open Services (services.msc) and verify that the required service is running.

Check the listening port:

netstat -ano | findstr <port>

Find the process using the PID:

tasklist /FI “PID eq <pid>”

2.2 Test the Health Check Endpoint Locally

Linux

curl -I http://localhost:<port>/<health-path>

or

curl http://127.0.0.1:<port>/<health-path>

Windows (PowerShell)

Invoke-WebRequest -Uri http://localhost:<port>/<health-path> -UseBasicParsing

Note: For HTTP or HTTPS health checks, the endpoint should return an HTTP status code between 200 and 299.

2.3 Check the Operating System Firewall

Linux

For systems using firewalld:

sudo firewall-cmd –list-all

For Ubuntu using UFW:

sudo ufw status

For systems using iptables:

sudo iptables -L -n

Windows

Get-NetFirewallRule | Where-Object { $_.Direction -eq ‘Inbound’ -and $_.Enabled -eq ‘True’ } | Format-Table DisplayName,Profile,Action

2.4 Verify That the Application Is Listening on the Private Interface

If the application is listening only on 127.0.0.1, the Load Balancer cannot reach it.

The application should listen on 0.0.0.0 or the VM’s private IP address.

Linux

ss -tulnp | grep <port>

or

netstat -tulnp | grep <port>

2.5 Test Connectivity from Another Internal Virtual Machine

Test the backend service from another VM on the same network.

curl http://<backend-private-IP>:<port>/<health-path>

or

telnet <backend-private-IP> <port>

or

nc -vz <backend-private-IP> <port>

Note: If direct connectivity fails, resolve the backend application, firewall, or operating system issue before troubleshooting the Load Balancer.

2.6 Check Application Logs

Linux

View the service logs:

journalctl -u <service-name> -n 200

View system logs:

tail -n 200 /var/log/messages

View web server logs:

tail -n 200 /var/log/nginx/error.log

or

tail -n 200 /var/log/httpd/error_log

Windows

• Open Event Viewer and review the Application and System logs.
• For IIS, review the logs located at:

C:\inetpub\logs\LogFiles

2.7 Check System Resources

Linux

top
free -m
df -h

Windows

Use Task Manager to check:

• CPU usage
• Memory usage
• Disk usage

3. Load Balancer Checks

3.1 Verify the Health Check Configuration

Confirm that the health check settings match the backend application.

Verify the following:

• Protocol: TCP, HTTP, or HTTPS
• Port: Matches the backend service port
• Health Check Path (HTTP/HTTPS): Returns an HTTP status code between 200 and 299

Recommended settings:

3.2 Verify Backend Pool Configuration

Confirm the following:

• The correct backend Virtual Machine is added to the backend pool.
• The correct private IP address is selected, especially if the VM has multiple network interfaces.
• The backend service port is configured correctly.

3.3 Verify Firewall and Security Rules

Ensure that the backend firewall or security rules allow inbound traffic from the Load Balancer’s private IP range on the configured health check port.

If the Load Balancer cannot reach the backend because of firewall restrictions, the health checks will fail.