Health Checks In DNS Steering: From Origin Status To Answer Policy

DNS steering should not return an endpoint only because it is nearby.

It should not return an endpoint only because a policy prefers it.

The endpoint also needs to be usable.

This is where health checks matter.

Health checks help a DNS steering system decide whether an endpoint should still be returned in DNS answers.

Example:

PH users prefer Manila.

If Manila is healthy:
Return Manila.

If Manila is unhealthy:
Return Singapore.

That sounds simple, but health checks can become difficult in real systems.

A server may respond to ping but fail HTTP requests.

A website may return HTTP 200 but still have a broken login flow.

A health check may fail because of a temporary network issue.

A health result may be old.

A check may work from one location but fail from another.

A good DNS steering system does not treat health checks as simple green or red labels.

It turns health information into answer policy.

That means the system decides when an endpoint is allowed, blocked, watched, or used only as fallback.

Why Health Checks Matter In DNS Steering

DNS steering influences which endpoint a requester receives.

If the DNS system returns an unhealthy endpoint, new users may fail before the application can help them.

Example:

app.example.com. 60 IN A 203.0.113.10

If 203.0.113.10 is down, the DNS answer sends users to a bad destination.

That is why health state should protect DNS answers.

A practical rule:

Do not return a known bad endpoint when a safer answer exists.

This rule is simple, but the work behind it needs care.

What Is A Health Check?

A health check is a test that asks:

Can this endpoint serve traffic right now?

The test can be simple or detailed.

Common health check types include:

ICMP ping
TCP port check
TLS handshake check
HTTP status check
Application health endpoint
DNS query check
Origin reachability check
Content availability check

Each check answers a different question.

A ping check can tell you whether a host may be reachable.

A TCP check can tell you whether a port accepts connections.

An HTTP check can tell you whether a web service returns an expected response.

An application health endpoint can tell you whether the service itself believes it is ready.

HTTP status code behavior is defined in HTTP standards such as RFC 9110.

Source:

https://www.rfc-editor.org/rfc/rfc9110

Health Checks Are Not All Equal

Different checks give different levels of confidence.

ICMP Ping

ICMP ping can tell whether a host responds to echo requests.

Example:

ping 203.0.113.10

This is useful for basic reachability.

But ping success does not mean the application is working.

A server may reply to ping while the web service is down.

TCP Port Check

A TCP check tests whether a port accepts connections.

Example:

Can 203.0.113.10 accept TCP connections on port 443?

This is stronger than ping for web services.

But it still does not prove the application is working correctly.

A port may be open while the application returns errors.

TCP behavior is defined in RFC 9293.

Source:

https://www.rfc-editor.org/rfc/rfc9293

TLS Check

A TLS check tests whether the secure connection can be established.

This can detect certificate or handshake problems.

Example:

Can the endpoint complete a TLS handshake for app.example.com?

This is useful for HTTPS services.

But a good TLS handshake still does not prove the application is healthy.

HTTP Status Check

An HTTP check requests a URL and checks the response.

Example:

GET https://app.example.com/health
Expected status: 200

This is often useful.

But it depends on what the health endpoint actually tests.

If /health always returns 200 OK, it may hide real application problems.

Application Health Check

An application health check can test deeper service readiness.

Example:

Can the app reach its database?
Can it read required configuration?
Can it access object storage?
Can it process a basic request?

This gives better confidence.

But deep checks can also become sensitive.

If the check depends on too many systems, small problems may mark the endpoint unhealthy too quickly.

Origin Health Versus Edge Health

In DNS steering, health may apply to different layers.

Origin Health

The origin is the source service.

Example:

Main application server
Primary API service
Video origin
Storage origin
Authentication service

If the origin is down, edge nodes or regional proxies may still be reachable, but they may fail when they need origin content.

Edge Health

An edge is a location closer to users.

Example:

Manila edge
Singapore edge
Tokyo edge

An edge may be reachable even if the origin is unhealthy.

Or the origin may be healthy while one edge has a local problem.

A DNS steering system should know which layer it is checking.

Example:

Manila edge:
Healthy

Origin:
Unhealthy

Answer policy:
Depends on whether Manila can serve cached or independent content.

For cached content, a healthy edge may still serve users even if the origin is temporarily unavailable.

For login or API traffic, origin failure may make the edge unusable.

Regional Health

Health can also be regional.

Example:

Manila region:
Healthy from Philippine probes.
Unhealthy from Singapore probe.

Singapore region:
Healthy from Singapore probe.
Unhealthy from Japan probe.

A single global health check may miss local network problems.

If users from one ISP cannot reach an endpoint, but other networks can, the endpoint may be partly healthy.

This leads to a harder question:

Should the endpoint be removed for everyone, or only for affected requesters?

A mature DNS steering design may use regional or ASN-specific health signals.

But that also adds complexity.

A simple system may start with global health only.

Then it can add more specific health signals when measurements show a real need.

Health Is Not Always Binary

Many systems use two states:

Healthy
Unhealthy

That is simple.

But real systems may need more states.

Example:

Healthy
Degraded
Unhealthy
Unknown
Stale
Maintenance
Draining

These states help answer policy make better decisions.

Healthy

The endpoint is safe to return.

Degraded

The endpoint works, but with problems.

The system may still return it if no better endpoint exists.

Unhealthy

The endpoint should not be returned when a safer answer exists.

Unknown

The system does not have enough information.

Unknown should not always mean unhealthy.

Stale

The last health result is too old.

Stale data needs a defined rule.

Maintenance

The endpoint is intentionally removed or reduced.

Draining

The endpoint should stop receiving new traffic, but existing traffic may still finish.

DNS steering can help shift new lookups away from a draining endpoint, but cached answers may still exist until TTL expires.

Why One Failed Check Should Not Always Remove An Endpoint

Networks have short failures.

A packet may be lost.

A probe server may have a problem.

A check may time out once.

If one failed check immediately removes an endpoint, the DNS answer may change too often.

This can create unstable routing.

A safer rule uses thresholds.

Example:

Mark unhealthy after 3 failed checks.
Mark healthy after 2 successful checks.

This reduces false alarms.

It also prevents endpoints from moving in and out of DNS answers too quickly.

This pattern is often called hysteresis in monitoring and failover design.

A simple explanation:

Do not change state too quickly.
Require enough proof before marking an endpoint good or bad.

Health Check Frequency

Health check frequency controls how often the system tests an endpoint.

Example:

Check every 10 seconds.
Check every 30 seconds.
Check every 60 seconds.

Faster checks can detect failure sooner.

But faster checks also create more traffic and more processing.

Slower checks reduce load.

But they may delay failover.

A practical design balances:

Failure detection speed
Probe traffic
Endpoint load
Control plane processing
Data plane update rate
False positive risk

There is no single best interval for every service.

A high-traffic API may need faster checks.

A low-risk static site may use slower checks.

Health Result Age

A health result should have an age.

Example:

Endpoint: Manila
Health: healthy
Checked at: 10:00:00
Current time: 10:07:00
Health age: 7 minutes

If the allowed age is 2 minutes, this result is stale.

Stale health data needs a rule.

Example:

If health is fresh, use it.
If health is stale, use last known safe state for a limited time.
If health is too old, use fallback or raise critical alert.

Do not treat old health data as fresh health data forever.

Health Checks And DNS TTL

TTL affects how fast DNS changes may reach users.

Example:

app.example.com. 300 IN A 203.0.113.10

A TTL of 300 seconds means a recursive resolver may cache the answer for 300 seconds.

If an endpoint fails right after the resolver caches the answer, some users may still receive or use the old answer until the cache expires.

That means health checks and DNS TTL must be designed together.

Example:

Health check interval:
30 seconds

Unhealthy threshold:
3 failures

Detection time:
About 90 seconds

DNS TTL:
300 seconds

Possible user impact:
Some resolvers may continue using the old answer for up to 300 seconds.

Lower TTL can help changes take effect faster.

But lower TTL can increase authoritative DNS query volume.

DNS TTL and caching behavior are part of standard DNS operation.

Sources:

https://www.rfc-editor.org/rfc/rfc1034
https://www.rfc-editor.org/rfc/rfc1035

From Health Check To Answer Policy

Raw health data is not enough.

The DNS system needs answer policy.

Raw data:

Manila HTTP check failed.
Singapore HTTP check passed.
Tokyo HTTP check passed.

Answer policy:

Do not return Manila for new DNS answers.
Return Singapore for PH users.
Keep Tokyo as global backup.

This is the important step.

A health check says what was observed.

An answer policy says what DNS should do.

Example: Basic Health-Based Failover

Assume this policy:

PH users prefer Manila.
Fallback is Singapore.

Health state:

Manila:
Unhealthy

Singapore:
Healthy

DNS answer:

app.example.com. 60 IN A 198.51.100.20

The system returns Singapore because Manila is not safe.

Example: Multi-Endpoint Pool

Assume Manila has three endpoints:

203.0.113.10
203.0.113.11
203.0.113.12

Health state:

203.0.113.10 healthy
203.0.113.11 unhealthy
203.0.113.12 healthy

DNS answer may include only healthy endpoints:

app.example.com. 60 IN A 203.0.113.10
app.example.com. 60 IN A 203.0.113.12

This prevents new lookups from receiving the known bad endpoint.

Example: Pool-Level Health

Sometimes an endpoint pool is considered healthy if enough members are healthy.

Example:

Manila pool:
3 total endpoints
2 healthy endpoints
Pool state: healthy

Policy:

Use Manila pool if at least 2 endpoints are healthy.
If fewer than 2 are healthy, use Singapore pool.

This prevents a weak pool from receiving too much traffic.

Example failed state:

Manila pool:
3 total endpoints
1 healthy endpoint
Pool state: degraded or unhealthy

Answer policy:

Return Singapore.

This kind of rule is useful when capacity matters.

One surviving endpoint may be technically healthy, but it may not handle all traffic alone.

Example: Maintenance Mode

Operators may intentionally remove an endpoint from DNS answers.

Example:

Endpoint:
203.0.113.10

State:
Maintenance

DNS policy:

Do not return this endpoint, even if health checks pass.

Maintenance state should override health.

This lets operators patch, reboot, or inspect a node without new DNS traffic being sent to it.

Example: Draining

Draining means an endpoint should stop receiving new traffic.

Example:

Endpoint:
203.0.113.10

State:
Draining

DNS policy:

Stop returning this endpoint in new DNS answers.
Keep service running for existing connections.

Because of DNS caching, some users may still reach the draining endpoint until TTL expires.

So draining should start before maintenance when possible.

Example:

Start drain.
Wait for TTL window.
Confirm traffic drops.
Begin maintenance.

Health Check Location Matters

Where the health check runs matters.

Example:

Probe from Singapore:
Manila endpoint looks healthy.

Probe from Manila ISP:
Manila endpoint looks unreachable.

Which result should the DNS system trust?

It depends on the service and policy.

If the problem affects only one ISP, removing the endpoint globally may be too aggressive.

A better policy may be:

For affected ASN:
Use Singapore fallback.

For other requesters:
Keep Manila.

This is more precise, but it requires better measurements and more careful policy.

Avoiding False Positives

A false positive happens when the system marks an endpoint unhealthy even though it can still serve users.

Causes may include:

Probe server failure
Temporary packet loss
Firewall blocking probes
Wrong health check path
Short timeout
DNS problem inside the probe system
Rate limiting on the endpoint

False positives can cause unnecessary failover.

That may increase latency or overload backup sites.

Ways to reduce false positives:

Use multiple checks before changing state.
Use probes from more than one location.
Set realistic timeouts.
Check the correct URL or service path.
Track probe failures separately.
Avoid using one probe as the only source of truth.

Avoiding False Negatives

A false negative happens when the system marks an endpoint healthy even though users are failing.

Causes may include:

Health endpoint is too shallow
Application path is broken
Database dependency is slow
Authentication service is failing
TLS works, but requests fail
Endpoint works from probes, but not from user networks

False negatives are dangerous because DNS keeps returning a bad endpoint.

Ways to reduce false negatives:

Check a meaningful service path.
Include key dependencies when needed.
Compare health checks with real user errors.
Use regional probes.
Watch application-level error rates.
Measure fallback success.

Health Checks Should Match Service Type

Different services need different health checks.

Web App

Useful checks:

HTTPS connection
HTTP 200 on /health
Basic application readiness
Database reachability, if required

API

Useful checks:

HTTPS connection
API readiness endpoint
Authentication dependency
Database or queue dependency

Video Edge

Useful checks:

HTTP response
Playlist availability
Segment availability
Cache service status
Origin reachability, depending on design

Object Storage Or Download Service

Useful checks:

HEAD request to known object
GET small test object
Storage backend reachability
TLS certificate validity

The health check should test what matters for users.

Health State Should Be Versioned

When health state affects DNS answers, operators need to know which health version was used.

Useful fields:

Endpoint ID
Endpoint state
Last check time
Last success time
Last failure time
Failure count
Success count
Health source
Policy version
Runtime snapshot version

These fields help explain DNS answers.

Example question:

Why did PH users receive Singapore at 10:05 AM?

Possible answer:

At 10:05 AM, Manila failed 3 checks.
Policy version 42 moved PH users to Singapore.
Data plane used snapshot 2026-07-03-1005.

Without versioning and logs, this is hard to explain.

Data Plane Should Not Run Heavy Health Checks Per Query

The authoritative DNS data plane should answer quickly.

It should not perform a full health check during each DNS query.

Bad design:

DNS query arrives.
Data plane calls HTTP health endpoint.
Data plane waits.
Data plane chooses answer.

This makes DNS answers slower and more fragile.

Better design:

Health system checks endpoints separately.
Control plane processes health state.
Runtime snapshot is updated.
Data plane reads prepared health state.
Data plane answers quickly.

The data plane should apply prepared health state, not run heavy checks during live query handling.

Health Checks And Runtime Snapshots

A runtime snapshot may contain prepared health state.

Example:

Endpoint:
203.0.113.10

State:
Healthy

Valid until:
10:05:00

Pool:
Manila

Allowed in answers:
Yes

When health changes, the control plane can publish an updated snapshot.

Example:

Endpoint:
203.0.113.10

State:
Unhealthy

Allowed in answers:
No

Fallback:
Singapore pool

The data plane loads the updated snapshot and changes future answers.

This keeps DNS answering fast.

What Happens When Health Data Is Missing?

Missing health data must have a rule.

Example:

Endpoint exists.
No health result is available.

Possible policies:

Treat as unknown.
Treat as unhealthy until proven healthy.
Use last known state for a limited time.
Allow only as fallback.
Remove from answer set.

The correct choice depends on the service.

For critical traffic, it may be safer to require fresh health.

For low-risk traffic, bounded last known state may be acceptable.

The key point:

Missing health must not produce random behavior.

Stale Health And Serving Stale Ideas

Some DNS systems use stale data in controlled ways to improve resilience.

RFC 8767 discusses serving stale DNS data when authoritative answers are unavailable.

Source:

https://www.rfc-editor.org/rfc/rfc8767

For DNS steering, a related idea is:

A bounded old safe answer may be better than no answer.

But stale data should have limits.

Example:

Use last known safe health for 5 minutes.
After 5 minutes, raise critical alert.
After 10 minutes, use emergency policy.

Stale data should be visible and time-limited.

Health And Fallback Policy

Health checks and fallback logic work together.

Example:

Preferred endpoint:
Manila

Health:
Unhealthy

Fallback:
Singapore

Answer:

Return Singapore.

Another example:

Preferred endpoint:
Manila

Health:
Unknown

Fallback rule:
Use Manila only if fresh healthy state exists.

Answer:
Return Singapore.

Another possible rule:

Preferred endpoint:
Manila

Health:
Unknown

Fallback rule:
Use last known healthy state for up to 2 minutes.

Answer:
Return Manila if within allowed age.

The right rule depends on risk.

But every rule should be defined before failure happens.

Health And Weighted DNS Answers

Some DNS steering systems use weights.

Example:

Manila endpoint A: weight 50
Manila endpoint B: weight 50

If one endpoint becomes unhealthy, it should be removed from the weighted set.

Before failure:

A: 50
B: 50

After B fails:

A: 100
B: 0

But the system must also consider capacity.

If A cannot handle all traffic alone, the safer policy may be:

A: partial traffic
Singapore backup: remaining traffic

Health policy should consider both availability and capacity.

Health And Capacity

An endpoint can be healthy but overloaded.

Example:

HTTP health check:
200 OK

CPU:
95 percent

Network:
Near limit

User errors:
Increasing

Should DNS continue sending traffic?

Maybe not.

A simple health check may say healthy.

A better policy may include capacity signals.

Example:

If endpoint is healthy but overloaded, reduce its DNS answer weight.
If endpoint is critically overloaded, remove it from new DNS answers.

This is more advanced, and it requires reliable metrics.

For many systems, basic health comes first.

Capacity-aware answer policy can come later.

Health And Brownouts

A brownout is a partial failure.

The endpoint is not fully down, but service quality is poor.

Example:

Some requests work.
Some requests timeout.
Latency is high.
Error rate is rising.
Health check still passes.

Brownouts are harder than full outages.

A DNS steering system can help only if it receives useful signals.

Examples:

High error rate
High latency
Failed application checks
Regional complaints
Synthetic probe failures
Real user monitoring, if available

For brownouts, health policy may need a degraded state.

Example:

Endpoint is degraded.
Keep it for local users.
Do not send extra fallback traffic to it.
Prefer backup for new traffic from affected networks.

Health And Maintenance Windows

Maintenance should be explicit.

Do not rely only on failed health checks to remove an endpoint.

Better flow:

Set endpoint to draining.
Wait for TTL window.
Set endpoint to maintenance.
Perform work.
Run health checks.
Return endpoint to service.
Monitor traffic.

This makes maintenance safer.

It also avoids waiting for health checks to fail before traffic shifts.

Health And DNS Response Codes

Health failure usually should not cause NXDOMAIN.

NXDOMAIN means the name does not exist.

If the service name exists but one endpoint is unhealthy, return a fallback answer instead.

Example:

Bad:
Endpoint unhealthy, return NXDOMAIN.

Better:
Endpoint unhealthy, return backup endpoint.

Negative DNS caching is described in RFC 2308.

Source:

https://www.rfc-editor.org/rfc/rfc2308

SERVFAIL should also be used carefully.

SERVFAIL means the server failed to complete the DNS request.

For normal endpoint health failure, a fallback answer is usually better than SERVFAIL.

Use SERVFAIL only when no safe answer exists and the server cannot answer correctly.

DNS response behavior is described in RFC 1035.

Source:

https://www.rfc-editor.org/rfc/rfc1035

Logs For Health-Based DNS Decisions

Health-based DNS answers should be explainable.

Useful log fields:

Requested name
Record type
Resolver IP
ECS prefix, if present
Classified country
Classified ASN
Matched policy
Preferred endpoint
Preferred health state
Fallback endpoint
Fallback reason
Returned answer
TTL
Health data age
Policy version
Snapshot version

These logs help answer:

Why did this query avoid Manila?
Was Manila unhealthy or in maintenance?
Was health data stale?
Which fallback was used?
Was the answer based on fresh data?

Metrics For Health-Based DNS Steering

Operators should measure health and answer behavior together.

Useful metrics:

Endpoint health state
Pool health state
Health check success rate
Health check failure rate
Health data age
Endpoint removal count
Endpoint restore count
Fallback count
Fallback rate by domain
Fallback rate by country
Fallback rate by ASN
Answer distribution by endpoint
SERVFAIL count
NXDOMAIN count
Stale health usage
Maintenance state duration

These metrics show whether health checks are protecting users or causing avoidable routing changes.

Alerting On Health Policy Changes

Health changes should trigger alerts when they affect DNS answers.

Example alerts:

Endpoint removed from DNS answers.
Pool became unhealthy.
Fallback traffic increased.
Health data is stale.
Last known good state is being used.
Emergency answer is active.
Endpoint restored to service.

Alerts should focus on user impact and answer policy.

A single failed probe may not need a critical alert.

A pool removed from DNS answers probably does.

Testing Health-Based DNS Behavior

Health logic should be tested.

Test cases:

Endpoint healthy
Endpoint unhealthy
Endpoint degraded
Endpoint in maintenance
Endpoint draining
Health data missing
Health data stale
Pool partly healthy
Pool fully unhealthy
Fallback endpoint healthy
Fallback endpoint unhealthy
All endpoints unhealthy
Bad health check path
Health check timeout
False positive probe failure

Each test should answer:

What DNS answer will be returned?
What TTL will be used?
What log entry will be created?
What metric will increase?
What alert will fire?

If the team cannot answer those questions, the health policy needs more work.

Common Health Check Mistakes

Mistake 1: Using Ping As Full Health

Ping only proves basic reachability.

It does not prove the application works.

Mistake 2: Checking A Weak Health Page

A health page that always returns 200 OK may hide real problems.

Mistake 3: Removing Endpoints Too Quickly

One failed check should not always remove an endpoint.

Use thresholds.

Mistake 4: Restoring Endpoints Too Quickly

One successful check should not always restore an endpoint.

Require enough successful checks.

Mistake 5: Ignoring Health Data Age

Old health data should not be treated as fresh forever.

Mistake 6: No Maintenance State

Maintenance should be explicit.

Do not wait for checks to fail.

Mistake 7: No Fallback Test

If fallback is not tested, operators may only learn its behavior during an outage.

Mistake 8: No Logs For Health Decisions

If health decisions are not logged, troubleshooting becomes slow.

Practical Health Policy Rules

A practical DNS steering system should follow these rules:

Use health checks that match the service type.
Use thresholds before changing endpoint state.
Track health data age.
Treat maintenance as a separate state.
Keep stale state bounded.
Do not return known bad endpoints when safer answers exist.
Use fallback instead of NXDOMAIN for endpoint failure.
Use SERVFAIL only when no safe answer exists.
Log health-based decisions.
Measure fallback and health state changes.
Test health failure scenarios before production.

These rules keep health checks useful and safer.

The Main Design Principle

The main design principle is this:

Health checks should not only detect failure. They should drive clear DNS answer policy.

Detection alone is not enough.

The system must know what to do with the result.

If Manila fails, should the system return Singapore?

If one endpoint in a pool fails, should the system remove only that endpoint?

If health is stale, should the system use last known state or move to backup?

If all endpoints are unhealthy, should the system use an emergency answer or return controlled failure?

These decisions should be designed before failure happens.

How This Fits The Article Series

The earlier articles built the foundation:

Article 1:
How authoritative DNS influences traffic direction.

Article 2:
The control plane and data plane of DNS steering.

Article 3:
Why DNS steering is not load balancing.

Article 4:
Why latency, geography, and network distance are different.

Article 5:
How requester classification works.

Article 6:
How EDNS Client Subnet helps and complicates classification.

Article 7:
Why fallback logic is required for reliable DNS steering.

This article explains how health status affects DNS answers.

Health checks tell the system which endpoints appear usable.

Answer policy tells the DNS system what to return.

Both are needed.

Key Takeaways

Health checks are essential in DNS steering because DNS should avoid returning known bad endpoints.

Different checks give different confidence levels.

Ping, TCP, TLS, HTTP, and application health checks do not prove the same thing.

Health can apply to origins, edges, pools, regions, and specific endpoints.

Health state should not always be only healthy or unhealthy.

Useful states may include healthy, degraded, unhealthy, unknown, stale, maintenance, and draining.

One failed check should not always remove an endpoint.

One successful check should not always restore an endpoint.

Health checks, TTL, fallback logic, and runtime snapshots must work together.

The data plane should use prepared health state, not run heavy checks during each DNS query.

Health decisions should be logged, measured, tested, and tied to clear answer policy.

The best question is not only:

Is this endpoint up?

The better question is:

Should this endpoint still be returned in DNS answers?

In the next article, we will discuss Building A Runtime Snapshot For Fast Authoritative DNS Decisions.

Sources

RFC 1034, Domain Names, Concepts and Facilities:
https://www.rfc-editor.org/rfc/rfc1034

RFC 1035, Domain Names, Implementation and Specification:
https://www.rfc-editor.org/rfc/rfc1035

RFC 2308, Negative Caching of DNS Queries:
https://www.rfc-editor.org/rfc/rfc2308

RFC 8767, Serving Stale Data to Improve DNS Resiliency:
https://www.rfc-editor.org/rfc/rfc8767

RFC 9110, HTTP Semantics:
https://www.rfc-editor.org/rfc/rfc9110

RFC 9293, Transmission Control Protocol:
https://www.rfc-editor.org/rfc/rfc9293