Keeping DNS Answers Fast While Updating Routing Policy

DNS steering systems change over time.

Operators add new endpoints.

Health state changes.

A new region goes online.

A failed site needs to be removed.

A routing rule is updated.

A fallback path is changed.

An ASN is moved from one endpoint group to another.

These changes are normal.

But there is one important rule:

Routing policy updates should not make DNS answers slow.

Authoritative DNS must keep answering quickly while policy changes are prepared, validated, published, and loaded.

A DNS steering system should not pause live DNS answers while it rebuilds policy.

It should not make every DNS query wait for database updates.

It should not expose users to half-applied routing state.

The goal is simple:

Update routing policy safely while the authoritative DNS data plane keeps answering fast.

Why Policy Updates Are Risky

A routing policy controls what DNS answers are returned.

That means a bad update can affect real users.

Example policy change:

Before:
PH users use Manila.

After:
PH users use Singapore.

This change may be intentional.

But if it is wrong, Philippine users may be sent to a slower endpoint.

Another example:

Before:
Manila fallback is Singapore.

After:
Manila fallback is removed by mistake.

If Manila fails, the system may have no safe backup.

Another example:

Before:
video.example.com has A records.

After:
video.example.com has no valid endpoint because of a bad policy reference.

This can break service resolution.

DNS policy updates are powerful.

They need validation, versioning, and safe publishing.

DNS Answers Should Not Wait For Policy Editing

Policy editing belongs in the control plane.

Live DNS answering belongs in the data plane.

The data plane should not wait for an operator to finish editing a rule.

It should not use draft policy.

It should not read unfinished database changes during a query.

A safe design separates the two:

Control plane:
Edit, validate, build, test, and publish policy.

Data plane:
Answer DNS queries using the active prepared version.

This keeps DNS answers fast while policy work happens elsewhere.

The Active Version Concept

A DNS steering system should have an active policy version.

Example:

Active policy version:
42

The data plane answers queries using version 42.

Meanwhile, an operator may prepare version 43.

Example:

Draft policy version:
43

The draft version should not affect live DNS answers until it is validated and published.

This gives operators a safe workflow:

Edit draft.
Validate draft.
Build snapshot.
Test snapshot.
Publish new version.
Confirm data plane loaded it.
Monitor answers.
Rollback if needed.

Why Runtime Snapshots Help

In the previous article, we discussed runtime snapshots.

A runtime snapshot is a prepared, validated, versioned copy of the data needed for live DNS decisions.

It may include:

Active zones
Records
Routing policies
Endpoint pools
Health state
Fallback rules
GeoIP and ASN indexes
TTL values
Default answers
Version metadata

The snapshot lets the data plane answer quickly.

Policy updates should create a new snapshot.

The data plane should continue using the current snapshot until the new one is ready.

This avoids query-time rebuilding.

Bad design:

DNS query arrives.
Policy is being updated.
Data plane waits for database.
Data plane reads partial state.
Answer becomes slow or wrong.

Better design:

DNS query arrives.
Data plane uses active snapshot.
Control plane builds new snapshot in the background of the request path.
When ready, data plane switches to the new snapshot.

The user-facing DNS query stays fast.

Update Flow Overview

A safe routing policy update flow may look like this:

1. Operator creates a policy change.
2. Control plane stores it as draft.
3. Control plane validates the draft.
4. Control plane builds a new runtime snapshot.
5. Snapshot is tested.
6. Snapshot is published.
7. Data plane loads the snapshot.
8. Data plane switches active version.
9. System monitors answer behavior.
10. Operators rollback if needed.

Each step has a purpose.

The update should not reach live DNS answers until it passes the required checks.

Step 1: Create A Draft Policy

Policy changes should start as drafts.

Example:

Draft change:
Move PH ASN 64502 from Manila to Singapore.

The draft should not affect live answers yet.

The live data plane continues using the active version.

Example:

Active version:
42

Draft version:
43

Data plane uses:
42

This avoids accidental live impact while the policy is being edited.

Step 2: Validate The Policy

Validation catches problems before publishing.

Examples of validation checks:

Does the domain exist?
Is the record type valid?
Does the target pool exist?
Does the pool contain valid endpoints?
Is the fallback path valid?
Does the fallback chain loop?
Is the TTL valid?
Does every policy have a default?
Are IP addresses valid?
Are ECS prefix rules valid?
Are ASN values valid?

A policy that fails validation should not be published.

Example:

Policy:
PH users use Manila pool.

Problem:
Manila pool does not exist.

Result:
Reject policy.
Keep active version unchanged.

The data plane should never need to discover this during live DNS queries.

Step 3: Build A New Runtime Snapshot

After validation, the control plane builds a new runtime snapshot.

Example:

Snapshot version:
43

Includes:
New routing rule
Current endpoint pools
Current health state
Fallback rules
TTL values
Classification indexes

The current data plane still uses the old version while the new snapshot is being built.

Example:

Data plane active snapshot:
42

Control plane building:
43

This allows DNS answers to continue without waiting.

Step 4: Test The Snapshot

Before publishing, the new snapshot should be tested.

Test inputs may include:

Known PH requester
Known SG requester
Unknown country
Known ASN
Unknown ASN
ECS present
ECS missing
Preferred endpoint healthy
Preferred endpoint unhealthy
Fallback path used
Default policy used
IPv4 answer
IPv6 answer

Each test should confirm:

Returned answer
TTL
Matched rule
Selected pool
Fallback reason
Health state used
Snapshot version

Testing helps catch unexpected answer changes.

Step 5: Compare Old And New Behavior

A policy update may be valid but still risky.

Example:

Old behavior:
PH users from ASN 64502 receive Manila.

New behavior:
PH users from ASN 64502 receive Singapore.

That may be expected.

But if the new snapshot changes too many answers, operators should review it.

Useful comparison checks:

Which countries changed answers?
Which ASNs changed answers?
Which domains changed answers?
Which pools gained traffic?
Which pools lost traffic?
Which fallbacks changed?
Which TTL values changed?

This gives operators a chance to catch mistakes before publishing.

Step 6: Publish The Snapshot

Publishing should be controlled.

The control plane marks the new snapshot as ready.

The data plane is notified or pulls the new snapshot.

Example:

Ready snapshot:
43

Current active snapshot:
42

The data plane loads version 43 separately.

It should not replace the active version until loading succeeds.

Step 7: Switch Active Version Cleanly

The data plane should switch from old to new state cleanly.

This is often called atomic switching.

The idea is simple:

Use snapshot 42.
Load snapshot 43 separately.
If 43 loads correctly, switch active pointer to 43.
Keep 42 available for rollback.

Bad behavior:

Some data from 42.
Some data from 43.
Some pools missing.
Some health state incomplete.

This can cause wrong answers.

Better behavior:

At one moment, active version is 42.
After switch, active version is 43.
No half-loaded state is used.

This protects DNS answer quality.

Step 8: Confirm Data Plane State

After publishing, operators need to know which DNS nodes loaded the new version.

Example:

DNS node 1:
Snapshot 43

DNS node 2:
Snapshot 43

DNS node 3:
Snapshot 42

In this case, one node did not update.

That may cause inconsistent DNS answers.

Useful metrics:

Active snapshot version per node
Last load time
Snapshot load success
Snapshot load failure
Snapshot age
Policy version
Health version

The system should expose active version status.

Step 9: Monitor Answer Behavior

After publishing, operators should monitor DNS behavior.

Important metrics:

Query volume
Response time
Answer distribution
Fallback rate
SERVFAIL count
NXDOMAIN count
Policy match rate
Default policy usage
Endpoint answer count
Country and ASN answer changes

A bad update may show up as:

Fallback rate suddenly increases.
One endpoint receives too much traffic.
A country receives an unexpected answer.
SERVFAIL count increases.
Default policy usage increases.
Response time increases.

Monitoring tells operators whether the new policy behaves as expected.

Step 10: Rollback If Needed

Rollback means switching back to a previous known good version.

Example:

Current active snapshot:
43

Previous known good snapshot:
42

Rollback:
Switch active snapshot to 42.

Rollback should be fast and logged.

A rollback should not require manually rebuilding every policy.

It should use a previously validated version.

This is important during incidents.

When users are affected, operators need a direct path back to known good behavior.

Keeping DNS Fast During Updates

The most important performance rule is this:

Do not put policy rebuild work in the live DNS query path.

Live DNS query handling should remain simple.

Example data plane flow:

Receive query.
Read active snapshot.
Classify requester.
Match policy.
Check prepared health state.
Return answer.

Policy updates should happen outside this path.

Example control plane flow:

Edit policy.
Validate.
Build new snapshot.
Test.
Publish.

This separation keeps DNS response time stable.

Avoiding Database Reads Per Query

A common mistake is making each DNS query read directly from the database.

Bad design:

Every query asks the database:
What policy applies?
What endpoint is healthy?
What fallback should be used?

This creates risk.

If the database slows down, DNS slows down.

If the database is unavailable, DNS may fail.

If a policy is being updated, the data plane may read inconsistent state.

Better design:

Database stores source data.
Control plane builds snapshot.
Data plane reads active snapshot.

The database remains important, but it is not part of every DNS answer.

Handling Frequent Health Updates

Health state may change more often than routing policy.

Example:

Policy updates:
A few times per day.

Health updates:
Every 10 to 60 seconds.

The system must decide how health changes reach the data plane.

Possible approaches:

Build a full snapshot after each health change.
Build a smaller health overlay.
Publish health state separately but versioned.
Use bounded update frequency.

The best choice depends on system size and traffic volume.

The key rule stays the same:

The data plane should use prepared health state, not run heavy checks per query.

Handling High Update Rate Safely

Too many updates can create instability.

Example:

Endpoint fails one check.
Endpoint is removed.
Endpoint passes one check.
Endpoint is restored.
Endpoint fails again.
Policy changes again.

This can cause answer flapping.

Answer flapping means DNS answers change too often.

To reduce this risk, use:

Health thresholds
Minimum state duration
Rate limits on publishing
Change batching
Cooldown periods
Clear degraded states

Example:

Mark unhealthy only after 3 failed checks.
Mark healthy only after 2 successful checks.
Do not publish more than once every 30 seconds unless critical.

This keeps updates controlled.

TTL And Policy Updates

TTL affects how quickly users may see new DNS answers.

Example:

app.example.com. 300 IN A 203.0.113.10

A recursive resolver may cache this answer for 300 seconds.

If a policy update happens 10 seconds later, some resolvers may still use the old answer until the TTL expires.

This means:

Publishing a new DNS policy does not instantly change all user traffic.

A lower TTL can help changes take effect faster.

But lower TTL increases DNS query volume.

DNS TTL and caching behavior are part of standard DNS operation.

Sources:

https://www.rfc-editor.org/rfc/rfc1034
https://www.rfc-editor.org/rfc/rfc1035

Planned Policy Updates And TTL

For planned changes, TTL can be managed ahead of time.

Example:

Current TTL:
300 seconds

Planned change:
Move traffic at 10:00 AM

Preparation:
Lower TTL before the change.
Wait for old TTL window.
Publish new policy.
Monitor results.
Raise TTL after stable behavior.

This can reduce the time old answers remain cached.

But it must be done carefully.

Lower TTL increases query volume to authoritative DNS.

Operators should confirm the DNS system can handle the extra load.

Emergency Policy Updates

Emergency updates are different.

Example:

Manila endpoint is down.
Traffic must move to Singapore.

In an emergency, the system may publish an updated snapshot quickly.

But old answers may still be cached.

This is why DNS steering should work with other layers.

Example:

DNS steering moves new lookups.
Load balancers stop sending traffic to failed backends.
Applications retry when possible.
Monitoring confirms user impact.

DNS helps, but it does not instantly move every active user.

Keeping Partial Updates Away From The Data Plane

Partial updates are dangerous.

Example:

Step 1:
Create new Singapore pool.

Step 2:
Move PH fallback to Singapore pool.

Problem:
Data plane sees step 2 before step 1.
Fallback points to a pool that does not exist.

A runtime snapshot avoids this.

The control plane builds one complete version.

The data plane receives the complete version only after validation.

A policy update should be treated as a complete unit.

Policy Dependencies

Routing policy may depend on several objects.

Example:

Domain
Record type
Policy rule
Country mapping
ASN mapping
Pool
Endpoint
Health state
Fallback path
TTL

If one dependency is missing, the policy may fail.

The control plane should check dependencies before publishing.

Example validation:

Policy uses Manila pool.
Manila pool exists.
Manila pool has endpoints.
Endpoints have valid IPs.
Fallback exists.
TTL is valid.
Health state is available or fallback rule exists.

A policy should not become active until dependencies are complete.

Safe Default Behavior During Updates

The system should define what happens when an update fails.

Example:

New snapshot build failed.
Action:
Keep old active snapshot.
Raise alert.
Do not publish partial state.

Another example:

New snapshot loaded on one node but failed on another.
Action:
Keep old version on failed node.
Raise alert.
Monitor version mismatch.

Another example:

New policy has no default answer.
Action:
Reject policy.
Keep old active version.

The safest update is often the one that does not publish.

Handling Version Mismatch Across Nodes

In multi-node DNS systems, some nodes may update before others.

This can happen because of:

Network delay
Node restart
Failed snapshot load
File transfer issue
Local validation failure

Temporary version mismatch may be acceptable during a short rollout.

But long mismatch should alert operators.

Example:

Expected version:
43

Node 1:
43

Node 2:
43

Node 3:
42 for more than 5 minutes

Action:
Alert operator.

DNS nodes should report their active version.

Canary Publishing

For high-risk updates, operators may publish to a small set of nodes first.

Example:

Publish snapshot 43 to one DNS node.
Monitor answer behavior.
If healthy, publish to remaining nodes.
If bad, rollback the canary node.

This can reduce blast radius.

But canary publishing must be used carefully.

If only one authoritative node has the new policy, DNS answers may differ depending on which node receives the query.

That may be acceptable for testing, but operators should understand it.

Policy Simulation

Policy simulation means testing how a policy would answer before publishing it.

Example test questions:

What answer will PH users receive?
What answer will PH ASN 64502 receive?
What answer will unknown countries receive?
What happens if Manila is unhealthy?
What happens if ECS is missing?
What happens for IPv6 queries?

Simulation can catch mistakes early.

It also helps operators understand the effect of a policy before it becomes active.

Dry Run Publishing

A dry run builds and validates a snapshot without activating it.

Example:

Build snapshot 43.
Run validation.
Run test queries.
Compare with snapshot 42.
Do not publish yet.

Dry runs are useful for risky changes.

They show whether the update is safe to publish.

Audit Trail

Routing policy updates should be logged.

Useful audit fields:

Who made the change
What changed
When it changed
Why it changed
Old value
New value
Validation result
Published version
Rollback version, if any

Audit logs help during incidents.

They also help teams understand why a routing behavior changed.

Access Control

Not everyone should be able to publish DNS steering policy.

A policy update can affect production traffic.

Access control should separate actions such as:

View policy
Create draft
Edit draft
Validate policy
Approve policy
Publish policy
Rollback policy
Manage emergency mode

This reduces accidental changes.

For critical systems, approval workflow may be required before publishing.

Update Windows

Some changes can be scheduled during lower traffic periods.

Example:

Add new endpoint:
Low risk, can publish anytime after validation.

Move a major ASN:
Higher risk, better during watched window.

Change global default:
High risk, needs approval and monitoring.

Not every update needs the same process.

The process should match the risk.

Emergency Mode

Emergency mode is a special policy state used during incidents.

Example:

Return only global backup endpoint.
Ignore normal regional routing.
Use short TTL.
Log emergency answer reason.
Alert operators.

Emergency mode should be simple and tested.

It should not depend on complex policy logic.

It is the final safe path when normal routing policy cannot be trusted.

Avoiding Slow Reloads

A large snapshot may take time to load.

If reload blocks DNS answering, users may feel it.

A better design loads the new snapshot separately while the old snapshot continues serving.

Example:

Old snapshot serves queries.
New snapshot loads in separate memory.
New snapshot passes local validation.
Active pointer switches to new snapshot.
Old snapshot remains available for rollback.

DNS answering should continue during the load.

Monitoring Update Performance

Policy updates have their own performance signals.

Useful metrics:

Policy validation time
Snapshot build time
Snapshot test time
Snapshot publish time
Snapshot load time per node
Version propagation delay
Rollback time
Snapshot size
Memory used by snapshot
Query response time during update

These metrics show whether updates are becoming too slow or too risky.

If snapshot build time grows from seconds to minutes, the system may need review.

Logs During Updates

During updates, logs should clearly show state changes.

Useful update logs:

Draft created
Validation started
Validation failed
Validation passed
Snapshot build started
Snapshot build failed
Snapshot build passed
Snapshot published
Node loaded snapshot
Node rejected snapshot
Active version changed
Rollback started
Rollback completed

These logs help operators trace update flow.

What To Do When Publish Fails

Publish failure should not break DNS.

Example:

Snapshot 43 publish failed.

Action:
Keep snapshot 42 active.
Raise alert.
Record failure reason.
Allow retry after fix.

The data plane should continue answering from the last known good version.

A failed update should not become a live outage by itself.

What To Do When Rollback Fails

Rollback failure is serious.

A safe system should prepare for it.

Possible safeguards:

Keep multiple previous snapshots.
Store emergency static answer.
Allow manual emergency mode.
Validate rollback snapshot before switch.
Monitor active version after rollback.

Rollback should be tested.

A rollback process that has never been tested may fail during a real incident.

Update Testing Checklist

Before publishing a policy update, test these cases:

Known country match
Unknown country fallback
Known ASN match
Unknown ASN fallback
ECS present
ECS missing
IPv4 answer
IPv6 answer
Preferred endpoint healthy
Preferred endpoint unhealthy
Fallback endpoint healthy
Fallback endpoint unhealthy
Default answer
Emergency answer
TTL value
Policy version
Snapshot version

The goal is to know what the DNS system will return before users depend on it.

Common Mistakes During Policy Updates

Mistake 1: Updating Live State Directly

Editing live state directly increases risk.

Use draft, validate, publish.

Mistake 2: No Versioning

Without versions, operators cannot easily know which policy caused an answer.

Mistake 3: No Snapshot Validation

Invalid policy can become live.

Mistake 4: Half-Loaded State

Partial updates can create wrong answers.

Use complete snapshots and clean switching.

Mistake 5: No Rollback

A bad policy needs a fast path back to known good behavior.

Mistake 6: Ignoring TTL

Policy changes do not affect all users instantly because resolvers may cache old answers.

Mistake 7: No Monitoring After Publish

A policy may pass tests but still behave poorly in production.

Mistake 8: Too Many Updates Too Fast

Frequent updates can cause unstable answer behavior.

Use batching, thresholds, and clear publish rules.

Practical Design Rules

A practical DNS steering update system should follow these rules:

Keep policy editing out of the live query path.
Use drafts for changes.
Validate before publishing.
Build complete runtime snapshots.
Test snapshots before activation.
Switch active versions cleanly.
Keep previous known good versions.
Track active version per DNS node.
Monitor answer behavior after publishing.
Use rollback when needed.
Respect TTL behavior.
Log every publish and rollback.

These rules keep DNS answers fast and policy changes safer.

The Main Design Principle

The main design principle is this:

A DNS steering system should update policy by replacing prepared versions, not by making live queries wait for changing state.

Prepared versions protect the data plane.

They also make updates easier to test, monitor, explain, and roll back.

How This Fits The Article Series

The earlier articles built the foundation:

Article 1:
How authoritative DNS influences traffic direction.

Article 2:
The control plane and data plane of DNS steering.

Article 3:
Why DNS steering is not load balancing.

Article 4:
Why latency, geography, and network distance are different.

Article 5:
How requester classification works.

Article 6:
How EDNS Client Subnet helps and complicates classification.

Article 7:
Why fallback logic is required.

Article 8:
How health checks become DNS answer policy.

Article 9:
How runtime snapshots help authoritative DNS answer quickly.

This article explains how policy changes should be handled while DNS answers remain fast.

Policy updates are normal.

Slow DNS answers should not be.

Key Takeaways

DNS routing policy changes should not slow down authoritative DNS answers.

Policy editing, validation, testing, and publishing belong in the control plane.

Live DNS answering belongs in the data plane.

The data plane should use an active prepared version, usually a runtime snapshot.

Draft changes should not affect live answers until they are validated and published.

New snapshots should be built outside the live DNS query path.

Snapshot loading should be clean, with no half-applied state.

Operators should track active policy and snapshot versions on every DNS node.

Rollback should be ready before it is needed.

TTL affects how quickly users may see new DNS answers.

A good update system keeps DNS answers fast, keeps policy changes controlled, and gives operators a clear path to test, publish, monitor, and rollback.

In the next article, we will discuss Weighted, Geo, ASN, And Failover Routing: Steering Techniques Compared.

Sources

RFC 1034, Domain Names, Concepts and Facilities:
https://www.rfc-editor.org/rfc/rfc1034

RFC 1035, Domain Names, Implementation and Specification:
https://www.rfc-editor.org/rfc/rfc1035

RFC 6891, Extension Mechanisms for DNS:
https://www.rfc-editor.org/rfc/rfc6891

RFC 7871, Client Subnet in DNS Queries:
https://www.rfc-editor.org/rfc/rfc7871

RFC 8767, Serving Stale Data to Improve DNS Resiliency:
https://www.rfc-editor.org/rfc/rfc8767