Firewall High Availability: What a Real Failover Test Should Prove

How firewall high availability works

In an active/passive pair, one firewall handles traffic while the other stays synchronized and takes over if the active unit fails. The pair shares virtual addresses, and the standby receives configuration and, depending on the platform, session state over dedicated HA links. In an active/active design, both units pass traffic, which uses capacity better but complicates routing, NAT and troubleshooting. In either case, each firewall must be able to carry the full load alone.

What can go wrong

• Split brain: the HA links fail, both units believe the other is down, and both become active.
• Sessions not preserved: some traffic types, decrypted sessions or VPN security associations may not be synchronized on every platform.
• No trigger: an upstream provider fails without the firewall's own interface going down, and nothing tells the pair to fail over.
• Stale switching tables: neighbors keep sending traffic to the failed unit until address tables update.
• Routing drops: dynamic routing neighbors reset and routes take time to return.
• Licensing gaps: security subscriptions active on one unit only.
• Configuration drift: settings that are not synchronized differ between units.
• Capacity: an active/active pair running both units at high load cannot absorb a failure.

Before the test

• Document the topology, HA settings, monitored interfaces and path monitoring.
• Agree a maintenance window, approvals and a rollback plan.
• Start continuous test traffic: a ping, a web transaction, a file download, a voice call, a site-to-site VPN and, for retailers, a test payment.
• Make sure logging and monitoring will capture timings.
• Have people ready at affected sites and the vendor's support contact at hand.

Scenarios worth testing

What to verify during each scenario

• How long each application was interrupted, and whether sessions survived.
• How quickly site-to-site and remote access VPNs recovered.
• Whether routing adjacencies and routes returned as expected.
• That NAT and published services still work.
• That logs continue to arrive in the central logging system from the newly active unit.
• That the on-call team received the right alerts.
• That threat prevention subscriptions are active on the new active unit.
• That administrators can still reach the management interface.

Check capacity honestly

Compare peak throughput, concurrent sessions and VPN users with the capacity of a single unit, with the inspection features you actually use turned on. Datasheet numbers without inspection enabled are rarely a fair comparison.

Record and repeat

Record each scenario with timings, observations and issues, assign actions and get sign-off from the service owner. Repeat the test after installation, after firmware upgrades and topology changes, and on an agreed schedule. NIST SP 800-41 recommends testing firewall policy and operation as part of managing firewalls over their lifecycle.