Failover Triggers in Software

Draw ECC200 in Software Failover Triggers

Many things can trigger failover on an appliance: loss of power, one or more interfaces failing, a card failing, a software problem like memory exhaustion, or someone forcing failover with the failover active command on the standby unit Based on the amount of time it takes to detect a problem, cutover to the standby unit might not be immediate Table 23-1 shows the cutover times for ASAs, and Table 23-2 for PIXs

Active unit motherboard interface 5 seconds is down Active unit 4-GE card interface is down Active unit IPS or CSC card fails Active unit interface is up, but has connection problems that cause interface testing 5 seconds 2 seconds 25 seconds

23:

Failover Condition Active unit loses power or stops normal operation Active unit interface is up, but has connection problems that cause interface testing

Basically two types of interfaces are monitored by the failover pair: failover link and data interfaces This section will discuss what monitoring on the failover link is, and the next section will discuss monitoring of the data interfaces Failover hello messages are generated on the failover link every 15 seconds by default (I ll show you how to change this later in the chapter) If three consecutive hello messages are missed from a failover mate, an ARP is generated on all the appliance interfaces If no response is received from the mate on any of the interfaces, failover will take place, with the unit promoting itself to an active state, assuming that it was in a standby state If no response to the ARP is seen on the failover link, but a response is seen on one of the other interfaces (stateful link or data interfaces), then failover will not occur In this situation, the failover link is marked as failed

Interface monitoring is used to monitor the status of any of the data or stateful link interfaces on the appliance Failover hello messages are generated on all active interfaces These are the same messages used on the failover link connection Up to 250 interfaces can be monitored per appliance If a hello message from a mate is not seen on a monitored interface for one-half the hold-down period, the appliance will run interface tests on the suspect interface to determine what, if any, problem exists The purpose of the interface tests is to determine which unit, if any, has had a failure Before each test begins, the received packet count statistic is cleared on the interface At the conclusion of each test, each appliance checks to see if any valid frames/packets were received; and if so, the interface is considered operational If no traffic is seen for a particular test, then the appliance proceeds to the next test The four interface tests that the appliance may run include the following: Link up/down test The suspect interface is disabled and re-enabled, where normal interface hardware diagnostics are run Network activity test The appliance looks for valid frames coming into the interface for up to 5 seconds

ARP test The appliance generates ARP queries for the two most recent entries in the ARP table, where the appliance is looking for any valid frame (not just an ARP reply) coming into the interface for up to 5 seconds Broadcast ping test The appliance generates a broadcast ping, where again, the appliance is looking for any valid frame (not just an ICMP echo reply) coming into the interface for up to 5 seconds

Normally the appliances are connected to switches for layer 2 connectivity Based on the fact that the appliances generate failover messages on all their active interfaces by default, it is important that nothing disrupts this process, causing inadvertent failovers, or, in a worst-case situation, temporarily having both appliances in an active state To greatly reduce the likelihood of unattended failovers, you should make sure that the paired interfaces on the mates can see each other s hello messages Therefore, on the switch, you need to make sure that the interfaces are in the same VLAN (If this is impossible, you can disable monitoring for the unconnected interface, which I discuss later in the chapter) Next make sure that any disruptions that STP might create won t cause the appliance interfaces to be placed into a nonforwarding state, like blocking For Cisco Catalyst switches, you should enable the PortFast feature, which keeps the ports in a forwarding state when changes in STP are occurring If you forget to do this and your switches are not using rapid STP (RSTP), but are using IEEE s original implementation (8021d), then STP recalculations, which could take anywhere from 30 to 45 seconds, will probably cause the appliances to miss three hello messages and cause possible failover problems

Failover Result Both appliances see no valid frames on the interface being tested Both appliances see a valid frame on the interface being tested The active unit sees a valid frame on the interface being tested, but the standby unit doesn t The active unit doesn t see a valid frame on the interface being tested, but the standby unit does

Failover Response No failover takes place No failover takes place No failover takes place Failover takes place