Heartbeat is a component of the Linux-HA code. This section describes setup of heartbeat. There is a writeup on setting up Linux-HA from rpms elsewhere here.

Here are comments I picked up from mostly from the Linux-HA mailing list

on using serial and/or ethernet heartbeat

anon

I have successfully setup a 2 node HA solution. The heartbeat is establishes vie ttyS0 and eth1. Both nodes have specific tasks and take over the tasks of the other node if something wents wrong. Now I have a few questions:

I think a serial line is the last thing that goes down when anything wents wrong. So it could be possible that all network services die and the node is still considered to be up and running if ttyS0 still provides heartbeat. In this case I have a half dead node with no network but the 2nd node won't takeover. Now I could only use heartbeat on eth1 or even eth0 but this setup is, according to the howto's, not the best.

Lorn Kay lorn_kay (at) hotmail (dot) com 02 Jul 2002

It can cause a failover when the only problem is a NIC failure. Depending on your app. this may not be a good thing. High network traffic can also clog the network path for the heartbeats and start a failover when you don't want one. (See past postings on the list from Alan for how to tune heartbeat to avoid this problem).

I believe the design philosophy of heartbeat [someone please correct me if I'm wrong] is that heartbeats are distinct from resource monitoring. Heartbeat's job is simply to listen to heartbeats and only decide if the other server is dead or alive (true to its name "heartbeat").

It is considered a hack to run heartbeats on the public or resource network and use them to detect a failure of a resource. Resource monitoring (at the moment) needs to be handled by another program like "mon".

I also tried to use the softdog module, but I don't quite know when it should be used. So I can't test that. The nicest thing would be to tell heartbeat what it should do if only the ttyS0 is up and everything else is down. Or better, that it should check the ip's it should take over and if the IP doesn't respond it should take it (?)

A design principle of HA systems (per "Blueprints for High Availability: Designing Resilient Distributed Systems") is three distinct paths to the servers: resource-path (or public path), heartbeat, and administrative. However, having said all this, I still think a simple one-nic server configuration with all three paths configured as virtual (IP alias) paths over one NIC makes sense in some situations. If the NIC goes down (in these cases) then failover is a good thing. It is also easier to configure heartbeat by itself than heartbeat+mon.

Maybe someone on the list will further explain why this is a bad thing, or what is in store for the future of heartbeat. But at the moment, some of us are doing it this way.

Alan Robertson alanr (at) unix (dot) sh 02 Jul 2002

OK... Here's the deal... If you monitor the network, and discover that you can't connect, then you can theoretically fail over gracefully to another machine very quickly, without any fencing or stonith, or anything else. If you can't communicate with the other node, then you must assume it is somehow deranged, and have to fence it off from the world, and you can't gracefully take things over. It takes longer, and if STONITH is configured, then it also reboots the other machine, which is really just fine, and doesn't need to be rebooted. Also, it could be that neither node has access to the outside world, and then taking things over is a wasted effort and may result in a cycle of each machine rebooting the other until network connectivity is restored.

We are in the throes of desigining a new cluster manager for linux-ha. That would solve this problem. We have a nice membership layer already tested (but not yet in CVS). The rest is probably 3-6K lines of code.

Daniel Khan

Thinking about a solution I had a (maybe naive) idea. If any heartbeat connection to node2 fails node1 uses the remaining connection (ttyS or eth) to ask node2 to run some self checks. With some basic checks it should be possible to find out if someting is wrong.

Alan Robertson alanr (at) unix (dot) sh 02 Jul 2002

That's commonly done in one form or another. The current code doesn't do that. But -- this is the general idea - that loss of communication reduces your options for recovery.

And another newbie question on stonith: I have allready read the basic info's on stonith, but haven't found a paper about its configuration with heartbeat. Can you point me to the right destination.

OK... First of all - you need a STONITH device - which is a piece of hardware. Then you need to configure your particular device.

I thought STONITH does some ssh trick. Sorry - no device - missunderstood the whole thing.

You *can* use a STONITH ssh device, but it doesn't work when the system is sick or incommunicado, so it's only useful for testing.

I read that Watchdog (Softdog in my case) will shutdown/reboot the Node itself if it appears to be dead (no heartbeat). How can I do a test on this. I don't quite understand how the node itself can declare itself to be sick (it could also be the node on the other side)

It's not quite like that...

Heartbeat listens for heartbeats from every machine in the cluster, including from itself. Every time it *hears* its own heartbeat, it tickles the watchdog timer. If the system is hung in the kernel, or heartbeat is hung internally somehow, or the scheduler or I/O subsystem doesn't allow it to run for a while, then after about a minute of not getting tickled by heartbeat, the watchdog timer will trigger, and the system will undergo an ungraceful reboot.

If you're using softdog, then certain kinds of kernel hangs (those that keep the timer interrupt from happening), will not be caught. Most will be caught, but not all.

If you have a hardware timer, then no software problem whatsoever will cause it to hang indefinitely.

malcolm lists (at) netpbx (dot) org 14 Jun 2006

If you plug in a null modem cable (as well as your current ethernet connection) for the heartbeat, it will resolve the issue. In order for a ping node to work, the heartbeat must still be active, you can't do it just over one link (e.g. ethernet) or you'll get a split brain.

Ard van Breemen ard (at) telegraafnet (dot) nl 25 Feb 2003

I always had troubles using IPaddr. I therefore made IPaddr_ip (http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/files/IPaddr_ip), which uses ip instead of ifconfig, and arping instead of sendarp.