LVS-mini-HOWTO French

Normalemen, les serveurs réels sont connectés à un réseau privé (par exemple 192.168.1.0/24). Ils ne sont pas contactés directement par les clients. De temps en temps, les serveurs réels sont des machines (sur le réseau local) utilisées pour d'autres choses, dans ce cas, laissez les sur leur réseau (d'adressage eventuellement public) d'origine.

Le directeur est joint par le(s) client(s) sur la VIP. Si la VIP doit être joignable de façon publique (la configuration normale), alors elle devrait normalement être sur un réseau différent de celui des serveurs réels, dans ce cas vous aurez un LVS à deux réseaux. La VIP sera laors sur le réseau qui relie le directeur et le routeur (ou le(s) client(s) de test). Si le(s) client(s) est(sont) sur le même réseau que les serveur réels, alors vous aurez seulement un LVS à un réseau.

Le nombre de réseau est un problème distinct du nombre de carte réseau. Une carte réseau peut avoir plusieurs adresses IP et peut être sur plusieurs réseaux (logiques sur le même réseau physique). Vous pouvez avoir 1 carte réseau sur le directeur avec un LVS à 2 réseaux.

	Note
	La VIP est habituellement sur une interface ethernet (par exemple eth0). Cependant dec (at) rm-f (dot) net 01 Jun 2002 l'a mis sur lo et le LVS a fonctionné sans problème.

Pour une première configuration, vous n'aurez besoin que d'une seule carte réseau sur le directeur Avoir 2 cartes réseau sur le directeur sépare les packets de l'intérieur et les packets de l'extérieur, ce qui simplifie les règles de filtrage et la sécurité. Les cartes à 100Mbps sont bon marché et pour la production vous pouvez en mettre autant que peut en accepter votre carte mère (il existe quelques cartes réseau à quatre ports, j'utilise la D-Link quad tulip sur mon directeur). Le debit de packet sera limité par le bus PCI/la pile TCP/IP (400Mbps pour des cartes mère en 2000) - c'est à dire 4 cartes réseau. Le directeur peut avoir un carte réseau pour chaque serveur réel même si chaque carte n'est pas utilisée à sa vitesse maximale. Augmenter le nombre de cartes réseau augmentera le débit tant que quelque chose d'autre ne sera pas le facteur limitant, par exemple la vitesse du CPU ou celle du bus PCI.

Le script de configuration prendra en compte 1 ou 2 carte(s) réseau sur le directeur. Dans le cas 1 carte, la carte est connectée à la fois au monde extérieur et au réseau des serveurs réels. Dans le cas 2 cartes, ces deux réseaux sont physiquement séparés, une carte connecté au monde extérieur, l'autre aux serveurs réels.

pour le directeur, vous avez besoin du noyau standard, qui peut être obtenu sur le site ftp noyau (Anglais). Pour les noyaux linux-2.4.x, vous devrez patcher avec le patch 'caché' (hidden); (quio a été testé seulement avec ces noyaux standards).

Note

Le noyau fourni avec votre distribution est généralement une version évoluée du noyau standard et aui ne sera surement pas patchée correctement avec LVS. Le noyau RedHat a les patchs LVS pré-appliqués, mais à une version qui sera déjà dépassé lorsque vous lirez ces lignes. SuSE (Mai 2003) fait de même. Si vous arrivez à faire fonctionner LVS avec le noyau standard mais pas avec celui de votre distribution, alors nous pouvons peut être trouver la source du probléme, mais vous feriez mieux de contacter votre distributeur - ils ont besoin de votre feedback, nous non. Si vous êtes vraiment intéressé par faire fonctionner LVS avec le noyau de votre distribution, vous pouvez le faire après avoir réssi avec le noyau standard.

Cliquez sur le lien 'software' sur la page de garde de LVS (Anglais). Récupérez la dernière archive tar ipvs pour le directeur (disponible en version 2.2.x, 2.4.x, et 2.5.x). Elle contient la patch pour le noyau et le code source pour quelques programmes et utilitaires.

Vous devez patcher le noyau standard avec le patch ip_vs correspondant, re-compiler le noyau, rebooter et compiler ensuite le programmes ipvsadm.

Le directeur fonctionne dans une (parmis plusieurs) méthode de transfert; LVS-NAT (network address translation (Translation d'Adresse Réseau)); LVS-DR (direct routing (routage direct)) et LVS-Tun (tunneling (par tunnel)). La méthode de transfert est configurée pour chaque service/serveur réel par ipvsadm.

Le code du directeur a été bien testé sur les processeurs Intel. Il est également utilisé avec succès par quelques personnes sur du matériel Alpha. Il devrait fonctionner sur n'importe quel matériel qui peut faire tourner Linux.

Les serveurs réels doivent être configurés correctement par rapport à la méthode de tranfert. Pour cela vous devez

Pour régler le problème arp pour LVS-DR/LVS-Tun, les serveurs réels avec des noyaus linux 2.2.x/2.4.x, faites tourner un noyau standard patché avec le patch "hidden" (disponible sur la page de patchs et de programmes de Julian (Anglais)). Pour les noyaux 2.2.x, ce patch est déjà dans le noyau standard (c'est à dire qu'il est prépatché pour vous). Pour les noyaux 2.4.x, vous devrez appliquer ce patch vous même. A la date de fin 2001, il était possible d'appliquer au noyau le patch hidden et le patch ipvs en même temps, ce qui permettait d'utiliser le même noyau pour le directeur et pour les serveurs réels. La partie du noyau touchée par le patch ne change pas beaucoup de version de noyau en version de noyau, donc le patch hidden pour le noyau 2.4.5 est toujours valable pour le noyau 2.4.19pre4.

Voci la liste des OS qui ont été testés avec les méthodes de transfert. (nous pensons qu'on peut arriver à faire fonctionner les OS avec toutes les méthodes d'une manière ou d'une autre.)

La manière la plus courante de régler le probleme arp est de cacher la VIP des requêtes arp.

Les exemples suivants vont vous montrer comment gérer le problème arp. Toutes ces méthode pour gérer le problème arp fonctionnent, avec à peu près le même indice de performance (débit, lantence) et sont à peu près aussi faciles/difficiles à mettre en place.

La métohde "hidden" pour dissimuler des interfaces aux requêtes arp est la plus proche du comportement standard NOARP d'unix et c'est la méthode la plus utilisée sur les serveurs réels Linux.

Pour des noyaux plus anciens (2.0.x et au moins les premiers 2.2.x), une autre méthode simple est d'installer une carte réseau supplémentaire dans les serveurs réels pour la VIP et de ne pas la connecter au réseau. La carte réseau ne reçoit pas de packets, c'est juste un moyen d'avoir la VIP sur les serveurs réels. La carte réseau peut être une vieille carte ISA 10Mbps. La prix de certaines cartes tulip PCI 100Mbps est maintenant inférieur au salaire que vous seriez payé pour recompiler un noyau avec le patch 'hidden'. (Note, 2002: - cela ne marche pas pour les noyaux 2.4.x).

Le seul moyen est de compiler un nouveau noyau, redémarrern puis complier/installer l'ipvsadm correspondant, comme si vous faisiez une installation depuis le début. Vous ne pouvez pas mettre à jour in situ, vous aurez à éteindre votre directeur.

J'utilise le script rc.system_map (qui est fourni avec le configure_script) pour être sur d'avoir la bonne version d'ipvsadm, du noyau et des fichiers system.map bien synchrones, quand j'utilise de multiples versions de LVS (c'est à dire une version 2.2 et 2.4).

You can set up the LVS by hand with ipvsadm. This is somewhat tedious and error prone. While do-able for a single LVS configuration, this is not the way to go for setting up lots of different LVS configurations.

The configure script is on the LVS software page (down at the bottom). This only sets up a single director and cannot handle director failover. For production systems you will probably want to use tools that handle director failover. Some people have setup two directors under Linux-HA using the configure script, but I have not been able to get details of how they did it.

The configure script was designed to set up LVS's quickly so I could do testing. The current version (0.9.x) does a wide range of checks, catching all the errors and deliberately pathological cases I could find in the last 2 yrs. Elementary checks are done on routing and connectivity and the script has always produced a working LVS when no errors are reported.

	Note
	the configure script tests and uses the links between all machines - the network should be working already. You should be able to ping between all machines on the same network. You will need NICs configured with all the RIPs, and the SIP and PIP (for versions 0.10.x and older) or DIP (0.9.x and earlier).

The configure script reads a conf file and sets up LVS-DR, LVS-NAT, LVS-Tun, with single or multi-NIC machines using regular ethernet devices or transparent proxy to accept packets. The output is a well commented rc.lvs and rc.mon file to run at bootup (or when you're setting up an LVS).

The configure script detects the kernel version of the director (for old versions of ipvs/kernels) and is run on the director to produce the rc.lvs file. The rc.lvs file is then run on the director and then the realservers. This allows the rc.lvs script to check the connections to the now configured director.

I have the configure files/directory nfs exported to the realservers. After running the rc.lvs script on the director, I can then run the same rc.lvs script from the realservers. (After setup, the network connections neccessary to export this directory can be removed.) Alternately the configure script will copy and execute the rc.lvs file on the realservers using ssh, using the "-i" option (to check that this will work run `ssh realserver hostname` - you should get back the hostname of the realserver). The rc.lvs file knows whether it's running on a director or realserver by looking for IPs on the machine it is running on and comparing them to the IPs in the conf file.

The rc.lvs script is verbose and reports on its progress. Watch for any errors. The script is idempotent i.e.- you can run the script as many times as you like on the same machine. You can copy the rc.lvs file from the director to the realservers for running. I have the conf files on the director in /etc/lvs directory (you can have them anywhere) which is nfs exported to the same directory on the realservers. In this way the rc.lvs script on the director can be run by the realservers. In a production system, you can umount the directories on the realservers after setup. Alternately, if you run configure with the -i (install flag), the configure script will use ssh to copy the output files to the realserver and execute the script on the realserver.

The configure script has been tested with directors on 2.2 and 2.4 kernels. It will issue warnings when it encounters a kernel later than it knows about. These warnings can usually be ignored as ipvs is not expected to change its configuration method a whole lot.

The conf file and the configure script can use either IPs or hostnames (from /etc/hosts) and can use either port numbers (eg 23) or service names (eg telnet) (from /etc/services). Likely additions needed to your /etc/services will be

ftp-data        20/tcp
ssh             22/tcp
domain          53/tcp  nameserver      dns #the string "dns" needs to be added here
domain          53/ucp  nameserver      dns #and here
shell           514/tcp cmd             rsh #add rsh here
https           443/tcp
https           443/udp
printer         515/tcp spooler         lpd #add the string "lpd" here
nfs             2049/udp        nfs
mysql           3306/tcp
mysql           3306/udp
netpipe         5002/tcp

In several instances, a machine will need multiple IPs. You can put multiple IP's on a single NIC with IP aliasing (an option when building the kernel) - just enter the aliased interface (eg eth0:12) for that IP into the *.conf file. (Note: versions of the configure script from 0.10.x onward will move to the iproute2 tools and will not be using aliases. `ifconfig` and `netstat -rn` will not work for networking set up by the iproute2 tools - you'll have to use the iproute2 tools only).

Documentation on the configure script is in perldoc format (upto v0.9, later versions will be in xml/html).

director:/etc/lvs:$ perldoc configure.pl

The script is perl and requires perl modules not needed in the demonstrations here. However the script will not run without them, so check that you have the required modules by running

$perl -w configure.pl

If you're missing a perl module (e.g. DNS.pm) you'll get an error message like

director:/etc/lvs# ./configure lvs_dr.conf.IP.two_NIC_two_network -i
Can't locate Net/DNS.pm in @INC (@INC contains: /usr/local/lib/perl5/5.6.0/i586-linux /usr/local/lib/perl5/5.6.0 /usr/local/lib/perl5/site_perl/5.6.0/i586-linux /usr/local/lib/perl5/site_perl/5.6.0 /usr/local/lib/perl5/site_perl .) at ./configure line 1365.
BEGIN failed--compilation aborted at ./configure line 1365.

Fetch the extra modules from the perl website or download them directly with the perl cpan module, which you can do with

director:# cpan
or
director:# perl -MCPAN -e "shell"

You will get the cpan prompt. Help, which you get by typing "help" at the cpan prompt, is cryptic. To get the hang of cpan, try the `r` command, which will list the most recent updates for modules you currently have installed. You should be able to install Net::DNS with

cpan> install Net::DNS
 

If your upgrade has any dependencies, you will be prompted to download them too.

The rc.lvs script uses fping to test connections. fping is available from

ftp://ftp.kernel.org/pub/software/admin/mon

The newer versions of ping have the functionality of fping and the current configure script tests for this functionality. For older versions of configure, if you don't want to download fping, you can use this script instead of fping (put it in /usr/local/bin and make it executable).

#!/bin/sh
#fping replacement

ping -c 1 $1
#-----fping-------------

Pick an appropriate template conf file (lvs_nat.conf, lvs_tun.conf or lvs_dr.conf) for the mode you are operating the LVS under (VS-NAT, LVS-Tun or LVS-DR respectively) and edit the IPs and services for your situation. There are example conf files for the various forwarding types, number of networks and number of NICs. The format of the conf file is the same for all LVS setups - only the IPs/networks change when you change from a 1-NIC director to a 2-NIC director. Several example conf files are provided but the only thing that changes between the diagrams is the network diagrams (and targets for default gw), not the format of the conf file directives. The files come preconfigured for telnet as the service. This is the simplest service for initial tests: the client is readily available and on connection, the login prompt will tell you which server you have connected to. Use round robin scheduling, so that you will connect to each server in turn, confirming that they are all working.

Other services can be added by uncommenting/adding or editing the entries in the example *.conf files. The *.conf files give suggested IP's (192.168.1.x/24, 10.1.1.x/24) for the various machines.

run the configure script

$ ./configure.pl lvs_nat.conf

This produces an rc.lvs_xxx script (eg rc.lvs_nat, rc.lvs_tun, rc.lvs_dr), and a mon_xxx.cf script. (After testing put rc.lvs_xxx into /etc/rc.d or /etc/init.d and put mon_xxx.cf in /etc/mon)

Run the rc.lvs script on the director and then the realservers with the command

$ . ./rc.lvs_dr

or possibly (if you get weird errors, eg it doesn't detect correctly whether it's running on a director or a realserver - this happens on my 2.0.36 realserver)

$sh rc.lvs_dr

The rc.lvs script -

The rc.lvs script does not -

Check the output from ipvsadm, ifconfig -a and netstat -rn on the director and then the realserver(s), to see that the services/IP's are correct. If not re-edit and re-run the script(s).

Telnet is a non-persistent service (in the http sense, rather than the LVS sense) and will allow you to check that all realservers are present (you'll get the login prompt from each realserver in turn).

Check that the service(s) are running on each server at the IP of the VIP (use netstat -an). Some services (eg telnet listen to 0.0.0.0, ie to all IPs on a machine).

If there are no errors on running the rc.lvs_xxx script, then telnet from the client to the VIP (here 192.168.1.110). You will be routed through to one of the realservers and you will get the login prompt (and real name) of a realserver and can login.

On the director look at the output of ipvsadm, you should see a connection to a realserver on port:23.

On the realserver do

$ netstat -an | grep 23

and look for connections to the telnet port.

Logout and telnet again to the VIP. You will get the login prompt from the next realserver.

Edit lvs_nat.conf activating http, rerun configure.pl and rerun the new rc.lvs_nat files.

http: Point your browser to the VIP http://192.168.1.110. You will get the DocumentRoot of one of the realservers. Open another copy of the browser and connect again. You should get the other server (this will be easier to see if the webpages are different). Look at the output of ipvsadm on the director for connections to the httpd ports, and on the server look at the output from netstat -an | grep 80 (or 8080 if you are running LVS-NAT and have remapped the ports) for connections.

If your httpd is persistent (in the http sense), you will/may connect to the same website each time.

client:      client's IP         =CIP
gateway:     gateway/router's IP =DGW (router will be the client in most test setups)
director:    director's IP       =DIP on director   eth0
             virtual IP          =VIP on director   eth0:x (eg eth0:1)
realserver:  realserver IP       =RIP on realserver eth0
             virtual IP          =VIP on realserver eth0:x/lo:0/tunl0/dummy0
             gateway             =SGW

	Note
	the DIP and the VIP must be different IPs (they can be on the same NIC).

	Note
	the DIP and the VIP will be moved to another director on director failover. These IPs will be setup as secondary IPs (aliases in the language of 2.0 and 2.2 kernels) so that you can later use director failover. The configure script sets up these IPs for you (i.e. you shouldn't have the VIP and the DIP as primary IPs already installed on your director).

Julian Anastasov ja (at) ssi (dot) bg 06 Nov 2001

If DIP==VIP the realserver will receive packets with saddr=local_ip (VIP) from the director. (This is the "source martian" problem.) You have to add additional IP(DIP) in your director and when you execute ip route get x.y.z.230 you have to see that x.y.z.180 is not your preferred source. The usual way to achieve this is to configure VIP on another device (eg. lo) or to configure them after the DIP is configured. By this way DIP will become your preferred source when talking to your subnet.

Adding DIP is not mandatory for any LVS setup to work but you will not be able to send non-LVS traffic between the director and the real servers (ping in you case).

Chris chrisd (at) better-investing (dot) org 15 Apr 2002

1. I downloaded the kernel from kernel.org.
2. I downloaded the ipvs patch (use the single patch)
3. I downloaded the hidden patch.
4. I uncompressed the kernel source in /usr/src
5. I uncompressed the hidden patch in /root
6. I uncompressed the ipvs single file.
7. I copied the hidden patch to /usr/src/linux
8. I executed: patch -p1 <hidden-2.4.5-1.diff
9. I copied the ipvs patch to /usr/src/linux
10. I executed: patch -p1 <linux-2.4.18-ipvs-1.0.2.patch
11. I did a make menu config, and configured my kernel
12. I did the exact same steps you did to compile it:

    make dep;make clean;make bzImage;make modules;make modules_install

I didn't have any glitches.

Things to watch out for:

• Don't apply each patch more than once!
• If it didn't work following these steps and you have RH 7.2, make sure you have all the gcc packages installed. If you must, put the cd back in the drive, and do an "upgrade", and install the tools necessary
• If that isn't it, make sure your gcc is up to date. An easy way of doing this is by downloading red-carpet from Ximian.org. It is an Xwindows updater. You do not have to have gnome to run this (anymore).
• If this doesn't work, track down a *real* guru, cuz I am fresh out of ideas!

The actual kernel compile instructions will vary with kernel patch number. You can build ip_vs directly into the kernel or have it as a loadable module - either will do for an initial setup.

You need to turn on "Prompt for developmental code... " under the "Code Maturity" section. In the "Networking" section, you have to turn on IP:masquerading before you get the ipvs options.

Some of the kernel compile options below are not explicitely required for LVS to work, but you'll need them anyhow - e.g. ip aliasing if you need to constuct a director with only one NIC, or tunneling if you are going to run LVS-Tun. Until you know what you're doing activate all of the options with an '*' at the start of the line. The kernel configuration options below are just for LVS. You will need other flags set (e.g. filesystems, hardware...).

The actual kernel compile instructions will vary with kernel patch number. Here's what I used for ipvs-0.9.9 on kernel 2.2.15pre9 in the Networking options.

           [*] Kernel/User netlink socket
           [*] Routing messages
           < > Netlink device emulation
*          [*] Network firewalls
           [*] Socket Filtering
           <*> Unix domain sockets
*          [*] TCP/IP networking
           [ ] IP: multicasting
*          [*] IP: advanced router
*          [*] IP: policy routing
           [ ] IP: equal cost multipath
           [ ] IP: use TOS value as routing key
           [ ] IP: verbose route monitoring
           [ ] IP: large routing tables
           [ ] IP: kernel level autoconfiguration
*          [*] IP: firewalling
           [ ] IP: firewall packet netlink device
*          [*] IP: use FWMARK value as routing key (NEW)
*          [*] IP: transparent proxy support
*          [*] IP: masquerading
           --- Protocol-specific masquerading support will be built as modules.
*          [*] IP: ICMP masquerading
           --- Protocol-specific masquerading support will be built as modules.
*          [*] IP: masquerading special modules support
*          <M> IP: ipautofw masq support (EXPERIMENTAL)
*          <M> IP: ipportfw masq support (EXPERIMENTAL)
*          <M> IP: ip fwmark masq-forwarding support (EXPERIMENTAL)
*          [*] IP: masquerading virtual server support (EXPERIMENTAL)
*          (12) IP masquerading LVS table size (the Nth power of 2)
*          <M> IPVS: round-robin scheduling
*          <M> IPVS: weighted round-robin scheduling
*          <M> IPVS: least-connection scheduling
*          <M> IPVS: weighted least-connection scheduling
*          [*] IP: optimize as router not host
*          <M> IP: tunneling
           <M> IP: GRE tunnels over IP
           [*] IP: broadcast GRE over IP
           [ ] IP: multicast routing
           [*] IP: PIM-SM version 1 support
           [*] IP: PIM-SM version 2 support
*          [*] IP: aliasing support
           [ ] IP: ARP daemon support (EXPERIMENTAL)
*          [*] IP: TCP syncookie support (not enabled per default)
           --- (it is safe to leave these untouched)
           < > IP: Reverse ARP
           [*] IP: Allow large windows (not recommended if <16Mb of memory)
           < > The IPv6 protocol (EXPERIMENTAL)

Do not change the IP masquerading LVS (hash) table size unless you know a lot more about ip_vs than we do. If you're still thinking of changing the hash table size, at least first read the HOWTO.

Do all the other kernel stuff - make modules, install the modules, copy the new kernel (and optionally System.map, required by the configure script.) into / or /boot, and edit your boot loader (lilo, grub) files. Make sure you keep your old kernel and kernel config, so you can recover if all does not go well. Reboot to the new kernel. When loading the new kernel (with ip_vs built as modules), make sure the ip_vs* modules get loaded. The README in the kernel source tree has all the necessary info in there. i

The patches to the early versions of the 2.4.x kernels were configured and installed separately to the "make menuconfig" for the kernel. This required moving files into the /lib/modules directories and loading the modules by hand. With later versions of the kernel, you can get a set of files where the patches are put into the source tree and configured by make configure. The setup varies with different ip_vs patches. Make sure you read the docs.

Note

The instructions below are for an early 2.4.x kernel. You can expect each kernel to be a little different. For 2.4.18 (Simon Young simon-lvs (at) blackstar (dot) co (dot) uk, 1 Oct 2002) IP Masquerading lives under: Networking options ---> IP: Netfilter Configuration ---> <M> IP tables support (required for filtering/masq/NAT) <M> Full NAT <M> MASQUERADE target support All the LVS stuff should be under: Networking options ---> IP: Virtual Server Configuration --->

Here's the networking config

	<*> Packet socket                                                      
	[ ] Packet socket: mmapped IO                                        
	[*] Kernel/User netlink socket                                         
	[*] Routing messages                                                 
	<*> Netlink device emulation                                         
	[*] Network packet filtering (replaces ipchains)                       
	[*] Network packet filtering debugging                               
	[*] Socket Filtering                                                   
	<*> Unix domain sockets                                                
	[*] TCP/IP networking                                                  
	[ ]   IP: multicasting                                                 
	[*]   IP: advanced router                                              
	[*]     IP: policy routing                                             
	[*]       IP: use netfilter MARK value as routing key                  
	[*]       IP: fast network address translation                         
	[*]     IP: equal cost multipath                                       
	[*]     IP: use TOS value as routing key                               
	[*]     IP: verbose route monitoring                                   
	[*]     IP: large routing tables                                       
	[*]   IP: kernel level autoconfiguration                               
	[ ]     IP: BOOTP support                                              
	[ ]     IP: RARP support                                               
	<M>   IP: tunneling                                                    
	< >   IP: GRE tunnels over IP                                          
	[ ]   IP: multicast routing                                            
	[ ]   IP: ARP daemon support (EXPERIMENTAL)                            
	[ ]   IP: TCP Explicit Congestion Notification support                 
	[ ]   IP: TCP syncookie support (disabled per default)                 
	  IP: Netfilter Configuration  --->                                    
	  IP: Virtual Server Configuration  --->                               
	< >   The IPv6 protocol (EXPERIMENTAL)                                 
	< >   Kernel httpd acceleration (EXPERIMENTAL)                         
	[ ] Asynchronous Transfer Mode (ATM) (EXPERIMENTAL)                    
                                                 
	

Here's my config for the IP: Virtual Server configuration (turn it all on)

	<M> virtual server support (EXPERIMENTAL)                                       
	[*]   IP virtual server debugging (NEW)                                         
	(12)   IPVS connection table size (the Nth power of 2) (NEW)                    
	--- IPVS scheduler                                                              
	<M>   round-robin scheduling (NEW)                                              
	<M>   weighted round-robin scheduling (NEW)                                     
	<M>   least-connection scheduling scheduling (NEW)                              
	<M>   weighted least-connection scheduling (NEW)                                
	<M>   locality-based least-connection scheduling (NEW)                          
	<M>   locality-based least-connection with replication scheduling (NEW)         
	<M>   destination hashing scheduling (NEW)                                      
	<M>   source hashing scheduling (NEW)                                           
	--- IPVS application helper                                                     
	<M>   FTP protocol helper (NEW)                                                 
	

Do not change the IPVS connection (hash) table size unless you know a lot more about ip_vs than we do. If you're still thinking of changing the hash table size, at least first read the HOWTO.

Here is my config for the netfilter section

	<M> Connection tracking (required for masq/NAT)
	<M>   FTP protocol support
	<M> Userspace queueing via NETLINK (EXPERIMENTAL)
	<M> IP tables support (required for filtering/masq/NAT)
	<M>   limit match support
	<M>   MAC address match support
	<M>   netfilter MARK match support
	<M>   Multiple port match support
	<M>   TOS match support
	<M>   Connection state match support
	<M>   Unclean match support (EXPERIMENTAL)
	<M>   Owner match support (EXPERIMENTAL)
	<M>   Packet filtering
	<M>     REJECT target support
	<M>     MIRROR target support (EXPERIMENTAL)
	<M>   Full NAT
	<M>     MASQUERADE target support
	<M>     REDIRECT target support
	<M>   Packet mangling
	<M>     TOS target support
	<M>     MARK target support
	<M>   LOG target support
	< > ipchains (2.2-style) support
	< > ipfwadm (2.0-style) support                                         
	 

	Note
	I have removed the ipchains option here. This was set to <M> in previous versions of the mini-HOWTO. However this raised problems for people who didn't understand the ipchains compatability problems.

ipchains in 2.2.x kernels has been replaced by iptables in 2.4.x kernels. For 2.4 kernels, ipchains is available for backwards compatibility. However ipchains and iptables can't be used at the same time. You should not be compiling ipchains into 2.4 kernels anymore (2.4.x has been out since 1999).

ipchains under 2.4 makes conntrack slow (see HOWTO).

The ip_tables module is incompatible with ipchains. If present, the ip_tables module must be unloaded for ipchains to work.

If you have ip_tables loaded, you'll get uninformative errors when you try to run ipchains commands with 2.4. Rather than saying that ipchains under 2.4 is there for compatibility, it would be more accurate to say that the ipchains commands available with 2.4 kernels will only cause you grief and it will be faster to rewrite your scripts to iptables, than to fall into all the holes you'll find using the compatibility. It won't take long before some script/program expects to run ip_tables on your 2.4 machine and as soon as that happens one or both (I don't know which) of your iptables or ipchains are hosed.

Do all the other kernel stuff - make modules, copy the new kernel into / or /boot (giving it a different name to your working kernel e.g. bzImage-0.9.4-2.4.12), edit lilo.conf and re-run lilo. Make sure you leave the old kernel info in lilo.conf, so you can recover if all does not go well. Reboot to the new kernel.

Boot to the new kernel and make sure /usr/src/linux points to your kernel source tree (you can use rc.system_map, which comes with the configure script), before building ipvs (if you're doing it separately) and ipvsadm.

Don't forget to run depmod -a after you install the ip_vs modules.

You can optionally compile ipvsadm with popt libraries, which allows ipvsadm to handle more complicated arguments on the command line. If your libpopt library is too old, your ipvsadm will segv. The default compile used to link against static popt libraries but now uses dynamic libraries. If you get errors about POPT not found, then you haven't installed popt, or you may not have a recent popt.

Torsten Buslei

I've just compiled and installed a new Kernel (2.4.18 with patch 2.4.19-pre8 and linux-2.4.18-ipvs-1.0.2.patch.gz). After reboot (everything's fine till here), compiling ipvsadm (either ipvs-1.0.2.tar.gz or ipvsadm-1.20.tar.gz) I've get the following error-messages:

ipvsadm.c:345: `POPT_ARGFLAG_OPTIONAL' undeclared (first use in this function)

I commented out this part of ipvsadm.c:345

{"persistent", 'p', POPT_ARG_STRING/*|POPT_ARGFLAG_OPTIONAL*/, &optarg, 'p'},

ipvsadm makes and runs fine now. Was this OK to do?

Horms horms (at) vergenet (dot) net 05 May 2002

You would appear to have a version of popt that does not support optional arguments (POPT_ARGFLAG_OPTIONAL). Your fix should be fine, with the side effect that you will have to give an argument to -p / --persistent if you use that option.

Ramish Patel wrote:

I am using Red Hat Linux 7.2, kernel 2.4.7-10, and I have attempted to apply numerous patches to it for lvs, only two of which have been successfully applied without errors -- they came from ipvs-1.0.4.tar.gz, which contained the two patches which were successfully applied to the kernel, linux_kernel_ksyms.c.diff , linux_net_netsyms.c.diff .

I re-compiled the kernel afterwards. I am still getting error messages:

"Could not open the /proc/net/ip_masq/vs file Are you sure that the IP Virtual Server is supported by the kernel?"

Horms horms (at) verge (dot) net (dot) au 31 Jul 2002

Firstly I would strongly recommend that you start with a kernel from kernel.org (such as 2.4.18) rather than using Red Hat's Kernel unless you know what you are doing. ipvs-1.0.4 does not work out of the box with the Kernel shipped with Red Hat 7.2 and vice versa.

However if you want to do this here is a rough guide.

Alex Kramarov has instructions for building/upgrading the RedHat kernel.

For linux,

If you are handling the arp problem by hiding the VIP device on the realservers, then you need to patch the realservers if

The current version(s) of 2.2.x, e.g. 2.2.19 are already patched with the arp hiding code i.e. you don't have to patch the current 2.2.x kernels. For Linux-2.0.x realservers, you can use the NOARP option when setting up a device for the VIP.

For 2.4.x linux realservers, apply the hidden patch (on the LVS software page) to your kernel tree and rebuild with your original options. For the 2.4.4 kernel -

realserver:/usr/src/linux# patch -p1 <../arch/hidden-2.4.4-1.diff

This list of realservers is from Ratz. About the only thing he hasn't tried yet is Plan 9. Remember to set netmask=/32 for the VIP on LVS-DR and LVS-Tun (for LVS-NAT you can setup with any netmask you like). If you are running non-Linux unix realservers, you can usually handle the arp problem by configuring the device carrying the VIP with the -arp switch.

Ratz's code to setup non-linux realservers is now in the configure script. This part of the script has not been well tested (you might find that it doesn't setup your non-linux unix box properly yet, please contact me - Joe).

	Note
	(In the 3yrs the configure script has been out, I've not heard of anyone using this part of the code, so there seems no point in maintaining it.) The v0.10 configure script (coming out sometime in 2003 I'd guess) will not have this code in it. The configure script will be using iproute2 to install network devices and routes. You'll have to setup non-Linux realservers yourself.

Here's the information for non-Linux unices. On some Unixes you have to plumb the interface before assigning an IP. The plumb instruction is not included here.

#uname      : FreeBSD
#uname -r   : 3.2-RELEASE
#<command>  : ifconfig lo0 alias <VIP> netmask 0xffffffff -arp up 
#ifconfig -a: lo0: flags=80c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST>mtu 16837
#                  inet 127.0.0.1 netmask 0xff000000
#                  inet <VIP> netmask 0xffffffff

#uname      : IRIX
#uname -r   : 6.5
#<command>  : ifconfig lo0 alias <VIP> netmask 0xffffffff -arp up
#ifconfig -a: lo0: flags=18c9<UP,LOOPBACK,RUNNING,NOARP,MULTICAST,CKSUM>
#                  inet 127.0.0.1 netmask 0xff000000
#                  inet <VIP> netmask 0xffffffff

#uname      : SunOS
#uname -r   : 5.7
#<command>  : ifconfig lo0:1 <VIP> netmask 255.255.255.255 up
#ifconfig -a: lo0:  flags=849<UP,LOOPBACK,RUNNING,MULTICAST>mtu 8232
#                   inet 127.0.0.1 netmask ff000000
#             lo0:1 flags=849<UP,LOOPBACK,RUNNING,MULTICAST>mtu 8232
#                   inet <VIP> netmask ffffffff

#uname      : HP-UX
#uname -r   : B.11.00
#<command>  : ifconfig lan1:1 10.10.10.10 netmask 0xffffff00 -arp up
#ifconfig -a: lan0:   flags=842<BROADCAST,RUNNING,MULTICAST>
#                     inet <some IP> netmask ffffff00
#             lan0:1: flags=8c2<BROADCAST,RUNNING,NOARP,MULTICAST>
#                     inet <VIP> netmask ffffff00
#

Ratz 16 Apr 2001

in most cases (when using the NOARP option) you need alias support. Some Unices have no support for aliased interfaces or only limited, such as QNX, Aegis or Amoeba for example. Others have interface flag inheritance problems like HP-UX where it is impossible to give an aliased interface a different flag vector as for the underlying physical interface (as happens with Linux 2.2 and 2.4 - Joe). So for HP/UX you need a special setup because with the standard depicted setup for DR it will NOT work. I've done most Unices as Realserver and was negatively astonished by all the different implementation variations of the different Unix flavours. This maybe resulted from unclear statements from the RFC's.

Windows is not handled by the configure script (which needs bash to run).

Instructions for setting up windows realservers is one of the more common questions on the mailing list. This must be a difficult part of the mini-HOWTO to find ;-\. There seem to be many ways of doing it. Here are some of the answers.

Wensong's original recipe for setting up the lo device on a NT realserver.

If you don't have MS Lookback Adapter Driver installed on your NT boxes, enter Network Control Panel, click the Adapter section, click to add a new adapter, select the MS Loopback Adapter. Your NT cdrom is needed here. Then add your VIP (Virtual IP) address on the MS Loopback Adapter, do not enter a gateway address on the Loopback Adapter. Since the netmask 255.255.255.255 is considered invalid in M$ NT, you just accept the default netmask, then enter MS-DOS prompt, remove the extra routing entry.

c:route delete <VIP's network> <VIP's netmask>

This will make the packets destined for this network will go through the other network interface, not this MS Loopback interface. As I remember, setting its netmask to 255.0.0.0 also works.

Jerome Richard jrichard (at) virtual-net (dot) fr

On Windows NT Server, you just have to install a network adapter called "MS Loopback" (Provided on the Windows NT CDROM in new network section) and then you setup the VIP on this interface.

o1004g o1004g (at) nbuster (dot) com

robert.gehr (at) web2cad (dot) de; 24 Oct 2001

The MS-Loopback adapter is a virtual device under Windows that does not answer any arp requests. It should be on a Server Edition CD of WinNT/2000. Install and assign it the appropriate IP Address. Because MS would not let you assign a "x.x.x.x/32" netmask to the MS-Loopback adapter, you will end up having two routes pointing into the same net. Lets say your RIP is 10.10.10.10 and your MS-Loopback VIP is 10.10.10.11. You will have two routes in your routing table both pointing to the 10.0.0.0 net. You delete the route that is bound to the VIP on the MS-Loopback adapter with a command like

route del 10.0.0.0 mask 255.0.0.0 10.10.10.11

Johan Ronkainen jr (at) mpoli (dot) fi

True with Windows NT4. However with Win2000 you can just configure high metric value for loopback interface. I tried this about year ago with metric value 254 without problems. I think you could change netmask to /32 with regedit. Haven't tried that with WinNT4/2000 though. I've used that trick with Win98SE and it worked.

Brent Knotts brent (dot) knotts (at) mylocalgov (dot) com 28 Jan 2003

Concerning Windows 2000 realservers:

It is possible to change the subnet mask to 255.255.255.255 in the registry, and it works fine for my DR setup. This is easier than deleting the extra route to the local interface after each boot.

In Windows 2000, the interfaces are found in:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\Interfaces

Find the interface with the appropriate IP address and change the subnet mask. Rebooting is not necessary, but you should bring the interface down and up.

Foreign Languages for MS

Sebastien Sebastien.Bonnet (at) experian (dot) fg 12 Apr 2002

The "MS Lookback Adapter" is called "carte de bouclage"

Malcolm Turnbull

I've set up a simple LVS DR loadbalancer with 4 IIS web servers (win2K) behind it. I've setup a local loopback adapter on each 2K server and set the priority to 254. The loadbalancer works fine... But seems to confuse Windows Networking, SMB Network Browsing no longer seems to work. (Required by RoboCopy).

Martijn Klingens mklingens (at) ism (dot) nl 10 Sep 2002. Subject: Re: LVS DR setup with NT2K servers

We're using an almost similar setup, just that we're currently using a simple xcopy and want to migrate to DFS. That doesn't change the basics though. Things that I encountered:

• Windows always adds a route to the subnet on the loopback adapter. If that subnet is also available on the normal LAN your routing table will get confused. In our case we migrated sites with existing IPs to LVS so we had to pick an entire class C subnet as mask. Solution: manually delete the route to the /24 on the loopback interface.
• Disabling the file and print services on the loopback interface usually also helps.
• Check the default gateways and the DNS on the various interfaces. If they are not identical you may find very strange behaviour (although there are cases where it's useful to have them differ, this is not that common).

Hope this helps. If not, please specify a bit more about your setup. If you have problems on the realservers, are they able to resolve another realserver using dns and/or ping another realserver?

Oh, and you mentioned network browsing, which is not entirely the same as opening a share on a computer with a well-known name. Do you really need the browser? We usually have the computer browser service stopped anyway so an intruder isn't instantly able to see all other w2k hosts on the net. A bit moot, since DNS provides the same data, but it doesn't really hurt either.

Malcolm Turnbull

Has anyone used Mac OS X for a real server in LVS/DR ?

Jerome RICHARD jrichard (at) virtual-net (dot) fr 20 Nov 2002

If you wish to configure multiple IP on Mac OS X, you should use the following command :

 
sudo ifconfig lo0 alias 10.0.0.80 netmask 255.255.0.0
 

There is more information in the AppleCare Documents

Has anyone used Mac OS X for a real server in LVS/DR ?

Roberto Nibali ratz (at) drugphish (dot) ch 20 Nov 2002

Yes, I gave a presentation about a year ago and I remember that one of the attendees had a new shiny G4 with Mac OS X. It worked flawlessly.

Is it OK to use the loopback interface ?

You can but you don't have to. OS X is BSDish, so you need to use BSDish syntax:

 
ifconfig lo0 alias VIP netmask 255.255.255.255 -arp up
 

Malcolm Turnbull Malcolm.Turnbull (at) crocus (dot) co (dot) uk 22 Nov 2002

Got it working:

• Solaris:

  ifconfig lo0:1 plumb ifconfig lo0:1 <vip-addr> netmask 255.255.255.255 up

• Mac OS X:

  ifconfig lo0 alias VIP netmask 255.255.255.255 -arp up