Michael McConnell:

Top doesn't display CPU usage of ipchains or ipvsadm. vmstat doesn't display CPU usage of ipchains or ipvsadm.

Joe

ipchains and ipvsadm are user tools that configure the kernel. After you've run them, they go away and the kernel does it's new thing (which you'll see in "system"). Unfortunately for some reason that no-one has explained to me "top/system" doesn't see everything. I can have a LVS-DR director which is running 50Mbps on a 100Mpbs link and the load average doesn't get above 0.03 and system to be negligable. I would expect it to be higher.

Julian 10 Sep 2001

Yes, the column is named "%CPU", i.e. the CPU spend for one process related to all processes. As for the load average, it is based on the length (number of processes except the current one) of the queue with all processes in running state. As we know, LVS does not interract with any processes except the ipvsadm. So, the normal mode is the LVS box just to forward packets without spending any CPU cycles for processes. This is the reason we want to see load average 0.00

OTOH, vmstat reads /proc/stat and there are the counters for all CPU times. Considering the current value for jiffies (the kernel tick counter) the user apps can see the system, the user and the idle CPU time. LVS is somewhere in the system time. For more accurate measurement for the CPU cycles in the kernel there are some kernel patches/tools that are exactly for this job - to see what time takes the CPU in some kernel functions.

The number of active/inactive connections are available from the output of ipvsadm.

Julian 22 May 2001

Conns is a counter and is incremented when a new connection is created. It is not incremented when a client re-uses a port to make a new connection (Joe, - the default with Linux).

director:/etc/lvs# ipvsadm
IP Virtual Server version 0.2.12 (size=16384)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port             Forward Weight ActiveConn InActConn
TCP  lvs2.mack.net:0 rr persistent 360
  -> RS2.mack.net:0                 Route   1      0          0
  -> RS1.mack.net:0                 Route   1      0          0
TCP  lvs2.mack.net:telnet rr
  -> RS2.mack.net:telnet            Route   1      0          0
  -> RS1.mack.net:telnet            Route   1      0          0

You can monitor connections with snmp.

Dennis Kruyt d (dot) kruyt (at) zx (dot) nl 30 Jun 2004

I use lvs-snmp (http://anakin.swiss-support.net/~romeo/lvs-snmp/) and cacti to graph the connections.

AJ Lemke

I am running a 2 node lvs-cluster and was wonder if the list could recommend a traffic monitoring program. My LVS is the frontend for a reverse proxy cache and I would like to know the traffic that each VIP is handling. I need to know the data rates on a per ip basis. I use mrtg at the switch level but I need to have more granularity, hence the need for per IP basis.

Kjetil Torgrim Homme kjetilho (at) ifi (dot) uio (dot) no 11 Jul 2004

munin (http://www.linpro.no/projects/munin/) has a plugin for this. you can get the numbers you need with ipvsadm

# ipvsadm -L -t smtp:smtp --stats
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  smtp.uio.no:smtp              1508879 38457326        0   10461M        0
  -> mail-mx6.uio.no:smtp           374117  9490846        0    2664M        0
  -> mail-mx3.uio.no:smtp           377646  9961956        0    2543M        0
  -> mail-mx2.uio.no:smtp           378502  9288837        0    2707M        0
  -> mail-mx1.uio.no:smtp           378614  9715687        0    2546M        0
# ipvsadm -L -t smtp:smtp --rate
Prot LocalAddress:Port                 CPS    InPPS   OutPPS    InBPS   OutBPS
  -> RemoteAddress:Port
TCP  smtp.uio.no:smtp                    7       85        0    20480        0
  -> mail-mx6.uio.no:smtp                1       17        0     1126        0
  -> mail-mx3.uio.no:smtp                1       17        0     2023        0
  -> mail-mx2.uio.no:smtp                2       26        0     6681        0
  -> mail-mx1.uio.no:smtp                2       25        0    10650        0

Cyril Bouthors:

Where can I get the info originally in /proc/net/ip_vs_stats and removed since 0.9.4?

Wensong Zhang wensong (at) gnuchina (dot) org 20 Nov 2001

for global stats /proc/net/ip_vs_stats

You can get per-service statistics by

ipvsadm -Ln --stats -t|u|f service-address

If you want to program to get statistics info, use libipvs. Here's the writeup that went with the original code.

Packet throughput (in 64-bit integers) is in /proc/net/ip_vs_stats or /proc/net/ip_masq/vs_stats. The counters are not resetable, you have to keep the previous reading and substract. Output is in hexadecimal.

kernel 2.4:#cat /proc/net/ip_vs_stats kernel 2.2:#cat /proc/net/ip_masq/vs_stats

Here's the statistics

director:# more /proc/net/ip_vs_stats TotalConns InPkts OutPkts InBytes OutBytes 98F9 13945999 13609E49 613AD3B2F 4F90FE6F9E Virtual Service Pro VirtService Conns InPkts OutPkts InBytes OutBytes TCP C0A8026E:0000 4 12 0 00000043B 000000000 TCP C0A8026E:0017 7 3A9 0 00000C3A5 000000000 Real Service Pro VirtService RealService Conns InPkts OutPkts InBytes OutBytes TCP C0A8026E:0000 C0A8010C:0000 4 14 0 0000004B4 000000000 TCP C0A8026E:0000 C0A8010B:0000 1 3 0 0000000B4 000000000 TCP C0A8026E:0017 C0A8010C:0017 4 A2 0 00000232A 000000000 TCP C0A8026E:0017 C0A8010B:0017 4 32A 0 00000A827 000000000

Joe

Can I zero out these counters if I want to get rates, or should I store the last count?

Ratz, May 2001

There was a recent (2 months ago) talk about zeroing in-kernel counters and I'm not so sure if all the kernel hacker gurus agreed but:

You must not zero a counter in the kernel!

I didn't really understand the arguments against or pro zeroing counters so I'm not a big help here, but if others agree we certainly can add this feature. It would be ipvsadm -Z as an analogy to ip{chains|tables}. BTW, we are proud of haveing 64-bit counters in the kernel :)

Storing ... there are different approaches to this (complexity order):

There are a family of monitoring tools descended from MRTG. These now include RRDtool (a descendant of MRTG, written by the same author, Tobias Oetiker) and wrappers around RRDtool like lrrd (which have spawned their own family of programs, e.g. cricket, to monitor and graph just about anything you like). lrrdtool can/does use nagios.

Laurie Baker lvs (at) easytrans (dot) com 20 Jan 2004

Nagios is a monitoring tool previously known as Netsaint.

I've read the documentation for mrtg and several of its descendants and haven't been about to figure out how they work enough to get them going. While the syntax of all of the commands is available, there is no global picture of how they are used to make a working set of programs. I saw Tobias give a talk at Usenix one year about MRTG and while I knew what it did, I didn't know how to set it up. Some people have got these packages going, presumably needing less documenation that I do. I'd like a worked example of how a single simple variable (e.g. the contents of /proc/loadavg) is sampled and plotted. The accompanying packages needed (e.g. SNMP, php, gd...) are not described. While a competent sysadmin will be able to work out what is missing from the output of the crashes, it would be better to prepare ahead of time for the packages needed, so that you can plan the time for the install and won't have to stop for lack of information that you could have handled ahead of time.

	Note
	This was the first attempt to produce a graphical monitoring tool for LVS. It doesn't seem to be under active developement anymore (Apr 2004) and people are now using rrdtool (or ganglia which uses rrdtool) (see below).

Note

Alexandre has a new address Alexandre (dot) Cassen (at) free (dot) fr and moved his pages to Alexandre's open source code (http://www.lnxos.net/). The links below to Alexandre's pages are dead. You can find files here lvsgsp-0.0.4.tar.gz (http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/files/lvsgsp-0.0.4.tar.gz). alex alshu (at) tut (dot) by 20 May 2008 has provided lvsgsp_newscripts.tar.bz2 (http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/files/lvsgsp_newscripts.tar.bz2) that he uses with lvsgsp.

Alexandre Cassen alexandre (dot) cassen (at) wanadoo (dot) fr, the author of keepalived has produced a package, LVSGSP that runs with MRTG to output LVS status information. Currently active and inactive connections are plotted (html/png).

The LVSGSP package includes directions for installing and a sample mrtg.cfg file for monitoring one service. The mrtg.cfg file can be expanded to multiple services

WorkDir: /usr/local/mrtg
IconDir: /usr/local/mrtg/images/

# VS1 10.10.10.2:1358
Target[VS1]: `/usr/local/bin/LVSGSP 10.10.10.2 1358`
Directory[VS1]: LVS
MaxBytes[VS1]: 150
.
.

# VS2 10.10.10.2:8080
Target[VS2]: `/usr/local/bin/LVSGSP 10.10.10.2 8080`
Directory[VS2]: LVS
MaxBytes[VS2]: 150
.
.

A note from Alexandre

Concerning the use of MRTG directly onto the director, we must take care of the computing CPU time monopolised by the MRTG graph generation. On a very overloaded director, the MRTG processing can degrade LVS performance.

Peter Nash peter (dot) nash (at) changeworks (dot) co (dot) uk 18 Nov 2003

I'm using a perl script to pull LVS statistics from my directors into MRTG using the ucd-snmp-lvs module. I'm sure this could be easily modified to work with RRDTool. I'm no perl programmer so I'm sure there are better ways to do this but it's been working for me for the last 3 months. Since my MRTG runs on a remote server (not the directors) using SNMP gives me the remote access I need. The main problem to overcome was that the "instance number" of a particular "real service" is dependent on the order in which the services are added to the IPVS table. If you are using something like ldirectord to add/remove services then this order can vary, so the script has to solve this problem. I also had a few problems getting the ucd-snmp-lvs module to compile with net-snmp on my RH8 directors but that was probably down to my lack of knowledge, I got there in the end!

The MRTG call to the script is as follows (director names, SNMP community and IP addresses are "dummies"):

Target[lvs-1]: `/home/agents/snmpipvsinfo.pl director1 communitystring
123.123.123.123 80 bytes` + `/home/agents/snmpipvsinfo.pl director2
communitystring 123.123.123.123 80 bytes`

This aggregates the results from both primary and backup director so it doesn't matter which one is "active". The script returns zeros if the requested service is not currently in the LVS table on the target director.

#!/usr/bin/perl

#
============================================================================
# LVS Stats info script for mrtg
#
#    File:      snmpipvsinfo.pl
#
#    Author:    Peter Nash 17/06/03
#
#    Version:   1.0
#
#    Purpose:   Uses SNMP to get the IPVS stats on an LVS director.
#               Needs to find the correct instance in the lvsServiceTable to
#               match a given virtual server (the instance number
#               depends on the order in which services are added).
#
#    Usage:     ./snmpipvsinfo.pl director community service_ip service_port [conn|packets|bytes]
#
#    Notes:     The instance number of a given service in the LVS table
#               depends on the order in which the services are added to the table.
#               For example, if a monitoring service such as ldirectord is used
#               to add/remove services to LVS then the instance number of a service
#               will be based on the polling sequence of ldirectord.  As services are
#               added or removed the instance numbers of existing services may
#               change.  Therefore this script has to determine the current SNMP
#               instance number for each LVS service every time it is run.
#               In addition to the director address and SNMP community if takes the
#               service IP and service PORT as parameters to identify a specific
#               service.  The last option determines the static to return.
#               Output is in MRTG compatible format.
#
============================================================================

$director=shift;
$community=shift;
$service_ip=shift;
$service_port=shift;
$mode=shift;
$instance="";

# First we need to find the LVS instance for this service
# Get all service addresses
@addresses=`snmpwalk -v 2c -c $community -m LVS-MIB $director
lvsServiceAddr`;
# Get all the service ports
@ports=`snmpwalk -v 2c -c $community -m LVS-MIB $director lvsServicePort`;

# Now for each service check to see if both address and port match
foreach $i (0 .. $#addresses) {
  ($address,)=splitnamevalue($addresses[$i]);
  ($port,$thisinstance)=splitnamevalue($ports[$i]);
  if ( $address =~ /$service_ip/ ) {
    if ( $port =~ /$service_port/ ) {
      $instance=$thisinstance;
    }
  }
}

# Now we've got the instance for the service get the requested data

if ( $instance eq "") {
  # If the instance does not exist return zero's (i.e. this may be the
backup director)
  $param1="0: = 0";
  $param2="0: = 0";
} else {
  if ( $mode eq "conn" ) {
    $param1=`snmpget -v 2c -c $community -m LVS-MIB $director lvsServiceStatsConns.$instance`;
    $param2=`snmpget -v 2c -c $community -m LVS-MIB $director lvsServiceStatsConns.$instance`;
  } elsif ( $mode eq "packets" ) {
    $param1=`snmpget -v 2c -c $community -m LVS-MIB $director lvsServiceStatsInPkts.$instance`;
    $param2=`snmpget -v 2c -c $community -m LVS-MIB $director lvsServiceStatsOutPkts.$instance`;
  } elsif ( $mode eq "bytes" ) {
    $param1=`snmpget -v 2c -c $community -m LVS-MIB $director lvsServiceStatsInBytes.$instance`;
    $param2=`snmpget -v 2c -c $community -m LVS-MIB $director lvsServiceStatsOutBytes.$instance`;
  } else {
    $param1="";
    $param2="";
    print "Error in mode parameter";
  }
}

# Get the uptime
$uptime=`snmpwalk -v 2c -c $community $director sysUpTime.0`;
$uptime =~ s/.*\)\s+(\w+)/$1/;
($value1,)=splitnamevalue($param1);
($value2,)=splitnamevalue($param2);
print "$value1\n";
print "$value2\n";
print "$uptime";
print "LVS $mode\n";


sub splitnamevalue {
    $namevalue=shift;
    chomp($namevalue);
    ($index,$value)=split(/ = /, $namevalue);
    $index =~ s/.*\.([0-9]{1,6})$/$1/;
    $value =~ s/.*:\s+(\w+)/$1/;
    return $value,$index;
}

Salvatore D. Tepedino sal (at) tepedino (dot) org 21 Nov 2003

I posted the new version on my site: http://tepedino.org/lvs-rrd/. The new version has a lot of code cleanup, much more flexibility in the coloring, a command line arg so you can just graph traffic to one port (ie: just port 80 traffic), and the update script has been changed slightly to remove a redundant loop (Thanks Francois! If I do something that obviously silly again, you can smack me!) and the removal of the need to specify what type of LVS yours is (Route, Masq, etc). Now it should collect data on all servers in the LVS. Next step is to figure out how to graph specific services (VIP/Port combinations instead of just specific ports)...

	Note
	Jun 2006. tepedino.org is not on the internet. The last entry in the wayback machine is 10 Feb 2005. Leon Keijser e-mailed me lvs-rrd-v0.7.tar.gz (http://www.austintek.com/WWW/LVS/LVS-HOWTO/HOWTO/files/lvs-rrd-v0.7.tar.gz) which has a Changelog of Jan 2006.

	Note
	Sebastian Vieira sebvieira (at) gmail (dot) com 10 Nov 2006

For those interested, the website of lvs-rrd is back up again at its usual address: http://tepedino.org/lvs-rrd/

Joe: I contacted Sal off-list, to find there'd been problems at the ISP. He's back, with the same e-mail address etc. v0.7 is still his latest code. If the server goes down again, you can contact him sal (dot) tepedino (at) gmail (dot) com.

21 Jan 2004

This new version allows you to graph connections to a specific VIP or realserver or VIP port or RS Port or any combination of those via command line options. It also adds in adds an option flip the graph for people with more inactive than active connections. Also it can spit out an HTML page for the specific graphs it created so a simple one line php page (included) can run the script and display the output.

	Note
	Joe: various people (including Francois Jeanmougin) have started sending patches to Salvatore.

17 Jan 2004

This new version allows you to graph connections to a specific VIP or realserver or VIP port or RS Port or any combination of those via command line options. It also adds in adds an option flip the graph for people with more inactive than active connections (you can have either the ActiveConn or InActConn plotted in the negative region below the X-axis). Also it can spit out an HTML page for the specific graphs it created so a simple one line php page (included) can run the script and display the output.

Joe - Jan 2004: lvs-rrd worked straight out of the box for me. You first install rrdtool from http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/ with the standard ./configure ; make ; make install. The rrdtool executables are standard ELF files (not perl scripts as I thought). rrdtool has the libraries it needs (zlib,

director:/usr/local/rrdtool-1.0.45/bin# ldd rrdtool
libm.so.6 => /lib/libm.so.6 (0x40017000)
libc.so.6 => /lib/libc.so.6 (0x4003a000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

gd) so you don't need any recursive downloading. Then you follow Salvatore's "Setup" instructions and you'll soon have gifs showing the activity on your LVS.

The filenames that Salvatore uses for his databases are derived from the ipvsadm (hex) information in /proc/net/ip_vs. Thus one of my rrd files is lvs.C0A8026E.0017.C0A8010C.0017.rrd representing VIP:port=192.168.2.110:23, RIP:port=192.168.1.12:23. You don't have to look at these files (they're binary rrd database files) and naming them this way was easier than outputting the IP in dotted quad with perl. Salvatore supplies utilites (which he grabbed off the internet) to convert the IP:ports between dotted quad and hex.

#/bin/bash
# from the internet, modified by Salvatore Tepedino and Joseph Mack
#
#IP is output in dotted quad
#run this in the directory with the lvs.*.rrd files.
#The files will have a name like
#lvs.C0A8026E.0017.C0A8010C.0017.rrd
#
#here's an example run
#   # ls
#   lvs.C0A8026E.0017.C0A8010C.0017.rrd
#   # ./ip_hex2quad.sh
#   192.168.2.110.0.23.192.168.1.11.0.23
#
#-----------------------
for file in lvs*rrd
do
        #echo $file
        IP=$(echo $(echo $file | tr a-z A-Z | sed 's/LVS\.//;s/\.RRD//;s/\.//g;s/\(..\)/;\1/g;s/^/i
base=16/' | bc) | sed 's/ /./g')
        echo $IP
done
#----------------------------

and

#/bin/bash
#ip_quad2hex.sh
#$1 is IP in dotted quad format
#converts dotted quad IP to hex

#------------------------
for i in `echo $1 | sed 's/\./ /g'`;do echo $i |awk '{printf "%02X", $1}';done;echo
#------------------------

Tore Anderson tore (at) linpro (dot) no 07 Dec 2003

There is also LRRD. Plugins for monitoring ipvsadm output are already included, for an demonstration you could take a look at "screenshot" pages at http://linpro.no/projects/lrrd/example/runbox.com/cujo.runbox.com.html

	Note
	Joe: Tore is one of the lrrd developers.

After getting Salvatore's code running, I reviewed the lrrd docs and tutorials to realise that there never was any hope of me understanding them without outside help. The docs are written for data coming from snmp, and I assumed that snmp was the only way of getting data. As Salvatore's code shows, rrdtool can use data from anywhere: if you can retreive/fetch/get your data in a script and send it as a parameter to rrdtool, then you can store and graph it with rrdtool.

taqu taqumd (at) gmail (dot) com 25 Nov 2008

Hi, I recently created another LVS graphing tool. This scripts parse ipvsadm -L -n --stats --exact and generate the following graphs: Project Home (http://sourceforge.jp/projects/lvs-stats) Screen Shot (http://sourceforge.jp/projects/lvs-stats/wiki/FrontPage)

cacti is another rrdtool based monitoring tool, which has been adapted for lvs.

Bruno Bonfils asyd (at) debian-fr (dot) org 26 Jan 2004

If there are some of you who running cacti in order to monitor LVS cluster, you'll probably interest by my xml data query and the associate template. Both are available on http://www.asyd.net/cacti/.

These images are to show that an LVS does balance the load (here number of connections) between the realservers.

Nagios is mentioned elsewhere in this HOWTO by various posters as a monitoring tool.

anon

I'm interested with LVS to do some load balancing with HTTP. I'm testing LVS with VMWare (i'm simulating two Windows 2003 real servers). Is there a way to do some load monitoring with windows realservers? I know the feedbackd project, but there's no win32 agent... If LVS cannot do Load monitoring I will use bigip or other proprietary solution that could handle load monitoring.

Peter Mueller pmueller (at) sidestep (dot) com 11 Jul 2005

You can try using the Nagios windows agents and some shell scripts to accomplish your goals. Two Nagios Windows programs that I am aware of are: http://nagios-wsc.sourceforge.net/ and http://nsclient.ready2run.nl/

Joe Sep 2011. The latest net-snmp-lvs-module from Robin Bowes robin (dot) bowes (at) yo61 (dot) com is

A MIB has been written for LVS by Romeo Benzoni rb (at) ssn (dot) tp (Nov 2001). It's available as

	Note
	Joe: there's a 64 bit problem with the net-snmp-lvs module that's been fixed. Make sure you're using at least the 0.0.4-2 module.

Jack Neely jjneely (at) ncsu (dot) edu 1 Oct 2008

I've setup net-snmp-lvs in an attempt to move the monitoring of my lvs/keepalived balancer into Cacti. I've gotten most things working and even have an RPM that folks might be interested in. I've built this on RHEL4 and 5 i386 (I believe it needs a bit of extra work for 64 bit arches.) http://install.linux.ncsu.edu/pub/yum/CLS/CLSTools.EL4/srpms/net-snmp-lvs-module-0.0.4-1.EL4.src.rpm

The values provided for the lvsServiceStatsInBytes attributes are not right. They don't seem to be bits or bytes. Cacti is graphing out my ssh service as doing terabytes of traffic per second. Running snmpwalk ; sleep 2 ; snmpwalk gives me the following:

SNMPv2-SMI::enterprises.8225.4711.17.1.13.2 = Counter64: 4927135724047695874 SNMPv2-SMI::enterprises.8225.4711.17.1.13.2 = Counter64: 4927141668282433538

Running ipvsadm -L --stats for the same firewall mark prints out the in bytes service counter at 9737M.

Laurentiu C. Badea (L.C.) lc (at) waat (dot) com 02 Oct 2008

I'm not sure why it's like this, but you just need to swap the two 32-bit fields around to get to the real value (which incidentally is the same as shifting >>32 when they are all zeroes).

Take for example 2670058611031409148 returned by SNMP for 2182G. In hex (easier to work with than binary) it is 250df429 000001fc, the real value is 000001fc 250df429 which is 2182465057833 or 2182G.

I just used the rates instead for the time being (just remember to change from COUNTER to GAUGE so cacti knows how it works).

I was looking at this last night. The problem was that ASN_COUNTER64 expects a "struct counter64" and is given a "long long" (__u64) instead. The attached patch should correct this problem.

Fix the bad values returned for the 64-bit byte counters.

--- lvs.c.orig  2006-01-02 14:31:54.000000000 +0000
+++ lvs.c       2008-10-03 18:43:48.000000000 +0000
@@ -131,6 +131,17 @@
        }
 }
 
+/** set a counter64 snmp response field from a __u64 value
+
+**/
+static
+void netsnmp_set_var_counter64_value(netsnmp_variable_list *var, __u64 *val){
+       struct counter64 val64;
+       val64.high = *val >> 32;
+       val64.low = *val & 0xffffffff;
+       snmp_set_var_typed_value(var, ASN_COUNTER64, (u_char *)&val64, sizeof(val64));
+}
+
 /** returns the first data point within the lvsServiceTable table data.
 
     Set the my_loop_context variable to the first data point structure
@@ -271,10 +282,10 @@
                                snmp_set_var_typed_value(var, ASN_COUNTER, (u_char *) &stats->outpkts, sizeof(stats->outpkts));
                                break;
                            case COLUMN_LVSSERVICESTATSINBYTES:
-                               snmp_set_var_typed_value(var, ASN_COUNTER64, (u_char *) &stats->inbytes, sizeof(stats->inbytes));
+                               netsnmp_set_var_counter64_value(var, &stats->inbytes);
                                break;
                            case COLUMN_LVSSERVICESTATSOUTBYTES:
-                               snmp_set_var_typed_value(var, ASN_COUNTER64, (u_char *)&stats->outbytes, sizeof(stats->outbytes));
+                               netsnmp_set_var_counter64_value(var, &stats->outbytes);
                                break;
                            case COLUMN_LVSSERVICERATECPS:
                                snmp_set_var_typed_value(var, ASN_GAUGE, (u_char *) &stats->cps, sizeof(stats->cps)); @@ -434,10 +445,10 @@
                                snmp_set_var_typed_value(var, ASN_COUNTER, (u_char *) &stats->outpkts, sizeof(stats->outpkts));
                                break;
                            case COLUMN_LVSREALSTATSINBYTES:
-                               snmp_set_var_typed_value(var, ASN_COUNTER64, (u_char *) &stats->inbytes, sizeof(stats->inbytes));
+                               netsnmp_set_var_counter64_value(var, &stats->inbytes);
                                break;
                            case COLUMN_LVSREALSTATSOUTBYTES:
-                               snmp_set_var_typed_value(var, ASN_COUNTER64, (u_char *) &stats->outbytes, sizeof(stats->outbytes));
+                               netsnmp_set_var_counter64_value(var, &stats->outbytes);
                                break;
                            case COLUMN_LVSREALRATECPS:
                                snmp_set_var_typed_value(var, ASN_GAUGE, (u_char *) &stats->cps, sizeof(stats->cps));

Jack

a src.rpm: net-snmp-lvs-module-0.0.4-2.EL4.src.rpm (http://install.linux.ncsu.edu/pub/yum/CLS/CLSTools.EL4/srpms/net-snmp-lvs-module-0.0.4-2.EL4.src.rpm)

Laurentiu

I would also create the necessary config and reload snmpd on install and uninstall, so it will "just work" immediately after install, something like this.

mkdir -p $RPM_BUILD_ROOT/etc/snmp
echo "dlmod lvs /usr/lib/libnetsnmplvs.so" > 
$RPM_BUILD_ROOT/etc/snmp/snmpd.local.conf

%files
...
%config(noreplace) /etc/snmp/snmpd.local.conf

%post
test $1 = 1 && chkconfig snmpd && service snmpd reload || true;

%postun
test $1 = 0 && chkconfig snmpd && service snmpd reload || true;

	Note
	If you can't access the snmp information (get timeouts, or snmpwalk crashes)

Michael Moody michael (at) gsc (dot) cc 24 Sep 2008

Depending on distribution, check /var/log/messages for more info. I had a problem where snmpd was running as "other than root", preventing the libnetsnmplvs from being able to access the ipvs tables.

Laurentiu C. Badea (L.C.) lc (at) waat (dot) com 12 Dec 2008

Build the net-snmp-lvs-module rpm posted earlier and install it on your servers running snmpd, then you can access the LVS data via SNMP in cacti. You can whip up a quick graph using the snmp-oid template, or you can make a more elaborate graph template that can be attached to hosts.

For example, the OIDs you need for bandwidth per service are:

bytes in:  1.3.6.1.4.1.8225.4711.17.1.13.i
bytes out: 1.3.6.1.4.1.8225.4711.17.1.14.i

where "i" is the service index (1...number of LVS services defined). You can use snmpwalk to find all the fields.

Jack Neely jjneely (at) ncsu (dot) edu 17 Dec 2008

In a similar vain...I'm using Cacti to graph services based on firewall mark. When a fwmark is removed I tell Cacti to reindex the LVS SNMP data so each data source is still groking the right part of the SNMP tree. This does that...but (today at least) also edited the path to the RRD for each data source to use the previous data source's RRD. I then became unhappy as I watched Cacti trash all my pretty graphs.

How do other folks handle this case when a service disappears from LVS with Cacti?

Ratz

The file linux/snmp.h represents the SNMP RFCs. IPVS is not specified in an RFC, so adding this has no chance I believe.

If you want to generate your own MIB, use one of the reserved sub trees of the MIB DB for such projects and peruse m2c. If you really plan on writing one, get back to us so we can sort out the header to freeze the API.

The simple approach we've been using for years:

If you need up to date values (not recommended though) you can also directly call shell scripts using the exec directive.

Joseph T. Duncan duncan (at) engr (dot) orst (dot) edu 21 Aug 2006

Presently I collect CPU, Memory in USE, and Network traffic statics from my windows terminal server "real servers" via snmp. I toss this information into an rrd database for making pretty graphs along with usage parsed from lvs stats. Finaly I take the CPU and Memory stats and use them to adjust my weight tables. My script duncan_main.pl for doing this is still in its infancy as I am getting stuff ready for this fall term, but it should be fun to see how it all works out. 28 Dec 2006: Here's an update lvs_weight.pl

Monitoring disks is not directly an LVS problem, however since disks are the most failure prone component of a computer, you need to have a plan to handle disk failure (I pre-emptively change out my disks at the end of their warrantee period, even if they're not giving problems).

Linux J. Jan 2004, p 74 has an article on the SMART tools for monitoring ATA and SCSI disks. Apparently for years now IDE and SCSI disks have been using the Self Monitoring, Analysis and Reporting Technology (SMART) standard to report low level errors (e.g. disk read errors, there's dozens of tests). This has been available in tools like Maxtor's PowerMax (for windows). (VAX's and Cray's continuously monitor and report disk errors - I've never known why this wasn't available on other machines.) The current SMARTv2 spec has been around since Apr 1996.

Apparently these SMART tools have been available on Linux for a while and run on mounted disks. The source code is at http://smartmontools.sourceforge.net/

There are two components,