Complicated websites can be hard to run under LVS (e.g. websites with servlets). The problem is that the website has been written with the assumption that all functions on the website will always be together on the same machine. This was a reasonable assumption not so long ago, but now with customers wanting high availability (and possibly high throughput), this assumption is no longer valid. The assumption will be invalid if the client writes to the server (see Section 34.28) or if the server maintains state information about the client (see Section 19.9.1. Often people setting up an LVS hope that LVS can look inside the packets for content and direct the packet to an appropriate realserver. LVS can't do this. In all cases, this simplest, most robust solution is to rewrite the application to run in a high availability environment. Administratively this sort of proposal is not well received by management, who don't like to hear that their expensive web application was not properly written. Management will likely be more receptive to canned solutions from glib sales people, who will tell management that an L7 loadbalancer is a simple solution (it is, but it is also slow and expensive).

Roberto Nibali ratz (at) tac (dot) ch 19 Apr 2001

LVS is a Layer4 load balancer and can't do content based (L7) load balancing.

You shouldn't try to solve this problem by changing the TCP Layer to provide a solution which should be handled by the Application Layer. You should never touch/tweak TCP settings out of the boundaries given in the various RFC's and their implementations.

If your application passes a cookie to the client, these are the general approaches:

Andreas J. Koenig andreas.koenig (at) anima (dot) de 2001-06-26

Related to the concept of persistent connection (whether implemented with LVS persistence or any other method) is the concept of single session. The client must appear to have only one session, as if the server is one machine. You must be able to recognise the client when they make multiple connections and data written on one realserver must be visible on another realserver. Also see distributed filesystems.

K Kopper 7 Jun 2006

Let's say you are running Java applications (aka Java Threads) inside of a Java container (virtual machine) you should be able to tell the container itself how you want it to store session information (like a shopping cart). The method of storage can therefore automatically make the session information from one cluster node (real server) available to all cluster nodes via a file sharing technique, multicasting to all the nodes, or by storing data on a database server (on a backend HA pair). If you are five pages deep into a shopping cart, for example, and the real server crashes it won't be a problem if you land on a new real server with your next click of "submit" and it can pull up your session information.

Check out 7-2 (page 64) of this document for the Oracle approach: http://download-west.oracle.com/otn_hosted_doc/ias/preview/web.1013/b14432.pdf

Or for the JBOSS way using multicasting via JGroups: http://www.jgroups.org/javagroupsnew/docs/index.html

Building an Oracle OC4J container that is highly available on the HA backend to store session information for a cluster works and seems like a good sound approach to me. The multicast way raises many doubts in my mind (especially if you need to lock the session information for any reason).

Dan Baughman dan.baughman () gmail ! com 2006-05-09

Internally, lvs must make decisions about where to send what connection request, right? I want to implement a hash table to keep track of where previous connections from an ip went, and send them to the same server. Any sort of timeout is optional. Once an ip gets a server it can always get that same server. I had previously thougth of giving the session a timeout, but now I'm leaning towards just having it maintain the hash forever, and I'll just restart the director deamon every night at 2 am (or never). Basically we are going to cluster some cold fusion servers and I don't want to pay the 10 grand Adobe wants for an enterprise license to do what we want. We have a lot of app's deployed that use a cookie stored on the client in conjunction with the user's ip to access server side session data. To reimplement the apps we've deployed to access a db instead of the session data would be considerable. Looking at the persistent option, that seems to be exactly what I'm looking for.

K Kopper karl_kopper (at) yahoo (dot) com 6 Jun 2006

To share files on the real servers and ensure that all real servers see the same changes at the same time a good NAS box or even a Linux NFS server built on top of a SAN (using Heartbeat to failover the NFS server service and IP address the real servers use to access it) works great. If you run "legacy" applications that perform POSIX-compliant locking you can use the instructions at http://linux-ha.org/HaNFS to build your own HA NFS solution with two NFS server boxes and a SAN (only one NFS server can mount the SAN disks at a time, but at failover time the backup server simply mounts the SAN disks and fails over the locking statd information). Of course purchasing a good HA NAS device has other benefits like non-volatile memory cache commits for faster write speed.

If you are building an application from scratch then your best bet is probably to store data using a database and not the file system. The database can be made highly available behind the real servers on a Heartbeat pair (again with SAN disks wired up to both machines in the HA pair, but only one server mounting the SAN disks where the database resides at a time). Heartbeat comes with a Filesystem script that helps with this failover job. If your applications store state/session information in SQL and can query back into the database at each request (a cookie, login id, etc.) then you will have a cluster that can tolerate the failure of a real server without losing session information--hopefully just a reload click on the web browser for all but the worst cases (like "in flight: transactions).

With either of these solutions your applications do not have to be made cluster-aware. If you are developing something from scratch you could try something like Zope Enterprise Objects (ZEO) for Python, or in Java (JBOSS) there is JGroups to multicast information to all Java containers/threads, but then you'll have to re-solve the locking problem (something NFS and SQL have a long track record of doing safely). But you were just asking about file systems and I got off topic . . .

Christian Bronk chbr (at) webde (dot) de 02 Jun 2006

As long as you want AOL customers on your site, you will need single session server for your cluster (any sort of database will do). Every request from AOL comes from an different proxy-IP and even setting a persitence-netmask will not fix that.

malcolm lists (at) netpbx (dot) org 02 Jun 2006

The SH scheduler gives exactly the same kind of response as persistence and it's layer 4 based on source hash... Their are hundreds of session implementations for web servers, it's one of the first things web programmers should learn (i.e. INSERT INTO sessiontable.....) LVS doesn't do L7 because L7 should be done by your app (i.e. that's what L7 is for.)

Martijn Grendelman, 2 Jun 2006

I couldn't get the SH scheduler to work (at the time not understanding the weight parameter) and I set up an Msession server for "session clustering" and used the RR scheduler. This setup works perfectly and is still in use today. However, since Msession is hopelessly outdated, and its successor (Mcache) doesn't seem to get off the ground, and I haven't found any workable (open source) alternatives, I would really like have another look at LVS persistence of some sort.

Martijn Grendelman martijn () grendelman ! net 2006-05-10 12:09:28

A centralized session manager would be nice, but I for one haven't been able to find a decent solution for use with PHP. I don't know about other systems or APIs. Msession is dead (I still have it running, but any attempt to build a stable daemon on an up-to-date system failed time after time) and its successor (MCache) seems to have died before it even got to beta. Other projects I have looked at (http://www.vl-srm.net/, for example) are also dead, or just aren't suitable (Memcached). I use Msession for a shared hosting cluster. The main advantage is, that Msession has a PHP extension, so it doesn't require any PHP client code. I need this, because I don't want to implement any dirty hacks based on "auto_prepend_file" or something like that, which I would need if I'd put my sessions in MySQL. Well, you can always buy Zend Platform, which features Session Clustering, but some of us don't have the $$$. Any suggestions for alternatives?

mike mike503 (at) gmail (dot) com 6 Jun 2006

IMHO storing data in blobs is a horrible idea.

If you are coding an application, I'd suggest checking out MogileFS. If this is for general purpose web hosting, where you need a normal POSIX filesystem to access, then that won't do. But for applications, it seems like a great idea (and from what small amount I read about the Google FS, it actually has a couple of the same traits)

As far as session management, a central session manager such as msession would work, or just roll your own off a database - it's simple in PHP (that is what I do) - then use DB failover/replication/etc. software to handle the DB clustering/failover.

mike mike503 () gmail ! com 2006-05-09

not sure how long you want to track the information, but you might be able to handle this with iptables and firewall marks. then you can group requests by any sort of iptables-configurable tracking (by port(s), ip ranges, etc...) - also i think there's a persistence configuration option in ldirectord (or is it keepalived, i always get them confused - or maybe both.)

I don't understand the need though for session persistence like this; I'd expect a centralized session manager (msession for instance) or just using a central database for the information would suffice. that's how I've been doing it, not sure why everyone has all these unique requirements to make sure they can persist sessions across IP addresses and AOL proxies and such. seems overkill, I've never had a problem.

mike mike503 (at) gmail (dot) com 10 May 2006

It's very simple to make your own which uses only a single database table in mysql. I used to use msession, but it had some overhead it seemed like, and a database-driven one was less "thick" - the other good thing about writing your own session handler is that you can call other things on session start or close, etc. I'd suggest using that (a mysql one) I'll try giving you mine though, also:

### SQL:

CREATE TABLE `session` (
 `ID` varchar(32) character set utf8 NOT NULL default '',
 `uid` int(10) unsigned NOT NULL default '0',
 `data` mediumtext character set utf8 NOT NULL,
 `accessed` int(10) unsigned NOT NULL default '0',
 PRIMARY KEY  (`ID`),
 KEY `accessed` (`accessed`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

### cron script (run it every so often, i have it set to 5 minutes -
otherwise there is no garbage collection - it will remove sessions
that are over 1800 seconds old.

$expiry = time() - 1800;
db_query("DELETE FROM session WHERE accessed < $expiry");

### the session PHP functions

function session_close() {
       return true;
}

function session_die($id) {
       db_query("DELETE FROM session WHERE ID='$id'");
       return true;
}

function session_gc($maxlifetime) {
       return true;
}

function session_open($path,$name) {
       return true;
}

function session_read($id) {
       $dchk = db_query("SELECT data FROM session WHERE ID='$id'");
       if(db_numrows($dchk) == 1) {
               if(!isset($_SESSION['row'])) { $_SESSION['row'] = 1; }
               list($data) = db_rows($dchk);
               return base64_decode($data);
       } else {
               return "";
       }
       db_free($dchk);
       return true;
}

function session_write($id,$data) {
       $data = base64_encode($data);
       if(!isset($_SESSION['row'])) {
               db_query("INSERT IGNORE INTO session
(ID,data,accessed) VALUES('$id','$data',UNIX_TIMESTAMP(NOW()))");
       } else {
               db_query("UPDATE session SET
data='$data',accessed=UNIX_TIMESTAMP(NOW()) WHERE ID='$id'");
       }
       return true;
}

### configuration in each script (common include)

ini_set("session.use_only_cookies","1");
ini_set("session.gc_probability","0");
ini_set("session.cookie_domain","yourdomain.com");
session_set_save_handler("session_open", "session_close",
"session_read", "session_write", "session_die", "session_gc");
session_start();
register_shutdown_function('session_write_close')

Alex Kramarov alex-lvs (at) incredimail (dot) com 22 Nov 2002

Microsoft's COM model is similar to the CORBA model. Generally, you have components, i.e. code that can be used from other applications. The concept is similar to using shared libraries, but still a little different. You can create an instance of the component and use in a simple fashion with asp (the IIS scripting language IIS). Every time a new user calls for a asp file, a new session component is created, and can be accessed through asp scripting. This component can store data like a perl hash (session(valuename) = value). The data is stored in the memory space of the IIS process. Each session has a unique identifier that is remembered along with the data, and this identifier is maintained during the session by a cookie. On subsequent access by a client, the server looks up the data stored for this session, and makes it available as members of the session component.

a simple sample - access autorisation (this code goes on top of a pages you would like to secure):

' check if this user has identified already

If Session("UserID") <> "" Then
 'check some conditions.
 'if check successful
    Session("UserID") = "approved"
 'else do not allow to proceed showing the page
    response.write "authorization failed"
    response.end
End If

There are components available that will replace the default session component with one that will store the session data in a shared db, and only minimal modification to the code are required, if any. Generally the session component is an implicit component the server provides. You could use your own component that does the same thing, and the only thing you would have to do is to initialize an instance and give it the unique identifier of the user, like this (purely fictional code)

mySession = CreateObject("my.own.session.component")
mySession.setUniqueId = server.request("some data, a cookie or other parameter").

When writing code for MS servers, one almost never deals with files, since the interface provided by MS for that purpose is very cumbersome, and on Windows, file locking problems are very severe issue. With Unix, you can write a cgi that manipulates files, reads and writes to and from a dozen files while running. You would be crazy to that on windows. All data you want to store can be more or less conveniently stored using the session object if this is a per user data, or in the application object (more or less the same idea), which retains data through the life of the application (from the start to the stop of the http service). All data is in memory, hence, it is fast. Long term data is always stored in databases. I believe that the difference in perspective comes from the fact, that in unix, you can have a bare bones system because of your security requirements, and then you want to write a small script that uses and stores some data, so you open some files and do that. On windows, you CANNOT HAVE a bare bones system. From the initial install, you already have some file based db structure (comparable to db3), and all the database connectivity libraries, which you cannot remove, unless you are a windows guru and you start deleting system libraries one by one. (You would be crazy to do that, since there is absolutely no documentation what each of the thousands of library files, which are installed by default, do.) All these libraries are a security risk, as is proven by all the buffer overflow vunerabilities. But since windows developers regard DB connectivity as a standard component of their OS, they use it. (This is a marketing strategy of MS, to sell their MS bloated sql server).

why doesn't the application doesn't keep it's own state?:

IIS assigns a unique identifier to be used in session management the first time a user accesses an asp file (even if you need it for only 1% of the pages on your site). This is completely transparent to the developer, and saves time in the development process. Writing apps where state is conserved manually (without sessions), is not as easy as it looks, and the mechanism provided by IIS is certainly convenient. Coding using "the Microsoft Way" for IIS took me 4 hours to learn, going through microsoft developer network articles. It is simple if you don't stray from the dictated path, but the second you do stray, it's hard to push something not designed by MS into their framework, and people are afraid of that.

IIS 6 includes the option to make the server session object store data in an odbc database, but it is still not released. 3rd party components that should do the task are commercially available, like the frameWerks session component, and it is pretty cheap, at 149$.

I also believe that not a lot of sites need and use several realservers to serve a simple logical site, so this was never such an issue, till recently. Now, microsoft woke up to the fact and writing their own implementation for the IIS 6, which will undoubtedly require the use the MS sql server.

Mark Weaver mark (at) npsl (dot) co (dot) uk 23 Nov 2002

The default ASP (= MS web scripting) sessions simply use a cookie and stores session state in server memory. The session is a dictionary object, and you just store a bunch of key-value pairs. Since the standard session object stores data in memory, it is not a lot of use for /robust/ load balancing.

.NET adds a component that stores session in a database. Such a component is pretty trivial to write, we have had one for a number of years. Storing on disk is a good option when there is no database, but since most of the sites that we have are pretty dynamic (i.e. most pages are generated from database calls), storing the session state in the DB is a good bet. I can probably release the source code for this if anyone is interested.

You can setup persistence several ways

One of the problems with persistence is removing a service (e.g. you just want it removed or the realserver has crashed). Even after the weight has been set in ipvsadm to 0, the service is still in the ipvsadm table and will stay there till the end of the client's timeout period. If the realserver has crashed, the client's connection will hang. You would like to have preserved the client's state information in your database, and give the client a new realserver. This problem has now been addressed with the LVS sysclt (see sysctl and bringing down persistent services). For older material on the topic see realserver crash on sticky connection.

The following examples here use telnet and http. You aren't likely to want to make these persistent in practice. They are used because the clients are simple to use in tests. You'll probably only want to make ftp or https persistent, but not much else.

Setup persistence on VIP, default persistence timeout (default timeout value varies a bit with ipvs versions, but it's about 10mins), port not specified (all ports made persistent). Telnet'ing to the VIP from one machine, you will always connect to the same realserver.

director:/etc/lvs# ipvsadm -A -t lvs -p
director:/etc/lvs# ipvsadm -a -t lvs -r rs1 -g
director:/etc/lvs# ipvsadm -a -t lvs -r rs2 -g
director:/etc/lvs# ipvsadm
IP Virtual Server version 0.9.4 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  lvs.mack.net:0 wlc persistent 360
  -> RS2.mack.net:0               Route   1      0          0
  -> RS1.mack.net:0               Route   1      0          0

Here's setting up with a specified persistence timeout (here 600secs), setting persistence granularity (the -M option) to a netmask of /24, and round robin scheduling.

	Note
	If you make the timeout > 15mins (900 sec), you'll also need to change the tcpip idle timeout.

director:/etc/lvs# ipvsadm -A -t lvs -p 600 -s rr -M 255.255.255.0
director:/etc/lvs# ipvsadm -a -t lvs -r rs1 -g
director:/etc/lvs# ipvsadm -a -t lvs -r rs2 -g
director:/etc/lvs# ipvsadm
IP Virtual Server version 0.9.4 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  lvs.mack.net:0 rr persistent 600 mask 255.255.255.0
  -> RS2.mack.net:0               Route   1      0          0
  -> RS1.mack.net:0               Route   1      0          0

Note: only a timeout value can follow "-p". Thus you can have any of

director:/etc/lvs# ipvsadm -A -t VIP -p
director:/etc/lvs# ipvsadm -A -t VIP -p 600
director:/etc/lvs# ipvsadm -A -t VIP -s rr -p

but you can't have

director:/etc/lvs# ipvsadm -A -t VIP -p -s rr

You can setup persistence by port

director:/etc/lvs# ipvsadm -A -t lvs:80 -p
director:/etc/lvs# ipvsadm -a -t lvs:80 -r rs1 -g
director:/etc/lvs# ipvsadm -a -t lvs:80 -r rs2 -g
director:/etc/lvs# ipvsadm
IP Virtual Server version 0.9.4 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  lvs.mack.net:http wlc persistent 360
  -> RS2.mack.net:http            Route   1      0          0
  -> RS1.mack.net:http            Route   1      0          0

We spend lot of time telling people not to use cookies to maintain state. I thought I should do a reality check, to see what people are using that's working.

Malcolm Turnbull malcolm (at) loadbalancer (dot) org 12 May 2004

I've been involved with a mid sized ecommerce company for about 4 years and we've had very few problems using cookies for state (stored in a single backend db.) Its fast and easy. If the odd customer doesn't have cookies turned on then they are no great loss.

Putting the sessionid in the URL i.e. GET is ugly and slightly less secure. I guess you could POST it on every page but would that be slower than cookie? (I think so)

	Note
	Joe: Having session data in the URL allows the user (or man in the middle) to manipulate it. This is not secure.

Joe 11 May 2004 14:23:02 -0400

With the security hazards of cookies, I have them turned off. Usually the application (e.g. mailman) runs a cookie test on me and tells me to turn on cookies. I've been trying to register for the Ottawa Linux Symposium for about 2months, and they've been having trouble with their registration software. Finally they ask me if I've got cookies turned on. I say "of course not". They don't do a cookie test and there's no notice on the web page that I need cookies turned on.

Jacob Coby jcoby (at) listingbook (dot) com 11 May 2004

We aren't an ecommerce site, but we do require some sort of login/authentication to use our site.

We haven't worried about cookies either, at least, until the past 4 months or so. It seems that the latest and greatest anti-virus software and pop-up blockers disable cookies (among other things that they have no business doing). Rumor has it that new versions of IE will disable cookies by default. A good portion of IE6 users won't accept cookies unless your site publishes a P3P of some sort. We get appx. 4000 people/day (9000 logins/day), and we were getting up to 10 cookie related problems a day to the helpdesk. I'd estimate that there were at probably 2-10x more that who had problems, but who never reported it. In 3 years of requiring cookies, we had only one nasty email about our requirement to have cookies enabled.

At any rate, we now use a different system to autheticate a user. We pass in a sid per page, and use cookies, IP address, browser ident, and other metrics to authenticate the user. Sensitive areas of the site (such as those requiring a credit card) also use SSL. All session data is stored in a single database, as a serialized PHP array. There can be up to 1/2 MB of session data, and part of the session data persists between logins, so it doesn't make sense for us to put session data in the cookie or to store it on the webservers.

The sid + (cookie, IP, browser ident) is only used to authenticate the user. The session data itself stores all sorts of things, such as temporary user prefs, some of the things the user looked at, a bit of caching to cut down on subsequent db queries, things like that. Only part of that session data persists between logins, but it has to be stored somewhere between pages.

Our situation is a little different from your average e-commerce store. We can't just identify a shopping cart + items by a unique sid. We need the session data to act as a ram drive of sorts for data that needs to be quickly accessed, multiple times per page.

All of that temp data is stored in an array, and serialize()'d. PHP's serialize is pretty compact, but it still expands out. For example, a simple int value of the login timestamp looks like:

s:8:"login_ts";i:1084802982;

The 1/2mb is the MAXIMUM we allow to store. Typical is more in the 3-10kb range. Average size over 49627 rows of session data is 3134b right now.

Joe: What's going to happen to your session data when IE6 disallows cookies?

It'll still work. The sid cookie is only one of several hints we can use to authenticate the user.

Malcolm Turnbull malcolm (at) loadbalancer (dot) org 17 May 2004

If your page is formated correctly with a PICS-Label IE6 will accept the cookie by default.

Note

For reference info about PICS Labels see PICS Label Distribution Label Syntax and Communications Protocols v1.1 (http://www.w3.org/TR/REC-PICS-labels-961031). For a sample HOWTO see How To Label Your Pages with PICS Meta Tag (http://256.com/gray/docs/pics/). These labels appear to be part of the so far futile effort to filter webpages for children. Here's a webpage by the people fronting this effort Information for webmasters. They want you to rate your own website (people with obnoxious content are always honest, right?). If the ICRA expect this approach to succeed, then why do we have spam? The politicians are no help of course. One of them has a bill to stop google from inserting advertisements into their gmail service, since this requires reading people's (private) e-mail. This bill would also stop programs from filtering web content. We have a long way to go.

POST is marginally slower than GET if you look at the HTTP spec. There is an additonal request header per variable. GET is only *very slightly* less secure. POST, and cookies are of equal security levels, and they're all trivial to send using command line tools.

Joe

until recently I'd thought that putting the session data into the URL (rather than a cookie) was the way to go, till someone pointed out that the user could manipulated the URL. In that case, could the session id be put in a long enough string in the URL such that any attempt to alter it would result in an invalid string?

Jacob

There is an upper limit on the GET string IE will send. Somewhere around 1 or 2 kb. When it hits the limit, if you use javascript to submit the form, it'll error out. If you just use a

<input type="submit">

it'll just not work. For sites that will never hit that limit, passing in the session data would work. However, there should still be checks to authenticate the user, mostly to prevent problems when they share links with friends. One solution to the user modifying the string is to pass in a public key:

Kpu = public key Kpr = private key md5 = standard MD5 sum algo session = your serialized session data Kpu = md5(md5(session + Kpr) + md5(Kpr)) (or some variation, see the HTTP RFC for an example with the HTTP DIGEST authentication)

Then you can authenticate that the session data hasn't been modified by checking your computed Kpu against the Kpu that was passed in from the GET/POST. If they match, the session data probably hasn't been modified. If they don't, there is a very good chance the data was either corrupted in transit or corrupted by the user.

It's only as strong as the Kpr used and whatever the collision probability of the md5 algorithm.

Horms 21 May 2004

Is there anything to stop a cookie or post from being manipulated by an end user? Sure, it might be margionally more difficult as you would probably need some (command like) tool, rather than just changing the URL directly in your given browser. But it should still be trivial. I rarely write web pages. But if I was to do something like this I would make the string in the URL a hash (md5, sha1 or something like that), which should make guessing a valid string rather difficult. I would do the same in a cookie or a post. I would imagine something like this is pretty common practice.

nick garratt nick-lvs (at) wordwork (dot) co (dot) za 12 May 2004

This discussion happens every so often on the list and, as always, I feel the need to mention the msession session service which we have been using very reliably for years from Mohawk Software (http://www.mohawksoft.com/devel/msession.html). It's light-weight, fast and depending on the scripting language you use (we're using php4) it is very easy to implement.

Matthew Smart msmart (at) smartsoftwareinc (dot) com 25 Jun 2007

The problem I have is that all clients from a given location get directed to the same realserver. Since the majority of clients are located in the same office, we are not getting a good load balance. I disabled persistence, moved sessions into mysql, and am relying on mysql's replication to ensure that all servers have the session data. We have state info stored server side in a PHP session. Client side there is a cookie that holds a session id only (no state). We are working on ways to replicate the server side session info across N real servers. I think relying on mysql will work in the short term. Just have to test it under load to see how it behaves. I can see an issue if mysql replication gets behind on a server, but that is not an LVS issue...