Similar to the article posted here:

http://fusionsecurity.blogspot.com/2013/07/ive-talked-about-subject-of-long-lived.html

I am wondering if there's an open source software that can do the same. If a server is down, the traffic is forwarded to the other server without the client ever knowing it. If a server is back online, it can intelligently rebalance the active TCP connections.

Additionally, is there a way to insert a custom messaging protocol between the load balancer and the servers? For example, a client is in the middle of transfering data, and you need to do a blue-green deploy, thus shutting down some of the servers and keeping the others online. To prevent the data from being truncated, the LB can inform the servers that are about to be shutdown some sort of message to indicate that it needs to complete whatever requests it's currently doing. Then once the servers are done sending messages, it informs the LB back, and the LB then shuts them down. All new and existing connections are forwarded to the other servers.

Then once the deploy is completed, it does the same for the other set of servers.

This sounds like I would need a custom load balancer, but I want to make sure that I am not going to rewrite things that's been done. It sounds like a common problem.

Zen Load Balancer (Community Edition) (I've not used it)

I don't know of anything that supports the special scenario your describe though. We usually just disable a set of servers in the LB configuration, wait for those servers to go idle and then shut them down manually.

Note that "smarts" really needs to know about the specific protocol you're talking. For example, each server may want authentication for incoming connections -- or, of not, then the load balancer needs to deal with authentication, at which point it is a proxy more than a load balancer.

A good, open source, "straight TCP" load balancer is HAProxy. However, that doesn't "transpanrently" re-connect to other back-ends for a given connection, because doing so requires very detailed knowledge of the underlying protocol, and because each protocol is different (except for standards like HTTP) you will likely need to build those things yourself.

I've done something like this in the past. Simply use perls Stanford::DNS::Server module to create a simple DNS server listening on udp port 53 that is authoritative for your subdomain, and then use DBI to connect it to a database cluster.

Have each of your backend servers report a health metric to the database cluster periodically, this health metric can be simple as number of users or something more advanced.

Once you have this data, use it to determine what back end servers ip you give out for a DNS request, making sure the DNS result has a short ttl, perhaps a few seconds.

This worked fine for me and will work fine for you, just make sure you use a cluster of databases and more than one separate DNS server for reliability...

Also, you don't have to run the perl script as root - creative use of iptables can forward udp 53 to an unprivileged port easy enough.

Let me know if you need more info...