Load balancing at Github: Why ldirectord?

@DonMacAskill Nice! You might also want to check this article out if you have not already seen it. http://bit.ly/8uHacI

3 months, 1 week ago by _carumba on Twitter

Matt,

Nice Blog, I agree Ldirectord rocks… I too hated paying crazy money for naff hardware load balancers, so built my own using LVS, Ldirectord and Heartbeat. Then I thought he this is pretty good why not sell it… 7 years later Loadbalancer.org owns a small corner of the load balancer appliance market and we’ve added SSL termination, TPROXY, HAProxy, Nginx, feedback agents , SNMP…. all sorts of stuff that initially I couldn’t see the point in (Why doesn’t everyone use LVS in DR mode with Ldirectord?). Still the customer is always right (even when they are wrong). We are a little different ’cause we allow full root access and open source any changes we make. It makes me sad to admit but now that processors are getting so fast… full proxies (SNAT) like HAProxy are so easy to insert into your architecture they will probably make Layer 4 routers aka. LVS almost redundant at some point. Ps. Love the line ‘Never underestimate the value of knowing where the bodies are buried’, exactly why I never worry when a customer says the load balancer is broken….It’s never the load balancer (well mostly never). And here comes the cheeky load balancing link . ... See all content Hide content

4 months ago by malcolm on Wordpress

Fascinating load balancing post from github hosters, includes contribution from haproxy author http://is.gd/4NFvY

4 months, 1 week ago by zomoco on Twitter

Tyler,

If you just want a “regular” IPVS load balancer, that’s all you need. For a high-availability load balancer (if the active node dies, the other node in the pair will take over), then you need to add Pacemaker, and heartbeat or corosync, to the list of things to put together.

4 months, 2 weeks ago by Davy Jones on Wordpress

Interesting discussion of ldirectord VS haproxy here: http://bit.ly/VDhfp (in the comments)

4 months, 2 weeks ago by alexandermensa on Twitter

@tyterflint: you’re almost right. LVS (or IPVS, 2 names for the same thing) is the Linux kernel subsystem performing the load balancing. Ipvsadm is the utility used to configure LVS. It does not do much, it creates server farms, adds and removes servers, and reports statistics. Ldirectord as well as keepalived which was indicated above are daemons which monitor the servers’ health and add/remove them from the farms. All that chain of tools provide a nice and complete layer 4 load balancer. ... See all content Hide content

4 months, 2 weeks ago by wtarreau on Wordpress

This is a great article! I am currently architecting an alternate solution to our current web solution. We too are running 4 web servers and are currently using PF as a load balancer. The main issue with PF is that we can’t implement weighted load balancing…blech. 2 of our servers are substantially larger than the other 2, being the reason we need weighted round robin.

I mostly wanted to ask one question. The article is centered around ldirectord, and as I have been researching, ldirectord doesn’t seem to the the load balancing engine at all. Rather, ipvsadm appears to be the load balancing utility, and ldirectord is the monitoring utility that checks the health of the real servers.

Am I way off here? ... See all content Hide content

4 months, 2 weeks ago by tylerflint on Wordpress

Hi Matt,

first, rest assured that I did not take any of your comments as criticism nor attacks, and I’m not dicussing your choices, just about your perception of what proxies can do, nothing else.

Concerning the performance, it depends what you are doing. Direct-routing load balancers installed in one-leg configuration where they only see upstream traffic will always be faster than a proxy because they see about half the packets. Common figures are about 4 times haproxy’s max connection rate. A load balancer set up in the middle of the stream will still generally be faster than the proxy on connection rate, but can be slower on high bit rates (multi-gigabit), because it cannot make use of TCP speedups the NIC provides, while a proxy can. For instance, haproxy can forward 10 Gbps of HTTP traffic with only 20% CPU on one of my machines, while this machine has high difficulties to simply IP-forward the same stream at this rate. This is because of the TCP off-loading capabilities of the NICs which can only work with the proxy. But granted, this is not everyone’s needs !

Concerning queuing, well, I can tell you that this has saved quite a number of medium to large sites. Yes connections can sometimes pile up. But better accept them, parse them, reject wrong ones and get the good ones ready to be served than simply drop the SYNs and wait for the client to resend them 3 seconds later. Also, it is quite possible that your site has a very predictible load. But gaming sites, sports sites, online newspapers, stock exchange, etc… are extremely unpredictable and need to be prepared for 4-10 times the load without failing, even if some requests get slightly delayed. And quite honnestly, it’s very rare that the queue delay is counted in seconds. This happens when something breaks behind. But even there, having the ability to automatically flush requests out of the queue is nice.

Concerning QoS, well, it also depends on your usages. Gaming sites like to reserve resources for paying customers. Other sites with many external partners will prefer to prioritize some requests vs some others to the same partner so that if they become bandwidth-limited, the most important objects are fetched first. But doing that really does not cost much. It’s just a switching rule between two queues. You may drom from 15000 connections per second to 14980 maybe, this is not particularly noticeable.

Concerning the logs, I don’t agree with you. The web servers will tell what they do, not how it’s perceived. The front LB will see how it’s perceived and will be able to quickly tell you that server XX has higher response times than others or fails to accept a connection 1/1000th of the times, etc… This is very valuable information when people start to complain about performance issues, and it’s even better when you can compare what the LB sees with what the server says. Most often, the difference is in (too short) system queues on the server itself (small TCP backlog, etc…) that can’t even be detected by the server software because the information is dropped before reaching it. That’s also the only place you can detect the famous 3-second stairs indicating some packet loss.

Overall, all a proxy can bring is a tradeoff, adding more intelligence in the decisions taken by the load balancer at the expense of an increased resources usage on one cheap box. If the smart decisions can improve the site’s responsiveness, reliability or scalability, that’s fine. If they don’t bring anything, it’s not a place for a proxy. That’s why you generally see them in front of HTTP servers, sometimes in front of mail servers, database servers and terminal servers, and rarely in front of anything else.

Most likely in your case, as you explain it, it would simply not bring anything for your current usage (and I have no problem with that). And I agree with you that with only 128 MB of RAM it’s hard to run a proxy with large numbers of connections, considering that the system itself will take 16 kB minimum per connection ! ... See all content Hide content

4 months, 2 weeks ago by wtarreau on Wordpress

Ldirectord does indeed rock. So I’ll just agree.

I do have something to add, though:

I have experience setting ldirectord up to replace a keepalived setup, because for some reason the keepalived process would go catatonic and drop all states in the kernel. Only a kill -9 and restart would cure it. Very unsatisfying.

So I’m not a particularly big fan of keepalived , but, that was 4 years ago. It could’ve been fixed by now, I don’t know.

Another thing I like about ldirectord is the fact that it does one thing well, then you have something like Heartbeat from Linux-HA to make the highly available part. Heartbeat does that very well.

And if the ldirectord process dies it just leaves the kernel IPVS system in whatever state it was in at the time of death, unlike my experiences with keepalived.

One bad thing about ldirectord: there’s a memory leak if you use negotiate checks on HTTPS services, and you’ll have to restart it every once in a while. This is easily done without interruptions, you just have to do it. ... See all content Hide content

4 months, 2 weeks ago by Morten Liebach on Wordpress

Yeah, Matt lost me here (in his reply to Willy Tarreau's [HAProxy] comment.) Packet rewriting is nice and efficient, but it's dumb. Unless you measure a latency/cpu need, in a system with lots of moving parts, having high-level ways to control the flow of traffic -- on-the-fly -- is a good thing.

4 months, 2 weeks ago by jeremyw on Hackernews

@codemonkeyism regarding haproxy question: http://bit.ly/2zqyId

4 months, 2 weeks ago by timfaas on Twitter

"One geek's opinion on load balancing" http://bit.ly/VDhfp Not sure what L4 appliance was using, but was obviously not right one.

4 months, 2 weeks ago by lmacvittie on Twitter

Social comments and analytics for this post…

This post was mentioned on Twitter by FastestFood: Load balancing at Github: Why ldirectord? | Anchor Web Hosting Blog http://tinyurl.com/y9f26w3…

4 months, 2 weeks ago by uberVU - social comments on Wordpress

Being the author of haproxy, I can’t agree with all your points against proxies, especially

the points about performance. 2500 connections per second is small for haproxy. This

is what it supports on my 5-watt Geode 500 MHz (it does about 2700 in fact). Some sites using more common hardware (core2duo, dual-opteron, …) and properly setup systems regularly see loads between 15000 and 25000 connections per second.

Using a proxy for load-balancing provides two very important features for scalability :

– content switching, which means dedicating servers to some tasks

– request queuing, to limit the number of servers required to sustain traffic surges

The first point, content switching, allows you to dedicate two fast web servers to static contents. Typically nginx or lighttpd. You don’t need more than two such servers in general, as both are capable of filling a gigabit pipe. Then you only have application servers. Not only you can decide to spread your site across multiple technologies, but you can ensure that you only have to add application servers when the existing ones get close to 100% usage. Such architectures ensure you use the best tool for the job.

The second point, request queuing, saves you from having to start large numbers of servers to support traffic peaks, and from rejecting users during such peaks. Requests are simply processed at full throttle without ever surpassing the server’s capacity. If the servers are saturated, the only effect is a slowdown for the users, which is generally not perceivable (a few milliseconds). But the real advantage of queuing is to set up QoS. It’s really nice to be able to always dedicate ressources to some categories of users (eg: the ones already logged in, the ones who’re paying for premium service, etc…). A proxy-based load-balancer can do that because it can parse a full request while an L4 LB can’t.

A third point which is less important but not to be neglected is security. Only really valid HTTP requests pass through a proxy. Half-openned connections, SYN floods, slowloris and all similar attacks simply never reach the servers. And if you identify an attack pattern, you add an ACL for it and block it. BTW, on a proxy, ACLs are extremely cheap because they are only evaluated when the request is available, not for each packet.

Of course, detailed logs are a major help with a proxy and LVS cannot provide that, nor can it provide timers nor response errors statistics. But almost everyone has to be beaten first to appreciate logs

Now, concerning the other points about non-HTTP protocols like SSH, I agree with you. If the proxy has no added value, let’s not install one on such protocols. Haproxy can work transparently (connect to the server with the client’s IP) on some Linux kernel versions, but the set setup is not easy and it generally is not worth it for such protocols.

Also, if you’re serving more than 1 Gigabit/s of traffic, using a proxy will require 10 Gbps NICs which are still a little bit expensive, while with LVS you can simply do DSR and only see the smaller forward traffic.

I still have a now-old (though not outdated yet) article on my site about scalable load-balancing architectures, for those interested : http://1wt.eu/articles/2006_lb/

That was another geek’s comments

Willy ... See all content Hide content

4 months, 2 weeks ago by wtarreau on Wordpress