Interesting analysis of linux kernel threading by IBM

[Mark Hahn]

> > > bigger thing for Alpha than for Intel).  On our newer systems we fully expect
> > > hundreds, if not thousands, of tasks.  The more commercially accepted Linux
> >
> > the issue is *runnable* tasks.  do your machines routinely report
> > loadaverages of 1000?  if so, I'm impressed!
> >
> 
> Yes, I did get that. :)  And yes, they do routinely report such load averages.  As
> a matter of fact, I was just stress-testing a system and when I looked the load
> average was 2002.  Granted the test I was running (AIM) is artificial (all tests
> of its ilk are), it was designed as a representative measure of system
> performance.  Representative in that it represents what systems actually do.

AIM is certainly NOT representative of what most systems do.
many of us have administered and used no many overloaded multiuser
machines, and they never look like that.  it's perfectly reasonable
for Linux to say that if you are among the .01% of machines that have
runqueues that long, try this patch to sched.c...


> Do we here have systems with that high a load average from actual use?  No - but
> we're not running any massive databases or web servers either.  Do I have hard
> proof that our customers do that?  I doubt anyone would tell me if they did anyway
> ;)

without proof that this is an issue, it's not an issue.
seriously, I have a hard time thinking of why a massive DB or webserver
would ever have large numbers of runnable processes (not blocked on IO.)


> Point is, it's possible, and it's becoming more and more probable every day.  (I
> had a co-worker testing a system and he gave up at a load average of somewhere
> around 16000 because he didn't want to wait any longer.)  Remember - this is the

high loadaverages are a sign that something's wrong.  servers should be 
IO-bound.

> age of server consolidation - more stuff on fewer systems.  Lots of little piddly
> things on tiny boxen scattered here there and everywhere was the NT way and it has
> proven to not work (on a large scale at the very least).

this is the age of clustering, actually.  please, let's not talk ourselves
into reliving the horrors of mainframes.


> > the issue here is whether someone can come up with a maintainable
> > scheduler that has the requisite performance.  since the runqueue is
> > normally short, the scheduler's performance function must have a
> > very small constant term.  if it's true that there are applications
> > that result in long runqueues, then the performance curve needs to
> > be as flat and horizontal as possible, again, without degrading the
> > constant term.
> >
> 
> Agreed.  That's one of the reasons that Tru64 has a per-cpu runqueue backed by a

per-cpu scalability is very much a good thing.  I expect 2.5.[1-4] will
contain massive per-cpu patches.  but this doesn't imply that the scheduler
should be optimized for many runnable tasks!


> > AFAIK, loopback volanomark does not resemble _any_ real application.
> >
> 
> No, probably not technically.  It just means that they didn't have to configure a
> big system with a whole bunch of clients and enough bandwidth to put the same type
> of load on the system.  With the advent of multi-hundred MB/s plus Internet
> connections and massive Intranet requirements, such bandwidth isn't impossible to
> imagine.

the issue is that volano consists of many threads, each blocked only by
talking to other threads.  if any real app were designed like this,
you'd stream "STUPID!" into the face of its designers.  real apps sometimes
use many threads, but those threads wind up blocked on IO most of the time.
similarly, big multiuser systems are normally blocked on brain IO.

rather than prematurely optimizing the scheduler, which seems to run
just fine for real loads, it might be interesting to look at scaling
_wakeup_, that is when many threads are going into or coming out of 
an IO sleep.  *that* is something that real, big machines might need.

regards, mark hahn.


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo en vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/