Auto-Adaptive scheduler - Final chapter ( the numbers ) ...

[Jamie Lokier]

Larry McVoy wrote:
> : _If_ we see that low RQ applications don't switch much, then we can
> : assert that low RQ applications aren't adversely affected by the patch.
> 
> Huh?  What about I/O servers?  FTP, Web, etc?  Are they not low RQ and 
> frequently switching applications?  Yes, they are.

If they are frequently switching, there must be several processes to
switch between and there must be more than one runnable at a time.  So I
assume you mean the classic "one process per connection" FTP server and
the classic "spawns CGI scripts" web server.

I'm not sure where the run queue values tend to be in that case.  If RQ
< 1, then the switching is to/from the idle task, in which case idle
time can absorb switching cost (at some latency cost).  If 1 < RQ < 2,
is there really that much switching going on?  Etc.

I simply don't know if you get servers like that switching a lot and
having short RQs at the same time.  Switching a lot yes, small RQs yes,
but do they both happen at the same time?

Those would go in the "badly designed for performance" category
according to the classification used so far on this thread.  ( They can
be better optimised for performance, it is said, by making them
single-threaded.  Or generally, optimised is N threads where N approx. =
NR_CPUS. )

> : The patch is to improve the performance of applications that _aren't_
> : written in the best way for performance.  Because changing the
> : applications is too difficult -- changing the kernel is not.
> 
> With all due respect, your statement is unsupportable.  Consider that
> there are many bad applications with _different_ forms of badness, some
> of which are mutually exclusive.  If you optimize for any of them, we can
> always find another one that just got hurt.  Now what?

Optimise for that one too :-)

I think the point is to optimise for the important applications.  If big
Java servers are the only important thing in the category of "a small
patch to improve the scheduler algorithm", they may as well get
considered.

It is really far more general than Java though.  We're talking about
people who want to use "some" threads where "some" is enough to keep the
CPUs busy and then some.

> The only answer I see is to leave the kernel optimized for good
> applications and wait for the people who have bad ones to fix them.
> They don't have to fix them, they can suffer the consequences of not
> fixing them, but ``fixing'' the kernel for what is a user level mistake
> is never a good idea.

The thing is, here it's not obviously a user level mistake.  Linux
threaded applications work the way they do because clone() provides
threading facilities and there is no mechanism to help user-level
context switching.

To be honest I think that's where we should be paying more attention:
not the scheduler algorithm, but in helping user-level context
switching.

So that application models that use conceptual threads (including but
not restricted to Java), can appear much more like "just a few" select
based loops to the kernel.

Then IBM would have nothing to bitch about -- their Java context
switches would be _really_ fast and everything would run with equivalent
performance.

I'm thinking here of some kind of "if A blocks unblock B" arrangement
and something for the reverse at suitable application synchronisation
points.  CLONE_COBLOCKING if you like, or WBLOCK.

That way threads can continue to use kernel facilities with the same ease
of use as clone(), but if they block, it's easy for the application
level to get on with something else.  Without having to have lots of
kernel level threads runnable -- which is less efficient anyway.

have a nice day,
-- Jamie



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo en vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/