«

Hello!

This is my second post. Ppl really helped me with the first. I hope there
are answers for this one as well

I own a game company (www.aepoxgames.net) releasing the beta for our first
sci-fi space game in about 2 weeks (www.andromedaonline.net)

We have multiple servers, but the game engine (Game Server Program) is a
Java program (don't ask why) on a windows 2003 enterprise system with dual
xeon p4 processors and 4GB of RAM. This program needs to process orders for
thousands of ships as fast as possible.

When we first set this up, the Game Server Program took exactly 50% of the
CPU cycles when idle, and up to 80% or 90% when working (processing a turn,
for example) and could process a turn of 10,000 ships in 15 seconds.

We revamped some of the major systems (the code and the computer itself),
did a fresh install of Windows 2003 on a different system (still dual Xeon,
but faster MB and CPU), migrated the Game Server Program there, changed the
threadpool priorities in the code, etc.

We've now noticed that on idle (IE not processing turns), the Game Server
Program takes 0% of the CPU load, and when processing a turn goes up to
3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn. It
is the only program running on the machine, and services are at a minimum.
Even with 99% of the CPU resources free (IE on System Idle), the game will
never take more than about 7% of the processor time.

This is extremely serious.

We've tried increasing the priority of Java, but
a) The only way I know to do this is through the Task manager, meaning
it would have to be done every time we run the server
b) I increased the priority to "Real Time" and this had absolutely no
impact on the server process. The CPU load didn't change.

This is the only program intended to run on this computer. We need it
hogging as many resources as it can. Database, development, everything's on
different systems. This machine has one job, run this Java program, and it's
not taking that job seriously.

We don't know if it's a Windows issue, or a Java issue, but I suspect it's a
combination of the two. All we know is it used to run fast as hell, and now
crawls.

This post is going to Java groups and Windows groups on the assumption that
something can be done on each end to improve things (perhaps there's a
command line parameter to tell java to run at a higher priority, perhaps
there's a windows issue not giving proper time to the Java App).

Any thoughts?

Thank you for your time.
Nick Soutter

If all of the underlying system and setup of Windows is identical to the
first system (with the exception of the hardware changes to faster
components). Then unless it is a hardware issue (some setting in the BIOS
to incorrectly set up the mother board or CPUs); then your statement about
"revamped some of the major systems (the code and the computer itself)," -
would lead you to possibly retest the system using the original code base
for the application that was running on the old server. This will help
eliminate your code changes as being the source of the error. (alternative
path is to put your new code version on the old server and retest)

--
Regards,

Mike
--
Mike Brannigan [Microsoft]

This posting is provided "AS IS" with no warranties, and confers no
rights

Please note I cannot respond to e-mailed questions, please use these
newsgroups

"BlackHawke" <> wrote in message
news:C_4Qb.23473$ ink.net...
> Hello!
>
> This is my second post. Ppl really helped me with the first. I hope there
> are answers for this one as well
>
> I own a game company (www.aepoxgames.net) releasing the beta for our first
> sci-fi space game in about 2 weeks (www.andromedaonline.net)
>
> We have multiple servers, but the game engine (Game Server Program) is a
> Java program (don't ask why) on a windows 2003 enterprise system with dual
> xeon p4 processors and 4GB of RAM. This program needs to process orders
for
> thousands of ships as fast as possible.
>
> When we first set this up, the Game Server Program took exactly 50% of the
> CPU cycles when idle, and up to 80% or 90% when working (processing a
turn,
> for example) and could process a turn of 10,000 ships in 15 seconds.
>
> We revamped some of the major systems (the code and the computer itself),
> did a fresh install of Windows 2003 on a different system (still dual
Xeon,
> but faster MB and CPU), migrated the Game Server Program there, changed
the
> threadpool priorities in the code, etc.
>
> We've now noticed that on idle (IE not processing turns), the Game Server
> Program takes 0% of the CPU load, and when processing a turn goes up to
> 3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn.
It
> is the only program running on the machine, and services are at a minimum.
> Even with 99% of the CPU resources free (IE on System Idle), the game will
> never take more than about 7% of the processor time.
>
> This is extremely serious.
>
> We've tried increasing the priority of Java, but
> a) The only way I know to do this is through the Task manager, meaning
> it would have to be done every time we run the server
> b) I increased the priority to "Real Time" and this had absolutely no
> impact on the server process. The CPU load didn't change.
>
> This is the only program intended to run on this computer. We need it
> hogging as many resources as it can. Database, development, everything's
on
> different systems. This machine has one job, run this Java program, and
it's
> not taking that job seriously.
>
> We don't know if it's a Windows issue, or a Java issue, but I suspect it's
a
> combination of the two. All we know is it used to run fast as hell, and
now
> crawls.
>
> This post is going to Java groups and Windows groups on the assumption
that
> something can be done on each end to improve things (perhaps there's a
> command line parameter to tell java to run at a higher priority, perhaps
> there's a windows issue not giving proper time to the Java App).
>
> Any thoughts?
>
> Thank you for your time.
> Nick Soutter
>
>

On Fri, 23 Jan 2004 09:32:34 +0100, BlackHawke wrote:

> Hello!
>
> This is my second post. Ppl really helped me with the first. I hope
> there are answers for this one as well
>
> When we first set this up, the Game Server Program took exactly 50% of
> the CPU cycles when idle, and up to 80% or 90% when working (processing
> a turn, for example) and could process a turn of 10,000 ships in 15
> seconds.
>
> We revamped some of the major systems (the code and the computer
> itself), did a fresh install of Windows 2003 on a different system
> (still dual Xeon, but faster MB and CPU), migrated the Game Server
> Program there, changed the threadpool priorities in the code, etc.
>
> We've now noticed that on idle (IE not processing turns), the Game
> Server Program takes 0% of the CPU load, and when processing a turn goes
> up to 3-7%. It never exceeds 7%, and takes FOREVER now when processing a
> turn. It is the only program running on the machine, and services are at
> a minimum. Even with 99% of the CPU resources free (IE on System Idle),
> the game will never take more than about 7% of the processor time.

You say you revamped the code. What does that mean? Was the code changed?
If the Java program can not hog 90%-100% of the CPU then I'd
look at code. I don't use windows, but I doubt it will prioritize the
idle process higher than your program.

Things to look at:
1 Synchronizations. This could kill an application on a multi cpu machine.
2 IO. Are you reading / writing to disk or network? If you are and this is
done synchnonously then you application will probably wait for the disk a
lot of the time.
3. Object creation. Are you creating lots of objects that has to be GCd?
Testing my applications shows that creating small objects that should be
GCd makes minor impact, but as soon as the size of the objects grows (a
few KB or so) the time spent on GC increases a lot. Could be related to
now nurseries are managed and if objects over a certain size are not put
into a nursery. You can probably configure your JVM to tell you how often
it has to do a GC and how much memory was freed.

Roger Lindsjö

What is the JDK you are running this Java program on?
Have you considered running this application on a Linux system?

Roman.

"BlackHawke" <> wrote in message
news:C_4Qb.23473$ ink.net...
> Hello!
>
> This is my second post. Ppl really helped me with the first. I hope there
> are answers for this one as well
>
> I own a game company (www.aepoxgames.net) releasing the beta for our first
> sci-fi space game in about 2 weeks (www.andromedaonline.net)
>
> We have multiple servers, but the game engine (Game Server Program) is a
> Java program (don't ask why) on a windows 2003 enterprise system with dual
> xeon p4 processors and 4GB of RAM. This program needs to process orders
for
> thousands of ships as fast as possible.
>
> When we first set this up, the Game Server Program took exactly 50% of the
> CPU cycles when idle, and up to 80% or 90% when working (processing a
turn,
> for example) and could process a turn of 10,000 ships in 15 seconds.
>
> We revamped some of the major systems (the code and the computer itself),
> did a fresh install of Windows 2003 on a different system (still dual
Xeon,
> but faster MB and CPU), migrated the Game Server Program there, changed
the
> threadpool priorities in the code, etc.
>
> We've now noticed that on idle (IE not processing turns), the Game Server
> Program takes 0% of the CPU load, and when processing a turn goes up to
> 3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn.
It
> is the only program running on the machine, and services are at a minimum.
> Even with 99% of the CPU resources free (IE on System Idle), the game will
> never take more than about 7% of the processor time.
>
> This is extremely serious.
>
> We've tried increasing the priority of Java, but
> a) The only way I know to do this is through the Task manager, meaning
> it would have to be done every time we run the server
> b) I increased the priority to "Real Time" and this had absolutely no
> impact on the server process. The CPU load didn't change.
>
> This is the only program intended to run on this computer. We need it
> hogging as many resources as it can. Database, development, everything's
on
> different systems. This machine has one job, run this Java program, and
it's
> not taking that job seriously.
>
> We don't know if it's a Windows issue, or a Java issue, but I suspect it's
a
> combination of the two. All we know is it used to run fast as hell, and
now
> crawls.
>
> This post is going to Java groups and Windows groups on the assumption
that
> something can be done on each end to improve things (perhaps there's a
> command line parameter to tell java to run at a higher priority, perhaps
> there's a windows issue not giving proper time to the Java App).
>
> Any thoughts?
>
> Thank you for your time.
> Nick Soutter
>
>

[Over-enthusiastic cross-posting removed]

BlackHawke wrote:

> When we first set this up, the Game Server Program took exactly 50% of the
> CPU cycles when idle, and up to 80% or 90% when working (processing a
> turn, for example) and could process a turn of 10,000 ships in 15 seconds.

50% CPU time on *IDLE* ?? What on earth was it doing ? Was it intended to do
that ? If you don't already know the answers to these questions (and I *guess*
that you don't or you'd have included them in you detailed description) then
you *need* to find the answers...


> We revamped some of the major systems (the code and the computer itself),
> did a fresh install of Windows 2003 on a different system (still dual
> Xeon, but faster MB and CPU), migrated the Game Server Program there,
> changed the threadpool priorities in the code, etc.
>
> We've now noticed that on idle (IE not processing turns), the Game Server
> Program takes 0% of the CPU load, and when processing a turn goes up to
> 3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn.


If it's only using 0% CPU on idle now then that's what I'd expect, but it's not
consistent with the first case. You should be able to find out the difference
without considering what it's doing during its "real" workload. That should
make the investigation easier. Also there is no way that changing, even, the
entire OS is going to make you app go from 50% to 0% CPU to do *nothing*, so
you can forget -- for now -- about possible OS links.

However, if the thing is running slowly, but is not consuming CPU then it
*must* be waiting for something. I can think of two possibilities:
network waits
internal blocking

An example of network waits (this is a real example from my own past). An
application was able to serve network requests at X/sec when the test workload
was delivered from the same box. As soon as I generalised the test to run the
client on a colleague's box the workrate plummeted to around 1/20th, when I'd
expected it to go *up* (since the server box wasn't running the client too).
That turned out to be because of my logging code which logged the hostname of
the client rather than the IP address -- reverse look up "localhost" was local
and fast, but for some reason reverse lookup of my colleague's box always
involved a network access (DHCP or something) which slowed it down massively.
Of course, almost any kind of network access could show the same kind of
problem, database access for instance.

Internal blocking could be almost anything. Perhaps you have
over-"synchronised" something so that all processing is now happening on just
one thread. Perhaps you are waiting for some conditions somewhere that never
becomes true (or where the code for signalling that it has become true is
broken), and so your app is halted until something times out, or otherwise
wakes up the sleeping threads.

I would advocate sitting down and *thinking* about what's going on -- messing
around trying to fix the symptoms (e.g. playing with task priorities) is just a
waste of time. (I don't blame you though, it's hard to think clearly in what
sounds like a panic situation). However, *while* you are thinking, it may be
worthwhile running some profiling on the application, that should be able to
find operations that are taking a long time without doing any real work.

HTH

-- chris

Sounds like deadlock. Turn on logging. If you don't have logging
then add it so you can solve these kinds of dynamic problems.
Deadlock can be in your communication protocols or threading.

It also sounds like you need spend some time with a profiler.


BlackHawke wrote:
> We've now noticed that on idle (IE not processing turns), the Game Server
> Program takes 0% of the CPU load, and when processing a turn goes up to
> 3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn. It
> is the only program running on the machine, and services are at a minimum.
> Even with 99% of the CPU resources free (IE on System Idle), the game will
> never take more than about 7% of the processor time.

"BlackHawke" <> wrote in message news:<C_4Qb.23473$ link.net>...

> We revamped some of the major systems (the code and the computer itself),

you have introduced a major race condition somewhere, do some research
on profilers and look at where the hotspot is.

"We should forget about small efficiencies, say about 97% of the time:
premature optimization is the root of all evil."

- Knuth

I know that there is CPU Throttling in Windows 2003... Is that only for IIS?

We found a way to compile this into native code... It's now revved up to 25%
of the CPU cycles during intense work... I notice that in the "performance"
section of the task manager it lists 4 CPU's (we only have 2)... It's
letting us max out one of the "4" to 100%.... Any way to let it max them
all?

Again, I increased the priority, but no luck....

Nick


"Mike Brannigan [MSFT]" <> wrote in message
news:...
> If all of the underlying system and setup of Windows is identical to the
> first system (with the exception of the hardware changes to faster
> components). Then unless it is a hardware issue (some setting in the BIOS
> to incorrectly set up the mother board or CPUs); then your statement about
> "revamped some of the major systems (the code and the computer itself)," -
> would lead you to possibly retest the system using the original code base
> for the application that was running on the old server. This will help
> eliminate your code changes as being the source of the error. (alternative
> path is to put your new code version on the old server and retest)
>
> --
> Regards,
>
> Mike
> --
> Mike Brannigan [Microsoft]
>
> This posting is provided "AS IS" with no warranties, and confers no
> rights
>
> Please note I cannot respond to e-mailed questions, please use these
> newsgroups
>
> "BlackHawke" <> wrote in message
> news:C_4Qb.23473$ ink.net...
> > Hello!
> >
> > This is my second post. Ppl really helped me with the first. I hope
there
> > are answers for this one as well
> >
> > I own a game company (www.aepoxgames.net) releasing the beta for our
first
> > sci-fi space game in about 2 weeks (www.andromedaonline.net)
> >
> > We have multiple servers, but the game engine (Game Server Program) is a
> > Java program (don't ask why) on a windows 2003 enterprise system with
dual
> > xeon p4 processors and 4GB of RAM. This program needs to process orders
> for
> > thousands of ships as fast as possible.
> >
> > When we first set this up, the Game Server Program took exactly 50% of
the
> > CPU cycles when idle, and up to 80% or 90% when working (processing a
> turn,
> > for example) and could process a turn of 10,000 ships in 15 seconds.
> >
> > We revamped some of the major systems (the code and the computer
itself),
> > did a fresh install of Windows 2003 on a different system (still dual
> Xeon,
> > but faster MB and CPU), migrated the Game Server Program there, changed
> the
> > threadpool priorities in the code, etc.
> >
> > We've now noticed that on idle (IE not processing turns), the Game
Server
> > Program takes 0% of the CPU load, and when processing a turn goes up to
> > 3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn.
> It
> > is the only program running on the machine, and services are at a
minimum.
> > Even with 99% of the CPU resources free (IE on System Idle), the game
will
> > never take more than about 7% of the processor time.
> >
> > This is extremely serious.
> >
> > We've tried increasing the priority of Java, but
> > a) The only way I know to do this is through the Task manager,
meaning
> > it would have to be done every time we run the server
> > b) I increased the priority to "Real Time" and this had absolutely
no
> > impact on the server process. The CPU load didn't change.
> >
> > This is the only program intended to run on this computer. We need it
> > hogging as many resources as it can. Database, development, everything's
> on
> > different systems. This machine has one job, run this Java program, and
> it's
> > not taking that job seriously.
> >
> > We don't know if it's a Windows issue, or a Java issue, but I suspect
it's
> a
> > combination of the two. All we know is it used to run fast as hell, and
> now
> > crawls.
> >
> > This post is going to Java groups and Windows groups on the assumption
> that
> > something can be done on each end to improve things (perhaps there's a
> > command line parameter to tell java to run at a higher priority, perhaps
> > there's a windows issue not giving proper time to the Java App).
> >
> > Any thoughts?
> >
> > Thank you for your time.
> > Nick Soutter
> >
> >
>
>

We have, but honestly no one here is an experienced Linux Admin. We can't
afford an admin (we're WAY over budget), but we all know windows systems
pretty well, so.....

I'd do it if I could...

Nick
"Roman" <> wrote in message
news:lIJQb.252834$JQ1.103869@pd7tw1no...
> What is the JDK you are running this Java program on?
> Have you considered running this application on a Linux system?
>
> Roman.
>
> "BlackHawke" <> wrote in message
> news:C_4Qb.23473$ ink.net...
> > Hello!
> >
> > This is my second post. Ppl really helped me with the first. I hope
there
> > are answers for this one as well
> >
> > I own a game company (www.aepoxgames.net) releasing the beta for our
first
> > sci-fi space game in about 2 weeks (www.andromedaonline.net)
> >
> > We have multiple servers, but the game engine (Game Server Program) is a
> > Java program (don't ask why) on a windows 2003 enterprise system with
dual
> > xeon p4 processors and 4GB of RAM. This program needs to process orders
> for
> > thousands of ships as fast as possible.
> >
> > When we first set this up, the Game Server Program took exactly 50% of
the
> > CPU cycles when idle, and up to 80% or 90% when working (processing a
> turn,
> > for example) and could process a turn of 10,000 ships in 15 seconds.
> >
> > We revamped some of the major systems (the code and the computer
itself),
> > did a fresh install of Windows 2003 on a different system (still dual
> Xeon,
> > but faster MB and CPU), migrated the Game Server Program there, changed
> the
> > threadpool priorities in the code, etc.
> >
> > We've now noticed that on idle (IE not processing turns), the Game
Server
> > Program takes 0% of the CPU load, and when processing a turn goes up to
> > 3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn.
> It
> > is the only program running on the machine, and services are at a
minimum.
> > Even with 99% of the CPU resources free (IE on System Idle), the game
will
> > never take more than about 7% of the processor time.
> >
> > This is extremely serious.
> >
> > We've tried increasing the priority of Java, but
> > a) The only way I know to do this is through the Task manager,
meaning
> > it would have to be done every time we run the server
> > b) I increased the priority to "Real Time" and this had absolutely
no
> > impact on the server process. The CPU load didn't change.
> >
> > This is the only program intended to run on this computer. We need it
> > hogging as many resources as it can. Database, development, everything's
> on
> > different systems. This machine has one job, run this Java program, and
> it's
> > not taking that job seriously.
> >
> > We don't know if it's a Windows issue, or a Java issue, but I suspect
it's
> a
> > combination of the two. All we know is it used to run fast as hell, and
> now
> > crawls.
> >
> > This post is going to Java groups and Windows groups on the assumption
> that
> > something can be done on each end to improve things (perhaps there's a
> > command line parameter to tell java to run at a higher priority, perhaps
> > there's a windows issue not giving proper time to the Java App).
> >
> > Any thoughts?
> >
> > Thank you for your time.
> > Nick Soutter
> >
> >
>
>

You know, this started RIGHT AFTER a revamp of all of the communications
protocols (and some threading issues too)....

Can you explain deadlock to me a bit more?

Thank you.

Nick
"thoff" <> wrote in message
news:...
> Sounds like deadlock. Turn on logging. If you don't have logging
> then add it so you can solve these kinds of dynamic problems.
> Deadlock can be in your communication protocols or threading.
>
> It also sounds like you need spend some time with a profiler.
>
>
> BlackHawke wrote:
> > We've now noticed that on idle (IE not processing turns), the Game
Server
> > Program takes 0% of the CPU load, and when processing a turn goes up to
> > 3-7%. It never exceeds 7%, and takes FOREVER now when processing a turn.
It
> > is the only program running on the machine, and services are at a
minimum.
> > Even with 99% of the CPU resources free (IE on System Idle), the game
will
> > never take more than about 7% of the processor time.