Friday, May 24, 2013

What happened to java -client?

On our shared server, users are limited to 25 processes. A login script further reduces the soft limit by another 5 processes. This safety margin allows the user to log in an kill something which is using up all their processes. If they want they can raise their soft limit back up to the hard limit.

This works well for most things but java insists on using lots of processes. Confusingly, it prints this message:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
# An error report file with more information is saved as:
# ./hs_err_pid27297.log
The log file is not of much use. It repeats the claim about 'insufficient memory' which is not true and none of the listed solutions will help.

The problem is that the default garbage collector (GC) scales up with the number of CPU cores. Our old server had 8 but the new server has 24. This is easily pushing the users over the 20 thread soft limit.

I don't if the log file lists the number of java threads. If it does I can't work out how to read it. If it could be found it could be compared with the number of available processes:
expr $(ulimit -u) - $(ps x | wc --lines)

In the past I used to solve this problem by adding this to the login script:
alias java="java -client"

but that does not work any more. I don't know when it was removed but it leaves me with the headache of finding a new workaround.

After some research I found this page https://wiki.csiro.au/pages/viewpage.action?pageId=545034311 which suggested using
-XX:ParallelGCThreads=1 -XX:+UseParallelGC
or even just
-XX:+UseSerialGC

This seems to solve the immediate problem but still is not all that useful. To specify the flag to run java you need this syntax:

java -XX:+UseSerialGC

but for javac you have to specify like this:
javac -J-XX:+UseSerialGC


and I don't expect anyone should have to remember that. One simple solution is to alias both those commands but there are many other tools which are part of java which would also need aliases. These other tools may (and do) use their own unique command line parameters which I would rather not have to learn.


Luckily there is also an environment variable you can set like this:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:+UseParallelGC"

however, that makes all the java programs print out this annoying message:
Picked up _JAVA_OPTIONS: -XX:ParallelGCThreads=1 -XX:+UseParallelGC
And the word on the internet is that there is no way to suppress that message.

My solution to use an alias for the most common tools java and javac while keeping the environment variable for all other programs. At first I tried to use this in my alias:
_JAVA_OPTIONS= /usr/bin/java -XX:+UseSerialGC
but an empty variable still shows the message (and best I can tell, bash won't unset a variable like that). I needed to unset it so I had to combine the env command like this:
env -u _JAVA_OPTIONS -XX:+UseSerialGC

My final profile script looks like this:
export _JAVA_OPTIONS="-XX:ParallelGCThreads=1 -XX:+UseParallelGC"
#export _JAVA_OPTIONS="-XX:+UseSerialGC"
alias java="env -u _JAVA_OPTIONS $(which --skip-alias java) $_JAVA_OPTIONS"
alias javac="env -u _JAVA_OPTIONS $(which --skip-alias javac)$(for i in $_JAVA_OPTIONS ; do echo -n " -J$i" ; done)"
You can select which GC you want. I have demonstrated with the more complex example which has two options to be prefixed for javac.

The java users can now happily compile away without java needing the entire machine resources for every invocation.

No comments:

Post a Comment