Virtualization
From APIDesign
Virtualization is an important aspect in cluelessness. Instead of improving own application, or making it more secure, one can run it in an isolated, dedicated virtual computer. It may require more memory or more powerful processors, but the need of software quality is lower and thus one can achieve more by knowing less - the credo of cluelessness.
Contents |
OpenJDK
I wanted to run this site's associated hudson builder in an isolated environment. Originally I was using vserver, but after my hard disk crashed, I decided to try VirtualBox.
I am running Kubuntu as host and also a stripped down version of Kubuntu as a guest. I have given the guest 384MB of memory and 400MB of swap. To my biggest surprise, running even smallest Java program on OpenJDK in such environment allocated 650MB(!!!) of virtual addressing space. Just check the top output:
Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie Cpu(s): 0.7%us, 4.7%sy, 0.0%ni, 94.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 379872k total, 306828k used, 73044k free, 40164k buffers Swap: 407544k total, 16064k used, 391480k free, 111272k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4129 hudson 20 0 655m 10m 7484 S 2.3 2.8 0:01.02 java.openjdk 4210 hudson 20 0 5248 3500 1556 S 0.0 0.9 0:00.87 bash 4144 root 20 0 8856 3048 2412 S 0.0 0.8 0:00.40 sshd 1619 postfix 20 0 6244 2544 1812 S 0.0 0.7 0:00.12 tlsmgr 951 hudson 20 0 5248 2008 1280 S 0.0 0.5 0:01.08 bash
650MB of virtual space is a lot. If you moreover realize that hudson starts multiple Java VMs in parallel, it is easy to see that the system can't work. Indeed it did not: swap was full all the time, sometimes the VirtualBox machine rebooted, etc.
For a while I was searching for a solution. I tried to pass in -mx64M into every started process, but as far as I can tell, this is not easy. There does not seem to be any configuration file the java launcher reads (jvm.cfg is not the one, one cannot specify there default command line arguments). Well, it is not documented, but there is:
_JAVA_OPTIONS="-Xmx64M -XX:MaxPermSize=32M" export _JAVA_OPTIONS
So now, instead of specifying the parameter on command line, which is not practical you can put the above code snippet into /etc/profile.d/java.sh and all java processes will pick it up! Hopefully it will work now!
The Previous Solutions
Then a friend of mine suggested to try official JDK, not just OpenJDK variant. I hesitated for a while, but then I decided to try it. Here is the top output:
Tasks: 81 total, 2 running, 79 sleeping, 0 stopped, 0 zombie Cpu(s): 1.0%us, 2.3%sy, 0.0%ni, 96.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 379872k total, 307924k used, 71948k free, 40388k buffers Swap: 407544k total, 16056k used, 391488k free, 111424k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4306 hudson 20 0 239m 10m 6956 S 0.0 2.8 0:00.48 java 4210 hudson 20 0 5248 3500 1556 S 0.0 0.9 0:00.88 bash 4144 root 20 0 8856 3048 2412 S 0.0 0.8 0:00.40 sshd
Better. The 240M of occupried space is still much more than I'd like to see (no wonder nobody offers free hosting of Java applications), but at least it fits into real memory. By using JDK 1.6.0_20 I can really run my Hudson in VirtualBox with 384M of memory.
Conclusion? OpenJDK seems to contain a bug in a launcher which makes it completely unsuitable for running in a virtualized environment! What a shame for a standard Java implementation present in each Linux distribution!
Too Big Initial Limit
The difference between OpenJDK and JDK seems to be result of this change. It increases the heap limit to 512MB and the perm gen size to 128MB claiming that if Eclipse needs it, every other Java application will need it as well.
No it won't. If you run multiple VMs with small applications (for example JUnit tests) in parallel, the performance will suffer.
It would indeed be great if the VM internal structures could start small and grow, but if that is not possible for some reasons now, then the default ergonomics need to take existing system resources into account. Especially it needs to prevent allocating more memory than is physically available.
<comments/>
Onno Molenkamp said ...
Nathan said ...
This isn't just a virtualization issue - I upgraded my PC to Ubuntu 10.04 which uninstalled the Sun JDK and installed OpenJDK. Any development work I did in Netbeans which launched Glassfish completely crippled my machine. After removing OpenJDK and re-installing the Sun JDK, all was well again. This on a straight Linux machine, no virtualized anything. Given this, I cannot honestly recommend OpenJDK for any purpose what-so-ever.
Added on top of this is that the OpenJDK applet plugin doesn't support the Java console, so debugging applets is infeasible (there's a work-around to launch FireFox from a console, but that's not helpful when trying to debug for an end-user).
--Nathan 02:26, 15 June 2010 (CEST)
I'll do my best to make sure this testimony is seen by OpenJDK guys. --JaroslavTulach 11:11, 15 June 2010 (UTC)
Daniel said ...
I agree that 512m is much too high for most Java applications. A recent version of Sun JDK bumped the default heap size to 256 MB, though, so if you're concerned about resources, I'd strongly recommend to find a way for setting the maximum heap and permgen size in Hudson (I guess it'd be fine with 64 MB too).
--Daniel 09:20, 15 June 2010 (CEST)
256MB!? Hopefully they at least check whether the system has 256MB of real memory! If not, I at least know where to report a bug. --JaroslavTulach 11:11, 15 June 2010 (UTC)
Antony John said ...
I faced the same problem with OpenJdk 1.6 in 10.4 Ubuntu and reverted to sun jdk 1.6.. and reduced the default heap allocations.. My machine even after having 2 GB of memory + flash based parallel swap went into grinding halts.. I think any app using more than 64 MB on startup is bloat-ware... Thanks Nathan for pointing out.. By the way I switched to Netbeans and they have this wonderful activation on demand mechanism for plugins that helps to manage memory more efficiently..
--Antony John 20:12, 15 June 2010 (CEST)
The VIRT column doesn't say much about the amount of memory used. This can be easily seen by comparing the total amounts of memory used in both cases: they're more or less the same, even though the VIRT value for the java process is very different.
--Onno Molenkamp 18:07, 14 June 2010 (CEST)
Well, that was my assumption as well. But for some reason it is not true. Starting many (maybe only Java) processes with huge VIRT size, just kills the system, forcing it to use huge amount of swap.
--JaroslavTulach 11:11, 15 June 2010 (UTC)