various compilers

[Derek Rayside <drayside@swen.uwaterloo.ca>]

Etienne thought it would be interesting if someone did an empirical study
to determine if:

	javac + soot == jikes + soot

I have not done that exact study, but I did do a small study comparing
about four Java compilers across five VM's (I got carried away with
assignment #1 in 621b last winter).  I looked at javac (with and without
"optimization"), jikes (with and without "optimization"), VisualAge for
Java, and VisualAge Micro Edition.  The VM's I looked at were IBM 1.1.8,
IBM 1.2.2, Sun 1.2.2, Hotspot, and IBM J9;  all with and without JIT's
(except for J9, which does not have a JIT).  The test program was
LotusXSL, which is built on top of XML4J (aka Xerces).

My little study showed that these compilers all produce about the same
code quality, and that the "optimization" flags don't really do anything. 
One strange result was that the code from Jikes ran significantly slower
in the IBM 1.2.2 JIT:  it seems that these two just do not get along (at
least for this test case).  The Jikes code was equivalent to code from the
other compilers in every other VM, and in IBM 1.2.2 with the JIT turned
off.  I ran this anomalous result a few times, just to make sure.
(Actually, I was quite rigorous about the testing:  I defragmented the
hard disk and rebooted between every run.)

Not counting the anomalous result, there was a maximum 5% standard
deviation across the compilers (i.e. using different compilers on each
VM).  There was 25% -- 30% standard deviation across the VM's (i.e. using
different VM's on each compiler).  In other words, your choice of VM
matters much more than your choice of current industrial compilers
(because none of them is as good as Soot ;).

The other interesting result was that IBM 1.1.8 was the only VM that was
faster with the JIT turned on.  IBM 1.2.2, Sun 1.2.2, and Hotspot all ran
the test case faster with the JIT turned off.  This result held across all
compilers.  Of course, I just had one test program with one sample input.