Benchmarking Java to native compilers

This entry was posted in Computers and tagged java linux on by
Java Duke

Java to native compilers have been around for some years, and I was curious about if one of this solutions could improve the performance of my Carballo Chess Engine.

I ran a tournament between binaries of the Carballo development version (1.5) compiled with different solutions to compare the performance. I used cutechess-cli to run a 3000 game tournament with time control 5″ + 0.1″ per move by side and with the Noomen Test Suite as the starting positions.

The compared binaries

  • The Pure Java Version: This is the carballo-1.5.jar ran with the Oracle JDK 1.8.0_73 VM under my 64bit linux (Debian Sid).
  • GCJ: The GNU’s Java compiler, incomplete and unfinished, but it works for Carballo. This binary was compiled with this script.
  • Excelsior Jet: A classic proprietary Java to native converter at http://www.excelsiorjet.com. I used Excelsior Jet 11 32bit for Linux (evaluation) to generate this binary. The 64bit version had worse results.
  • RoboVM: (https://robovm.com) A solution to run Java apps on iOS. Recently it was bought by Xamarin, and after the Microsoft acquisition of Xamarin, RoboVM was discontinued. RoboVM also has the option to compile Java apps to desktop binaries. I built this binary with the last RoboVM free version (1.8). Now RoboVM is forked in BugVM, but I was not able to build the binary with BugVM.
  • C# compiled with Mono: There is a C# version of Carballo converted with the Sharpen tool. I compiled this binary with MonoDevelop 5.10. The converted code is sub-optimal but it is a good solution if you need a native version (or if you need to integrate Java code in a C# project).

Test results

Rank Name                          ELO   Games   Score   Draws
   1 carballo-1.5-gcj               89    1200     62%     28%
   2 carballo-1.5                   37    1200     55%     27%
   3 carballo-1.5-mono              -5    1200     49%     30%
   4 carballo-1.5-jet              -33    1200     45%     28%
   5 carballo-1.5-robovm           -88    1200     38%     27%

Conclusions

The JVM performance is very good, better than almost all the Java to native solutions.

The exception is GCJ, but it’s incomplete and it will not work for all the Java apps.

I expected better results from Excelsior Jet, as some time ago Carballo Jet binaries where available an used for testing.

The C# version is a bit worse but acceptable.