Classpath hell just froze over

Classpath Hell

The classpath is dead … that’s what Mark Reinhold (Principal Engineer @ Sun) said at a general session during JavaOne 2009. It’s the type of statement that should resonate with any of us who have been victims of classpath/jar hell. Mark announced this in the context of reviewing the new language features due to be released in JDK 7 (ETA 2010) or more specifically the new Java module system codenamed Jigsaw.

Module Systems and Jigsaw

Jigsaw is a low level implementation of a module system, which also includes some language and VM changes that are specified in JSR 294 (note that Jigsaw is being done outside of any JSR and is not a reference implementation of 294). Another implementation of a module system most Java developers are likely more familiar with is OSGi. But what is a module system and why do we need one?

Depencies on multiple versions (of module B in this example)

Depencies on multiple versions (of module B in this example)

A module system enables modular programming, it allows developers to separate code into clearly defined modules that can specify what other modules they depend on and their versions, can be internally modified without breaking other modules that depend on it and can hide its own internals from external modules.

A module system can solve the classpath/jar hell problem by leveraging the previously mentioned features in order to allow an application to use the right versions of the set of libraries it requires. In the diagram shown here, we have an APP module that depends on the A, B and C modules. But module C also depends on B, however this dependency is to a different version of B than the one APP needs. A useful module system would then allow us to have 2 versions of the B module, with the compatible version and implementation visible to the appropriate modules (in this case APP and C).

Also, by clearly expressing module dependencies, we no longer have to look at random lib folders or hidden documentation when using a 3rd party library, just so we can guess what the corresponding jar files are needed to run it. With the system implemented by Jigsaw, we can simply introspect the module and let it tell us what those dependencies are. It takes the guesswork and the dark magic completely out of the equation.

But back to Mark’s presentation at the general session. The main reasons to introduce Jigsaw into JDK 7 are the following:

  • Eliminate classpath complexity (classpath/jar hell problem already discussed).
  • Increase performance. The VM download size can be made smaller, and startup time should improve because for small programs (say HelloWorld) unnecessary classes don’t have to be loaded at all. Mark showed how he could run HelloWorld by only using 2 modules listed by the new jmod command; jdk.base@7-ea and jdk.boot@7-ea.
  • Enable native integration for installing modules.

Mark demonstrated that last point by showing a demo of him installing various JDK packages via the Synaptic Package Manager.

Installing Java modules via Synaptic Package Manager

Installing Java modules via Synaptic Package Manager

Alex Buckley’s presentation

I attended Alex Buckley‘s (Computational Theologist @ Sun) session on “Modularity in the Java Programming Language: JSR 294 and beyond”, which had a lot more details on the specific language changes to be implemented in order to support this new feature.

The most noticeable aspect of this is the introduction of module-info.java. This file goes at the top of your module filesystem, and is the class that specifies at compile and runtime the definition of the module. Here is an example, coming from Alex’s slides;

module org.planetjdk.aggregator {
          system jigsaw;
          requires module jdom;
          requires module tagsoup;
          requires module rome;
          requires module rome-fetcher;
          requires module joda-time;
          requires module java-xml;
          requires module java-base;
          class org.planetjdk.aggregator.Main;
 }

Our module is “org.planetjdk.aggregator” and uses the jigsaw module system. Depenencies are denoted by using requires and providing the module name, Alex calls this part of the module metadata the “logical classpath” of the module. You can also define the main class for the module which in this case is “org.planetjdk.aggregator.Main“.

You’ll also notice that the module name is quite long “org.planetjdk.aggregator”, while the list of required modules uses shorter names. This can be done by supplying a module alias. In the previous example, we could have added the line;

provides module aggregator;

This would then allow other modules to say “requires module aggregator” and successfully link to this module by this shorter form of the name.

Also, you can specify your module version and version dependencies in module-info.java. This updated example illustrates this mechanism;

module org.planetjdk.aggregator @ 1.0 {
          system jigsaw;
          requires module jdom @ 1.*;
          requires module tagsoup @ 1.2.*;
          requires module rome @ =1.0;
          requires module rome-fetcher @ =1.0;
          requires module joda-time @ [1.6,2.0);
          requires module java-xml @ 7.*;
          requires module java-base @ 7.*;
          class org.planetjdk.aggregator.Main;
}

Note also that you can only have one module-info.java per module, in only one location, so if you have a big source tree and you want to create multiple modules from it you will have to either separate your source into different trees or use some creative scripting to accomplish this at build time. I believe Mark mentioned they had to use some scripted solution for the jdk base as I believe it's one tree.

The rest of Alex's presentation focused on how to locate modules at compile and runtime by using the "modulepath", the use of the new module keyword and how it affects accessibility, and he gives a glimpse as to how they plan to support other module systems (for example, the version notation can be different when using other module systems).

This is all better explained in Alex's slides, so I encourage everyone to take a look at his presentation (link provided above).

The Modular Java Platform: Q&A

Jury

There were rumors in the meeting that the clown is a secret member of the Java Posse

Wednesday night, I attended a BOF like meeting hosted by Mark and Alex setup in order for the community to openly ask and voice any concerns over the work presented by them on modular Java. The setup was sort of a round table configuration (without the round table) in a very informal setting (btw it was hard for participants in the back to listen well, hopefully microphones are used next time). The first question was kind of funny, somebody asked Alex what a "Computational Theologist" does (the title makes it sound like the equivalent of a Thomas Aquinas for the nerd world).

After that, it seemed some people were very concerned about how to deploy modules, patch jars and other deployment issues. My impression was that a lot of people had a problem with the native installation part of the proposal. Mark answered that Java developers have gotten used to being in our own little jar world, separate from the OS integration issues of module and library management and that it was time for Java to integrate more with these systems. He has a point, do we really have to deploy log4j jars with all our products over and over instead of simply having a log4j installed in the system that all modules can refer to?

However, installation with native package systems is a big change. And it is also unclear what that is going to look like in Windows. It seemed like the Windows part of the equation has not been finalized, would Java 7 include some type of Java specific module registry or would it leverage something like .NET's Global Assembly Cache?

Somebody bought up the point that this new system was imposing more work on open source projects. That is, today you want to release your library as a jar, you know it's going to work everywhere. With Jigsaw, projects may have to deliver a large set of native module packages depending on all the platforms they want to deploy to. Many suggested to Mark and Alex that a cross platform package format was needed, but Mark acknowledged that Sun is not really working on that and they're just worrying mainly about specifying the module file structure and leaving the rest to the native package systems.

I noticed a few members from the OSGi community in attendance, I recognized Peter Kriens (member of the OSGi Alliance), Richard S. Hall (From Apache Felix, works for Sun in the Glassfish team) and BJ Hargrave (CTO of the OSGi Alliance, committer on Eclipse Equinox and IBM employee). Peter and Richard are both members of the expert group for JSR 294, so I was a bit surprised they weren't at all in "harmony" with Mark and Alex and the work they've done with Jigsaw. See, last time I saw discussions on this topic, JSR 277 was still active (it is on hold for now) and I was under the impression that Sun and OSGi had reached some type of "gentleman's agreement". That doesn't seem to be the case at all.

OSGi is a very mature technology and has more features than Jigsaw. However OSGi doesn't address the issue of native package installation, and you could argue that it would not be suitable to solve the issue of modularizing the JDK itself. I think on that last point, many people don't mind Jigsaw even on the OSGi side, the arguments come when you say Jigsaw could be used as an application level module system. Additionally OSGi still has some key features that are missing in Jigsaw; it allows more dynamic loading of modules (useful for plugins) and is backwards compatible (Jigsaw will only be supported from Java 7 and up).

Having said that, one thing I like more about Jigsaw is how it defines dependencies is the module-info.java file. It is a much better syntax than the error prone use of manifest files used in OSGi, where an extra space can break the whole thing for you. Adding keywords for module accessibility is also a very nice thing, and it seems the goal is to also let a modular system like OSGi take advantage of these features. Now how can these both systems inter-operate between each other in a single VM? I have no idea how that works ... yet.

In the end, the Q&A showed that there is a lot of controversy and confusion over this new feature. There was also a noticeable lack of questions during the meeting, with a few of the same people asking all the questions. I got the impression that the need and functionality of a module system in Java is not generally fully understood, so to me that means a lot more work needs to be done to explain just what is going on and why there is so much effort going into this area.

Some key questions I'd like to see clarified:

  • A more technical and detailed explanation of why Jigsaw was chosen over OSGi.
  • Will the JDK provide tools for wrapping modules in native packages?
  • Is a cross platform package system completely out of the question?
  • How would multiple modules systems work together in the same VM/application instance?
  • What happens when an applet or javafx application requires core JDK modules that are not present in the current installation?
  • Native library module integration issues (I sort of asked this in the Q&A). Where are native packages placed, and what happens when multiple modules include the same native libraries (say 2 modules have the same DLL embedded in them)?

This doesn't affect me, why should I care?

These changes will affect everybody, at the very least under the hood your core Java packages will be loaded in a more modularized fashion. For people who would like to run Java in other devices, you will notice that you have better ways to split up your Java distribution to only the parts that you need.

However, if you are interested in scaling your code the right way and modularizing it, you should start becoming familiar with what is going on in this space. For your applications, you'll still have the choice of using existing modular systems like OSGi or using the new facilities provided by Java 7. In the future, we should expect more details to come out as to how you can combine both (Jigsaw and OSGi). Others might prefer not to depend on an external Java project like OSGi and leverage the core support that will be provided for modules.

On the other hand, some people criticize that his is a repeat of the java.util.logging fiasco and say they can mostly ignore these changes. The jury is still out, but this is a very fundamental change in the language and platform and a very different scenario than choosing what logging package to include in the core JDK.

Oh and you might soon see new annoying interview or exam code questions trying to trip you up with gems like this one (from Alex's slides and this blog entry):

package module;
// BTW you should NEVER write code like this!!!
module class module {
        module module module = new
            module();
        module module() {}
}

Anyways, sorry for the longish blog post but hopefully this sheds some light into some of the discussions about this topic that took place at JavaOne. If you are interested in getting more in depth details about this topic, please take a look at some of the following links;

Update. Some additional links that have come to my attention after publishing this blog post;

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

17 Comments »

 
  • Babak says:

    It surprises me that you never mention maven. That technology, right now, is the defacto dependency management tool being used by much Java code out there. I believe an argument must first be made why we can't build on what we already have (something like maven).

  • Tbee says:

    I'm also very confused about it all. Maven is a only build tool, missing all the runtime requirements for this problem. The stripped down Maven dependency management, Ivy, is closer to what you would want; a centrale secury component repository.

    But that aside; I've wanted something like assemblies ever since I first tried DotNet, many many many moons ago. But I do not understand why Sun is doing the not-invented-here thing. Taking OSGi and simply adding language support would have been perfect. Native distribution? Please not!

  • Peter Kriens says:

    Very good article, thanks!

    I start writing a reply to answer some of your questions but it turned into a blog :-(

    http://www.osgi.org/blog/2009/06/classpath-hell-j

    @babak: Yes, maven is a long way but they are not a Java runtime standard. Jason van Zyl has clearly indicated his strong support for OSGi in this area. He seems to see the maven-osgi combination as the future. His Sonatype company is very active in the OSGi ecosystem and are increasing their role.

    Kind regards,

    Peter Kriens

  • alopecoid says:

    The "native" distribution app is a very bad idea in my opinion. The app, if there needs to be one, should be written in Java, bundled with the JRE, and should do nothing more than download Jars into a standard directory. The app should allow additional repositories to be specified and where each repository stores downloaded Jars.

    At any point in time, a person should be able to Zip up the directory and copy it elsewhere without some intimate knowledge of how things work. The same is true for the JRE/JDK itself.

  • [...] Sellhorn published an excellent review titled Classpath hell just froze over of Java modularity based on his experiences at JavaOne. He also includes references to additional [...]

  • Augusto says:

    @alopecoid

    Like everything, it has its pros and cons. I see some of the benefits of integrating with the native package management of a system, but it would be really nice to get the complete picture to feel more comfortable with the concept. Right now, nobody really knows how this is going to work in Windows (or Apple), and then there's even more details like what if I don't want to update modules at the system level but just for my application?

    The native packaging issues related to Jigsaw could have been a dedicated technical track by themselves. I would hope more details and ideas come out soon just to see if people are comfortable with this or not.

    @Peter

    Thanks for your reply, I'll try to follow up in the comments section of your blog.

  • bubak says:

    Just another modular framework. Sun already dropped one, what makes you sure this one will survive? I will stick with OSGi.

  • Augusto says:

    I don't know what you mean when you say you're not sure this one "will survive". It's going in Java 7, even if everybody keeps using OSGi it will be there.

    As for Sun dropping the ball on another modular framework, which one are you referring to, the Netbeans Module system?

  • Augusto says:

    Note for anybody that attended the Q&A:

    When talking about OSGi, Alex Buckley brought up some issue somebody had (I think with the JDK's corba packages) while using OSGi but I don't remember main problem in the blog post or what blog it was. It was hard to hear, all I remember is that he read it out loud and said something like "this is the type of thing we are trying/want to avoid". If anybody knows what this blog post was or what that part of the discussion was about please let me know.

  • Al says:

    Just take a look at POMStrap, it solves classpath hell using Maven POM files recursively.

  • [...] Project Jigsaw, which will modularize the JDK. Most of the audience got it when Mark announced that Classpath is Dead, but for a deeper dive, check out this JavaPosse episode, where both Mark and Alex got a [...]

  • Christian Steinert says:

    As a bystander, I find it increasingly confusing, what Jigsaw actually trying to BE.

    I understand, that Sun is looking for some way that allows them to modularize the core java components, in such a way that the VM itself can benefit from it.

    Then, there is the whole native packaging side of things, which – so far – seems pretty unrelated from a technical point of view. The whole idea of native packaging is both interesting and frightening at the same time, depending on how it is carried out. Having an easy way to install java apps in a linux distribution sounds great to me as a Debian user. Packaging tools would be nice, but they could, for example be developed in cooperation with icedTead – I don't see why they should go into the JDK. It should definitely not be necessary to generate "installable binaries" for each target platform, when deploying java code.

    In my opinion, one of the major reasons one point of failure for java apps on the desktops was that they used .jar as extension for "executable jar files". Why not have the convention of .javaapp files which are nothing but renamed executable .jars with an icon in them that are executed automatically through a Java-specific shell extension? This would make a LOAD of windows-specific java starter/installer programs go away over time. This does not say, that either installation support or a module system are not useful, but sometimes, a LITTLE more integration can make a big difference, without having to re-invent the universe.

    Then, and that is the point that confuses me most, Mark Reinhold claimed that one of the possible future advantages of their module is that there is an act of installation which would, for example allow to pre-compile code to improve startup times.

    To me, this raises a lot of questions

    - what is preventing Sun from pre-compiling their own Java code before download or during installation? No public plattform-specific installation mechanism is required for that. Was this not mainly about JVM startup times for Desktop and Web Apps?

    - what is preventing the VM to cache compiled versions of code, once it has been run for the first time? Especially with module metadata, the VM could identify recurring uses of the same module and re-use previously compiled code. This would move part of the installation cost to the first usage of an application, but after that, startup times could be automatically improved by some VM trickery, based on module support in the Java runtime, yet without an explicit installation step.

    - Downloaded code cannot be pre-compiled during installation anyway. Was not a lot of this motivated by wanting to speed up JavaFX applications? These will run without being installed before anyway.

    - what does "installation" actually mean to Sun? Do they indeed have something like a global assembly cache in mind? If so, will its behavior be specified so that it can be provided by other VM implenentations? How will a package get into it? Only through Sun- and OS-specific installation procedures?

    Whatever tricks the Sun's Java implementation does behind the scenes to perform its job does not matter as long as it helps performance and is an internal implementation detail. But if Sun should create deployment mechanisms that are specific to their own VM, then this would be rather dangerous, as it would put alternative implementations into disadvantage. What has been going on on the spec-side with Apache Harmony is sad enough, so I really hope that Sun is not adding mechanisms that hinder interoperability with their implementation in an area as important as deployment. (Btw: I have not used Harmony at all so far and I am not associated with any of the various "camps", but as a Java developer I care about the long-term openness and interoperability of Java [as a plattform, not just of Sun's version of it]).

    Somehow, Sun's decision to "do their own thing" with Jigsaw reminds me of what could be described as an extension of the Applet livecycle with the dragable Applets that were introduced during the 6u10 "maintainance" release, or their added support for JNLP-Applets. Shouldn't this stuff have been spec'ed? Yes this did not add publically callable APIs in java.*-Packages, so I guess it does not violate the letter of the JCP, but nonetheless, these are proprietary extensions in inportant areas of the platform and therefore make me rather nervous, despite their technical merit.

    Sorry for the very long comment. Overall, I like what Sun is trying to _achieve_ and I understand that they grow somewhat impatient. Nonetheless, it's important that they strive to keep their implementation as interoperable as possile.

  • Heiko Erhardt says:

    Java has established itself as main implementation language for business-critical systems (i. e. systems having 24×7 requirements).

    Not addressing dynamic module management in a language-level approach to module management in Java 7 is a no-go IMO.

    Also, Java being a cross-platform language, introducing platform-dependent package formats is the wrong way to go.

    The best solution would be to have cross-platform package format, which could be mapped to a native package format by the Java installation at the time of the import of the package (if required at all!). However, this is an under-the-hood process that the package developer should not have to deal with.

    Today, many applications are built upon (or are migrating to) OSGI. They won't abandon this platform once Java 7 comes out. As Augusto said, this is likely to repeat the java.util.logging disaster.

    Please, for the sake of the language and the people who are using it, get yourself together at JSR 294 and

    - build upon proven and mature technology such as OSGI

    - improve and extend it where neccessary

  • Andy Tripp says:

    Like others, I think the new module system is the wrong path to take. In fact, I'm still stuck, not able to agree on the need for it. Looking at MR's reasons:

    * Eliminate classpath complexity

    It's not "eliminated", it's simply reformated from a single, simple CLASSPATH variable to a larger module system.

    * Increase performance. The VM download size can be made smaller

    Download size decrease is not a factor: end users (other than applets) don't download the JRE.

    , and startup time should improve because for small programs (say HelloWorld) unnecessary classes don’t have to be loaded at all.

    unnecessary classes aren't loaded at all today by the classloader.

    Mark showed how he could run HelloWorld by only using 2 modules listed by the new jmod command; jdk.base@7-ea and jdk.boot@7-ea.

    Did Mark show that there were in fact, any less classes loaded using the module system? If so, how could that be? The classloader is supposed to only load classes that are actually used.

    * Enable native integration for installing modules.

    As others have pointed out, this is an inherently bad idea, and goes against the cross-platform nature of Java. The current "bundle all needed jars with each app" approach is the better way.

  • It is proved if want to get higher economy and progress in the agriculture we have to teach men and women farmer equally because she also does work hard in the field with her partner, if it does happen surely the crop production increased and directly you will raise your economy and prosperity as well.

 

Leave a Reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>