Modularization of NetBeans Platform
From APIDesign
Original page is at https://openide.netbeans.org/proposals/arch/modularize.html - the copy of that page follows:
Status: issue 19443 and colors in the text - red means not done yet, yellow means doing it now, green is for done items. Some paragraphs are marked as grey which means they can be finished later in trunk.
Contents |
Abstract
The heart of any application based on NetBeans is build around openide and core which (althrough the overall NetBeans architecture is very modularized and general) are themselves more monolithic than they should be. The current state complicates certain things in infrastructure, makes other hard to achieve and finally contributes to poor module developer experience. The modularization effort described in this document is going to solve these problems and improve current situation.
Requirements
Compile independently independent parts of the application. This increases stability and improves the code base, as it avoids unwanted hacks used by indepenedent parts to work around some kind of communication problem (for example explorer calling into loaders or filesystems to handle some special state).
Independent JavaDoc - the logical pieces of openide has to form independent modules in API reference in the same way as for example project modules do. Then the module writer will not be confused by thinking openide is something special.
Package independent functionality into independent JAR files. This improves the customization - JARs can be removed, added more easily than to do any changes in code. This is needed in order to satisfy (oposite) requirements from different products builds on top of NetBeans platform and given them a bit of freedom in adapting new functionality in favor of deprecated one (for example replacement of loaders, remove of TopManager, replacement of old actions system).
Standard Infrastructure - there is a default infrastructure to build NetBeans module in nbbuild and also in apisupport. Currently it is not used to build openide and various hacks treating openide as something special are spread in unexpected places. By structuring openide parts in standard way we can remove those hacks and safe many infrastructure maintanance nightmares.
Independent JARs shall be converted into real modules and not be statically added to classpath. This allows step by step deprecation of obsoleted functionalities. If packaged as module autoload JAR, such functionality is enabled only when needed (there is a module requesting this deprecated one), thus increasing the compatiblity without preventing future improvements and deprecations.
The heart of NetBeans shall be really small and include only minimum of classes needed to boot the module system. This is the final phase of making JARs independent which gives the configurators of a system absolute freedom in selection of the functionality that they really want to use from NetBeans Platform.
Compile Time Separation
The first step, useful of its own, but definitively necessary for the other requirements to happen is to separate the compilation of opende/src into separate parts and thus ensuring that there are no hidden interdependencies between them. The status of this part is tracked under issue 34758 and is available in branch compile_34758 of openide.
Compile time separation (phase I)
Let's modify the openide/build.xml to compile sources not all at once, but chunk by chunk. This is possible by passing javac an argument "--sourcepath " which disables javac's search for files to be compiled. Benefit: we will understand and separate the openide to pieces without breaking other people ability to check in (work will be done on a branch, but as the only modified file will be build.xml, it will be easy to merge the branch, not as big problem as when doing openide/loaders). CVS separation (phase II)
When phase one is finished, the CVS will be closed for one day and all sources moved to new locations. That will be easy, as the locations will have been defined by the build.xml "chunks".
While moving the sources, we will loose history, but it will remain accessible in the old CVS location, if really needed. Moreover as other moves we did (like loaders) show, the history is usually needed just for one release back, rarely more. So loosing it is not that big pain comparing to current problems that it is going to solve.
The sources will be formated using jalopy to follow coding standards (visit issue 57941 to see the patch that does the move and formating). Reformating usually hurts history, but because we loose it anyway, we can exploit this chance to improve our current mess in source files.
For each openide subproject a standard NetBeans project type will be created, initially creating its JAR in extra cluster, as the current openide/build.xml will stay and package the final openide.jar file as we do now. The main build script in nbbuild will contain new targets for openide subproject, but the all-openide will remain there and will depend on all of the subtargets.
All of this will be integrated to trunk at once during a night to not disturb the majority of developers working on openide.
Independent JavaDoc
Together with the source code CVS changes, the openide javadoc generation will be moved to the separated parts. This will allow us to remove special hack support for openide in nbbuild/javadoctools and will provide consistent view of all NetBeans javadocs to module developers at reference API page . Also the indexing and overview presentation of all javadocs will be allowed to be enhanced as there will now be a common and standard way how a module delivers its API packages and documentation.
As part of this split we have to remove the huge introduction page in current javadoc and move its appropriate sections into individual arch-*.xml documents. If put into arch-what section, they will appear in the reference API page .
The apichanges.xml will have to be separated into individual documents each for one submodule. The spec version of each created module will start at 6.2.
nbbuild/build.properties will be modified to build javadoc from openide submodules and not the whole openide one.
We introduced new naming scheme. Each API can be referenced by using @org-netbeans-code-base-name@ of the module. That is why we need to get rid of the old @MODULE/SUBMODULE@ links in API docs. We will do it by reviewing all javadocs and fixing broken links. Owners has been informed to fix their javadocs.
As soon as all javadocs are fixed, we will change the build-javadoc to verify that the whole javadoc of all modules is consistent and without broken links. Otherwise the build fails. That way we prevent regressions in future.
At some point (not necessarily now), I think we should move e.g. openide/arch/arch-openide-windows.xml to openide/windows/arch.xml. However we also need to clean these up seriously: many of them describe not just the API, but also the impl in core. There should be e.g. a separate core/windows/arch.xml that describes what the impl module does.
Independent JAR files
When every source will have its own new place and the logical parts will be defined the huge openide.jar can be split to smaller parts (e.g. org-openide-filesystems.jar, org-openide-nodes.jar, etc.). Beyond improving the customization by allowing individual JARs to be removed, this "jarification" will be the major step towards turning the JARs into modules.
This is likely a step with very small effect, but huge affect. Every build script refering to openide.jar will have to be updated to refer to all or only appropriate parts of openide-*.jar it needs. Based on the previous work in issue 36494 a FixDependencies ant task was created that makes it easy to replace and reduce dependencies for every projectized module. Just execute ant fix-dependencies and your nbproject/project.xml will be upgraded. Still we need to improve the task to modify only dependencies that are used for compilation, not those that are there only for runtime.
Move fix-dependencies to common.xml module so other applications based on NetBeans and using the apisupport harness can invoke it as well.
Just splitting openide.jar into smaller JARs has potential performance impact. The startup will be slower (last time we tried it was by 10%). The performance guys agreed that this is ok, if it gets resolved by release. And it will because the modularization and miniaturization described later will return the time back to previous values. If not, we will return back to one gigantic openide.jar. Modularization and Miniaturization
When the JARs are separated it is time to turn them into modules. This however requires that there are not hidden mutual dependencies between those JARs as they will be loaded by own classloaders and that the other part of platform the core.jar does not depend on them or can be turn into modules as well.
The ultimate goal of all these efforts is to spit out a miniature part of the core in NetBeans (the module system and few other utilities) and create a modular application platfrom bootstrap.
This requires changes to module system and also to startup sequence that can contribute to its clean up and clarification. The result will be small microkernel bootstrap with nearly no overhead capable of bootstraping any NetBeans application customized to absolute degree.
We already have core/bootstrap which contains classloaders for openide and core, and the plan is to move into it also the module system. Together the openide/util, openide/modules, openide/fs and boot.jar would stay on dynamic or even real classpath and the rest of the openide modules plus core would be turned into real, dynamically loaded modules. The current thinking about the layout of platform6 directory is:
- lib/boot.jar contains what it has now, together with part of what is now org.netbeans.core.modules.* the part that is already marked as being "independent of other NB APIs".
- lib/org-openide-util.jar will next to boot.jar to provide the basic infrastructure used by boot.jar.
- lib/org-openide-modules.jar provides the APIs that the boot.jar implements.
- core/core.jar will be loaded dynamically by the boot.jar and will contain most of the startup and the rest of org.netbeans.core.modules. For its implementation it will need filesystems which will be in the same folder as well.
- core/org-openide-filesystems.jar the filesystem APIs needed for the standard launch code to read module config files and listen on their changes.
- modules/org-openide-nodes.jar and other openide libraries will be turned into real modules.
- modules/org-netbeans-core.jar will also be a module and will contain the rest of core, or if possible just parts of it if it gets split even more (like moving the UI part to core/ui, etc.).
Currently there is core/org-netbeans-swing-plaf.jar in the dynamic classpath. It is not too nice and maybe it is not that necessary. Consider calling its methods later and making it real module. This could be done by adding new methods to CoreBridge, implementing them in CoreBridgeImpl and calling the methods in the plaf library. The core method would then be called after initialization of module system.
Sources should be rearanged to reflect the logical partitioning in the launcher. There should be three sets of classes, each in their own package (we can move them, as they are not part of supported API). Having them in own packages will prevent linkage errors when two classes loaded by different classloaders try to access their package private methods. The sets are:
- independent module system - placed on java classpath in directory lib in boot.jar. All the sources will be hosted in CVS module core/bootstrap and in addition to existing ones, they will also contain the generic part of core module system. The list of these classes is defined in core/build.xml in target nb-modules. They are currently in org.netbeans.core.modules package, but as that package is going to be split in half, those classes shall be repackaged to org.netbeans, so all classes in boot.jar are in the same package.
- NetBeans specific module system - the part of NetBeans launcher that recognizes NetBeans format of modules, knows their locations, shows splash screen with progress and does other NetBeans specific hacks during startup. The list is again defined in core/build.xml as selector nb-modules. They are spread in various packages with entry point in org.netbeans.core.Main. They shall be moved to one separate packages ( org.netbeans.core.startup, org.netbeans.core.startup.modules, org.netbeans.core.startup.layers) to prevent linkage errors with other classes. The choosen name is org.netbeans.core.modules and the name for their final jar file is core/core.jar. The CVS module for these classes is going to be core/startup. The only possible problem is the move of the Main class, as some applications might depend on it, but as it was not part of public API, we should not really care. We should still do grep on our sources to search for usages of org.netbeans.core.Main and org.netbeans.core.Plain and making sure they continue to work.
- The rest of the original core classes will remain in the same places where they are (for now, in future they can be moved more) and will be contained in modules/org-netbeans-core.jar module.
TestModuleDeployer was changed to class with static methods and is kept in org-netbeans-core.jar.
The communication between boot.jar and core.jar is done using property netbeans.mainclass that will point to org.netbeans.modules.Main. The communication between core.jar and rest of the modules is handled thru CoreBridge calls, regular ModuleInstall and later thru RunLevel interface.
The ParseXMLModule would be changed to correctly insert compile time classpath entries for everyone depending on org.openide. Meaning that keeping a dep on "org.openide" would result in compilation against all split JARs (so it would succeed), yet the JAR would be created with the old OpenIDE-Module-IDE-Dependencies as before, thus still producing a runtime warning. We would still need to do a batch conversion of project.xml's for trunk modules to state minimal specific deps.
There is a ModuleAutoDeps still in effect for anyone missing an org.openide dep, since the module system currently assumes the equivalent of "IDE/1 > 0.0" if not otherwise stated, for compatibility. This will have to be changed (and modules which are really libs and do *not* use any org.openide.** will no longer need to state the phony dep). We will have to get rid of the compat trick used in http://www.netbeans.org/download/dev/javadoc/OpenAPIs/org/openide/doc-files/upgrade.html#3.5i-sep but this shouldn't matter much since the modules referred to by the ModuleAutoDeps are long gone.
We should probably get rid of OpenIDE-Module-IDE-Dependencies altogether - deprecate the keyword. A warning will be printed if the tag appears in the module's manifest, the ParseProjectXML will not deal with that tag anymore. Remove all but the last auto dependency in openide/modules. Add URL to the warning about openide ide split.
Sometimes we have used IDE/1 versions to signify changes not in any class libraries, but in the module JAR format itself. We will define special token and use Requires vs. Provides for this. We use:
OpenIDE-Module-Requires: org.openide.modules.ModuleFormat1
and let the module system just automatically provide e.g. org.openide.modules.ModuleFormat1, org.openide.modules.ModuleFormat2, and org.openide.modules.ModuleFormat3 for all modules, if it were currently at rev 3 of the module format.
Performance measurements show that the startup time exhibits less then five percent regression. This was not merge stopper as there are things that can and will be done to improve the startup time: We got rid of org.openide.compat, then org.openide.explorer and org.openide.util were made sealed packages and the startup, also thanks to removal of old vcs modules got under control.
One has to check that autoupdate continues to work, that we can enable, disable module. Disabling of a module is part of commit validation, autoupdating of ant docs with old dependencies seems to work fine, dependencies upgraded, ant help included in our help system.
Unit tests and commit validation has to pass successfully.
Describe startup sequence somewhere, probably in core/arch/arch-launcher.xml.
Rewrite upgrader.jar to use plain swing. Now it relies on JOptionPane.
Follow
