'. '

OSGiAndNetBeans

From APIDesign

(Difference between revisions)
Jump to: navigation, search
(Netigso)
Current revision (08:18, 21 June 2012) (edit) (undo)
 
(25 intermediate revisions not shown.)
Line 1: Line 1:
== [[OSGi]] and [[NetBeans Runtime Container]] can now co-exist! ==
== [[OSGi]] and [[NetBeans Runtime Container]] can now co-exist! ==
-
Those who read [[Chapter 15]] of [[TheAPIBook]] know that I am not afraid to create [[bridge]]s. I'd rather spend time to create a [[bridge]] between two [[API]]s to allow easily [[upgradability]] and [[co-existence]] between old and new version, than organize a big bang rewrite that stops the world for few weeks/months and introduces new, shiny [[API]]. I am always afraid to find out that the new shiny replacement is buggy and incomplete (which is quite common).
+
Those who read [[Chapter 15]] of [[TheAPIBook]] know that I am not afraid to create [[bridge]]s. I'd rather spend time to create a [[bridge]] between two [[API]]s to allow easily [[upgradability]] and [[co-existence]] between old and new version, than organize a [[Big Bang]] rewrite that stops the world for few weeks/months and introduces new, shiny [[API]]. I am always afraid to find out that the new shiny replacement is buggy and incomplete (which is quite common).
I am not afraid to write bridges. Still it took me a while before I created this [[Bridge|alien bridge]]. Not that it would be too much work (at the end it did not took more than few weeks of my time), but everyone (and aspecially [[OSGi]] guys) were always claiming that writing [[bridge]] between two [[module system]]s is close to impossible. Also there was an attempt to provide yet another [[module system]] for [[Java]] (JSR 277 and friends) which always delayed my attempts to start.
I am not afraid to write bridges. Still it took me a while before I created this [[Bridge|alien bridge]]. Not that it would be too much work (at the end it did not took more than few weeks of my time), but everyone (and aspecially [[OSGi]] guys) were always claiming that writing [[bridge]] between two [[module system]]s is close to impossible. Also there was an attempt to provide yet another [[module system]] for [[Java]] (JSR 277 and friends) which always delayed my attempts to start.
Line 11: Line 11:
For a while I considered an option of sneaking [[OSGi]] under the [[NetBeans]] module system (especially [[OSGi]] guys were always suggesting this as the right alternative, not surprisingly). However this would be horribly incompatible solution - an earthquake for everyone who already wrote anything on top of [[NetBeans Runtime Container]]. Moreover I found out that there are some aspects in [[Dependencies]] between [[NetBeans]] modules that cannot be easily expressed in [[OSGi]]. Basically 90% of both [[module system]]s are quite similar, but the devil lies in the details. Some things just cannot be mapped naturally (see Richard S. Hall's excellent yet complex [http://wiki.netbeans.org/OSGiAndNetBeansMetadataMapping 1:1 mapping]).
For a while I considered an option of sneaking [[OSGi]] under the [[NetBeans]] module system (especially [[OSGi]] guys were always suggesting this as the right alternative, not surprisingly). However this would be horribly incompatible solution - an earthquake for everyone who already wrote anything on top of [[NetBeans Runtime Container]]. Moreover I found out that there are some aspects in [[Dependencies]] between [[NetBeans]] modules that cannot be easily expressed in [[OSGi]]. Basically 90% of both [[module system]]s are quite similar, but the devil lies in the details. Some things just cannot be mapped naturally (see Richard S. Hall's excellent yet complex [http://wiki.netbeans.org/OSGiAndNetBeansMetadataMapping 1:1 mapping]).
-
Thus I decided to resolve to writing [[bridge]]. The basic idea (as shown in [[Chapter 15]]) is to let two independent [[API]]s to live along each other and bridge information between them in a way that allows them to talk to each other.
+
Thus I decided to resolve to writing a [[bridge]]. The basic idea (as shown in [[Chapter 15]]) is to let two independent [[API]]s to live along each other and bridge information between them in a way that allows them to talk to each other.
-
Thus [[Netigso]] runs two containers in the same [[Java]] virtual machine, along each other. One is the regular [[NetBeans Runtime Container]] and the second is [[Felix]] - an [[OSGi]] implementation written by Richard Hall. The [[bridge]] goes through all registered [[JAR]] files and decides whether they represent an [[OSGi]] or [[NetBeans]] [[JAR]] (unsurpisingly both systems use [[JAR]]s as basic packaging units and also store additional informations about the [[JAR]]s in their manifest). Each [[JAR]] is then passed to the right container to receive appropriate runtime environment.
+
Thus [[Netigso]] runs two containers in the same [[Java]] virtual machine, along each other. One is the regular [[NetBeans Runtime Container]] and the second is [[Felix]] - an [[OSGi]] implementation written by [[Richard Hall]]. The [[bridge]] goes through all registered [[JAR]] files and decides whether they represent an [[OSGi]] or [[NetBeans]] [[JAR]] (unsurpisingly both systems use [[JAR]]s as basic packaging units and also store additional informations about the [[JAR]]s in their manifest). Each [[JAR]] is then passed to the right container to receive appropriate runtime environment.
-
Moreover each [[JAR]] as a ''shadow'' - a fake pair registered in the other container - that just sits there and is ready to do bridging. What kind of bridging? Well, mostly class or resource loading. As soon as there is [[NetBeans]] module that has module dependency (aka requires bundle) on some [[OSGi]] bundle, the ''shadow'' associated with that bundle gets activated. The [[NetBeans]] module then delegates all class or resource loading to that bundle. Similiarly, if there is a [[NetBeans]] module exposing some public packages, any [[OSGi]] bundle can import them which activates the ''shadow'' bundle associated with that module. The activation is then intercepted and alternative bundle's loading mechanism is injected into it, that delegates all resource loading to the [[NetBeans]] module.
+
Moreover each [[JAR]] has a ''shadow'' - a fake peer registered in the other container - that just sits there and is ready to do bridging. What kind of bridging? Well, mostly class or resource loading. As soon as there is [[NetBeans]] module that has module dependency (aka requires bundle) on some [[OSGi]] bundle, the ''shadow'' associated with that bundle gets activated. The [[NetBeans]] module then delegates all class or resource loading to that bundle. Similiarly, if there is a [[NetBeans]] module exposing some public packages, any [[OSGi]] bundle can import them which activates the ''shadow'' bundle associated with that module. The activation is then intercepted and alternative bundle's loading mechanism is injected into it, that delegates all resource loading to the [[NetBeans]] module.
=== Compatibility ===
=== Compatibility ===
Line 21: Line 21:
The beauty of this solution is that it remains fully compatible for both worlds. [[OSGi]] bundles existing so far see exactly the environment they are used to (e.g. [[Felix]]). [[NetBeans]] modules written so far are also managed the same way they used to be until now - e.g. by [[NetBeans Runtime Container]]. This means absolute [[BackwardCompatibility]].
The beauty of this solution is that it remains fully compatible for both worlds. [[OSGi]] bundles existing so far see exactly the environment they are used to (e.g. [[Felix]]). [[NetBeans]] modules written so far are also managed the same way they used to be until now - e.g. by [[NetBeans Runtime Container]]. This means absolute [[BackwardCompatibility]].
-
Only when people start to write new [[NetBeans]] modules or [[OSGi]] bundles that have dependencies on bundles and modules, we are entering the new world, world of cooperation. However as this world never existed before, there are no compatibility requirements. There is no thread to [[BackwardCompatibility]]. Things obviously need to work, but whatever works in the first release version, will define the actual [[Amoeba Model|amoeba shape]] of the whole symbiosis. This is much safer than trying to disassemble [[NetBeans Runtime Container]] and rebuilt it on top of plain [[OSGi]].
+
Only when people start to write new [[NetBeans]] modules or [[OSGi]] bundles that have dependencies on bundles and modules, we are entering the new world, world of cooperation. However as this world never existed before, there are no compatibility requirements. There is no threat to [[BackwardCompatibility]]. Things obviously need to work, but whatever works in the first release version, will define the actual [[Amoeba Model|amoeba shape]] of the whole symbiosis. This is much safer than trying to disassemble [[NetBeans Runtime Container]] and rebuilt it on top of plain [[OSGi]].
Moreover this opens door to two directions. Over time, as people find out that [[OSGi]] is the way to go, they can replace individual [[NetBeans]] modules with bundles offering the same [[API]], just packaged as [[OSGi]] bundles. Or, if people find out that [[OSGi]] has no future, they may rewrite all the bundles to [[NetBeans]] modules and run the whole system easily in [[Netigso]] ;-)
Moreover this opens door to two directions. Over time, as people find out that [[OSGi]] is the way to go, they can replace individual [[NetBeans]] modules with bundles offering the same [[API]], just packaged as [[OSGi]] bundles. Or, if people find out that [[OSGi]] has no future, they may rewrite all the bundles to [[NetBeans]] modules and run the whole system easily in [[Netigso]] ;-)
-
This exactly shows why [[API]] are like [[DiamondsVsStars|stars]] and that they can be slowly and fully compatible be sent into a blackhole. As soon as people stop to use one of the module systems, the whole parallel execution will no longer be necessary. Only its half will be needed during runtime, the rest may stay disabled (just in case some particle of the vanished module system escapes from its blackhole).
+
This exactly shows why [[API]]s in modular systems can be seen as [[DiamondsVsStars|stars]]. They can be slowly and fully compatibly sent into a '''blackhole'''. As soon as people stop to use one of the [[module system]]s, the whole parallel execution will no longer be necessary. Only its half will be activated during runtime, the rest may stay disabled (just in case some particle of the vanished [[module system]] escapes from its blackhole and wants to get executed in proper runtime environment).
=== Tricks and Tooling ===
=== Tricks and Tooling ===
-
Writing the bridge between these two ''aliens'' was not that complicated. It was easy on the side of [[NetBeans]], as I have full right to do changes into the code base to open backdoors for integration of [[Netigso]]. But not many changes were necessary anyway, the [[NetBeans Runtime Container]] already had a '''ModuleFactory''' which allowed other module provides to be plugged in.
+
Writing the bridge between these two ''aliens'' was not that complicated. It was easy on the side of [[NetBeans]], as I have full right to do changes into the code base to open backdoors for integration of [[Netigso]]. But not many changes were necessary anyway, the [[NetBeans Runtime Container]] already had a '''ModuleFactory''' concept which allowed other module providers to be plugged in.
-
The changes on the [[Felix]] side were slightly harder to do, as I did not want to modify the code base. But three reflection calls made the trick. I still need to turn these into a patch and convince Richard Hall to accept them into [[Felix]] (not [[OSGi]]) [[API]] for some future version of [[Felix]]. If the [[OSGi]] aliance accepts similar resource provider injection [[API]] in future, the [[Netigso]] could then run with any [[OSGi]] container.
+
The changes on the [[Felix]] side were slightly harder to do, as I did not want to modify the code base. But three reflection calls made the trick. I still need to turn these into a patch and convince [[Richard Hall]] to accept them into [[Felix]] (not [[OSGi]]) [[API]] for some future version of [[Felix]]. If the [[OSGi]] aliance accepts similar resource provider injection [[API]] in future, the [[Netigso]] could then run with any [[OSGi]] container.
[[Image:NetigsoTooling.png]]
[[Image:NetigsoTooling.png]]
Surprisingly the biggest amount of work was in the tooling space. Basically I needed to convince the [[NetBeans]] IDE ''apisupport'' modules to deal with [[OSGi]] bundles. Whenever they expected or generated [[NetBeans]] module, I needed to provide a branch in code to also understand the ''Bundle-SymbolicName'' and similar attributes. This was a lot of work and although the system seems to work acceptably right now, not everything is finished. More work will be needed to polish it into first class citizen.
Surprisingly the biggest amount of work was in the tooling space. Basically I needed to convince the [[NetBeans]] IDE ''apisupport'' modules to deal with [[OSGi]] bundles. Whenever they expected or generated [[NetBeans]] module, I needed to provide a branch in code to also understand the ''Bundle-SymbolicName'' and similar attributes. This was a lot of work and although the system seems to work acceptably right now, not everything is finished. More work will be needed to polish it into first class citizen.
 +
 +
On the other hand: Do you know how big difference is between [[NetBeans]] module and [[OSGi]] bundle in source form? Minimal. One line. By default [[NetBeans]] module projects have a skeletal ''manifest.mf'' file and it contains:
 +
 +
<source lang="bash">
 +
OpenIDE-Module: org.apidesign.your.module
 +
# plus some additional tags
 +
</source>
 +
 +
all that one has to do in [[Netigso]] to convert such project into [[OSGi]] bundle is to change that line to:
 +
 +
<source lang="bash">
 +
Bundle-SymbolicName: org.apidesign.your.module
 +
</source>
 +
 +
then the build system (based on [[Ant]]) ''gets it'' and instead of [[NetBeans]] module [[JAR]] generates [[OSGi]] [[JAR]]. No additional change needed, all the other manifest meta-data are inferred from already existing [[NetBeans]] module project information. Another indication how similar these two [[module system]]s are.
 +
 +
=== [[NetBeans]] and [[Equinox]] ===
 +
 +
Some people asked for support of [[Equinox]] instead of [[Felix]] in the [[Netigso]] project. Good news are that [[Netbinox|progress]] has been made and there is now [[Netbinox]] subproject available for fans of [[Equinox]] [[OSGi]] container.
=== Resources ===
=== Resources ===
-
* [http://wiki.netbeans.org/OSGiAndNetBeans Dev page]...
+
* List of [http://netbeans.org/bugzilla/buglist.cgi?query_format=advanced&product=platform&component=Netigso&bug_status=NEW&bug_status=STARTED&bug_status=REOPENED bugs] - report [https://netbeans.org/bugzilla/enter_bug.cgi?product=platform&component=Netigso new bug]!
-
* [http://hg.netbeans.org/netigso/ Repository] with sources.
+
* [http://lists.apidesign.org/mailman/listinfo/netigso mailing list] - subscribe, view archives, etc.
 +
* [[netbeans:OSGiAndNetBeans|Dev page]]...
 +
* Sources are part of standard [[NetBeans]] Hg [http://hg.netbeans.org/main-silver/ repository] since release 6.9.
* [http://hudson.apidesign.org/hudson/job/netigso/ Builds] made daily
* [http://hudson.apidesign.org/hudson/job/netigso/ Builds] made daily
== Legal Issues ==
== Legal Issues ==
-
An interesting aspect of the whole [[Netigso]] effort is its legal implications. I know US is (much more than Europe) full of [[wikipedia::Intellectual_property|IP]] issues. I am not in the inner circle to know, but the strong [[wikipedia::Sun_Microsystems|Sun]]'s pushback towards accepting [[OSGi]] as the default module system for [[Java]] (e.g. all the attempts with JSR 277, JSR 294, Jigsaw, etc.) might be motivated by fear that as soon as [[OSGi]] is essential part of [[Java]], the [[wikipedia::IBM|IBM]] knocks on the door and says: "Hey, do you know that by using [[OSGi]] you are violating 137 our patents? Shall we bring the course to court or will you give the control over [[Java]] to us without such hassle?"
+
The following paragraphs contain just my personal speculations. I know that at least one [[OSGi]] guy does not find them amusing (which I learned [[Talk:OSGiAndNetBeans#Neil_Bartlett_said_...|here]]). Also now I know that these opinions do not match those of my [[Sun|employer]] (as I have found out via other channels). Take it as oversimplification of complex and not easily understandable actions seen from the other side of Atlantic ocean.
 +
 
 +
An interesting aspect of the whole [[Netigso]] effort is its legal implications. I know US is (much more than Europe) full of [[wikipedia::Intellectual_property|IP]] issues. I am not in the inner circle to know, but the strong [[Sun]]'s pushback towards accepting [[OSGi]] as the default module system for [[Java]] (e.g. all the attempts with JSR 277, JSR 294, [[Jigsaw]], etc.) might be motivated by fear that as soon as [[OSGi]] is essential part of [[Java]], the [[wikipedia::IBM|IBM]] knocks on the door and says: "Hey, do you know that by using [[OSGi]] you are violating 137 our patents? Shall we bring the course to court or will you give the control over [[Java]] to us without such hassle?"
 +
 
 +
As [[NetBeans Runtime Container]] has been working sooner than [[OSGi]] finished its (first usable) specification and as [[Netigso]] clearly shows that there is 1:1 mapping for most of the concepts in both systems, it is now possible to challenge the patents because there is a [[wikipedia::prior art|prior art]]. No need to be afraid of bringing [[module system]] into [[Java]], everything has already been invented by [[NetBeans]] and has been available for free (without any patents) for last decade.
-
As [[NetBeans Runtime Container]] has been working sooner than [[OSGi]] finished its (first usable) specification and as [[Netigso]] clearly shows that there is 1:1 mapping for most of the concepts in both systems, it is now possible to challenge the patents because there is a [[wikipedia::prior art|prior art]]. No need to be afraid of bringing [[module system]] into [[Java]], everything has already be invented by [[NetBeans]] and has been available for free (without any patents) for last decade.
+
The only concept not invented by [[NetBeans]] is ''package dependency'', e.g. ability to import a package, not the whole module/bundle. But it leads to strange coding practices anyway, so there is no harm if it is omitted. Anyway good luck, bringing modularity to [[Java]]!
-
The only concept not invented by [[NetBeans]] is ''package dependency'', e.g. ability to import a package, not the whole module/bundle. This probably means that whatever module system ends in [[Java]] it shall be based on module/bundle dependencies and not package ones. Good luck, bringing modularity to [[Java]]!
+
<comments/>

Current revision

Contents

OSGi and NetBeans Runtime Container can now co-exist!

Those who read Chapter 15 of TheAPIBook know that I am not afraid to create bridges. I'd rather spend time to create a bridge between two APIs to allow easily upgradability and co-existence between old and new version, than organize a Big Bang rewrite that stops the world for few weeks/months and introduces new, shiny API. I am always afraid to find out that the new shiny replacement is buggy and incomplete (which is quite common).

I am not afraid to write bridges. Still it took me a while before I created this alien bridge. Not that it would be too much work (at the end it did not took more than few weeks of my time), but everyone (and aspecially OSGi guys) were always claiming that writing bridge between two module systems is close to impossible. Also there was an attempt to provide yet another module system for Java (JSR 277 and friends) which always delayed my attempts to start.

Netigso

Last Chrismas break I said to myself: enough! Let's really try to create a bridge between NetBeans Runtime Container and OSGi.

For a while I considered an option of sneaking OSGi under the NetBeans module system (especially OSGi guys were always suggesting this as the right alternative, not surprisingly). However this would be horribly incompatible solution - an earthquake for everyone who already wrote anything on top of NetBeans Runtime Container. Moreover I found out that there are some aspects in Dependencies between NetBeans modules that cannot be easily expressed in OSGi. Basically 90% of both module systems are quite similar, but the devil lies in the details. Some things just cannot be mapped naturally (see Richard S. Hall's excellent yet complex 1:1 mapping).

Thus I decided to resolve to writing a bridge. The basic idea (as shown in Chapter 15) is to let two independent APIs to live along each other and bridge information between them in a way that allows them to talk to each other.

Thus Netigso runs two containers in the same Java virtual machine, along each other. One is the regular NetBeans Runtime Container and the second is Felix - an OSGi implementation written by Richard Hall. The bridge goes through all registered JAR files and decides whether they represent an OSGi or NetBeans JAR (unsurpisingly both systems use JARs as basic packaging units and also store additional informations about the JARs in their manifest). Each JAR is then passed to the right container to receive appropriate runtime environment.

Moreover each JAR has a shadow - a fake peer registered in the other container - that just sits there and is ready to do bridging. What kind of bridging? Well, mostly class or resource loading. As soon as there is NetBeans module that has module dependency (aka requires bundle) on some OSGi bundle, the shadow associated with that bundle gets activated. The NetBeans module then delegates all class or resource loading to that bundle. Similiarly, if there is a NetBeans module exposing some public packages, any OSGi bundle can import them which activates the shadow bundle associated with that module. The activation is then intercepted and alternative bundle's loading mechanism is injected into it, that delegates all resource loading to the NetBeans module.

Compatibility

The beauty of this solution is that it remains fully compatible for both worlds. OSGi bundles existing so far see exactly the environment they are used to (e.g. Felix). NetBeans modules written so far are also managed the same way they used to be until now - e.g. by NetBeans Runtime Container. This means absolute BackwardCompatibility.

Only when people start to write new NetBeans modules or OSGi bundles that have dependencies on bundles and modules, we are entering the new world, world of cooperation. However as this world never existed before, there are no compatibility requirements. There is no threat to BackwardCompatibility. Things obviously need to work, but whatever works in the first release version, will define the actual amoeba shape of the whole symbiosis. This is much safer than trying to disassemble NetBeans Runtime Container and rebuilt it on top of plain OSGi.

Moreover this opens door to two directions. Over time, as people find out that OSGi is the way to go, they can replace individual NetBeans modules with bundles offering the same API, just packaged as OSGi bundles. Or, if people find out that OSGi has no future, they may rewrite all the bundles to NetBeans modules and run the whole system easily in Netigso ;-)

This exactly shows why APIs in modular systems can be seen as stars. They can be slowly and fully compatibly sent into a blackhole. As soon as people stop to use one of the module systems, the whole parallel execution will no longer be necessary. Only its half will be activated during runtime, the rest may stay disabled (just in case some particle of the vanished module system escapes from its blackhole and wants to get executed in proper runtime environment).

Tricks and Tooling

Writing the bridge between these two aliens was not that complicated. It was easy on the side of NetBeans, as I have full right to do changes into the code base to open backdoors for integration of Netigso. But not many changes were necessary anyway, the NetBeans Runtime Container already had a ModuleFactory concept which allowed other module providers to be plugged in.

The changes on the Felix side were slightly harder to do, as I did not want to modify the code base. But three reflection calls made the trick. I still need to turn these into a patch and convince Richard Hall to accept them into Felix (not OSGi) API for some future version of Felix. If the OSGi aliance accepts similar resource provider injection API in future, the Netigso could then run with any OSGi container.

Image:NetigsoTooling.png

Surprisingly the biggest amount of work was in the tooling space. Basically I needed to convince the NetBeans IDE apisupport modules to deal with OSGi bundles. Whenever they expected or generated NetBeans module, I needed to provide a branch in code to also understand the Bundle-SymbolicName and similar attributes. This was a lot of work and although the system seems to work acceptably right now, not everything is finished. More work will be needed to polish it into first class citizen.

On the other hand: Do you know how big difference is between NetBeans module and OSGi bundle in source form? Minimal. One line. By default NetBeans module projects have a skeletal manifest.mf file and it contains:

OpenIDE-Module: org.apidesign.your.module
# plus some additional tags

all that one has to do in Netigso to convert such project into OSGi bundle is to change that line to:

Bundle-SymbolicName: org.apidesign.your.module

then the build system (based on Ant) gets it and instead of NetBeans module JAR generates OSGi JAR. No additional change needed, all the other manifest meta-data are inferred from already existing NetBeans module project information. Another indication how similar these two module systems are.

NetBeans and Equinox

Some people asked for support of Equinox instead of Felix in the Netigso project. Good news are that progress has been made and there is now Netbinox subproject available for fans of Equinox OSGi container.

Resources

Legal Issues

The following paragraphs contain just my personal speculations. I know that at least one OSGi guy does not find them amusing (which I learned here). Also now I know that these opinions do not match those of my employer (as I have found out via other channels). Take it as oversimplification of complex and not easily understandable actions seen from the other side of Atlantic ocean.

An interesting aspect of the whole Netigso effort is its legal implications. I know US is (much more than Europe) full of IP issues. I am not in the inner circle to know, but the strong Sun's pushback towards accepting OSGi as the default module system for Java (e.g. all the attempts with JSR 277, JSR 294, Jigsaw, etc.) might be motivated by fear that as soon as OSGi is essential part of Java, the IBM knocks on the door and says: "Hey, do you know that by using OSGi you are violating 137 our patents? Shall we bring the course to court or will you give the control over Java to us without such hassle?"

As NetBeans Runtime Container has been working sooner than OSGi finished its (first usable) specification and as Netigso clearly shows that there is 1:1 mapping for most of the concepts in both systems, it is now possible to challenge the patents because there is a prior art. No need to be afraid of bringing module system into Java, everything has already been invented by NetBeans and has been available for free (without any patents) for last decade.

The only concept not invented by NetBeans is package dependency, e.g. ability to import a package, not the whole module/bundle. But it leads to strange coding practices anyway, so there is no harm if it is omitted. Anyway good luck, bringing modularity to Java!

<comments/>

Personal tools
buy