Maven integration and Orbeon dependency mess

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Maven integration and Orbeon dependency mess

Mike Ahlers
Hi,

I am not sure if this has been brought up as topic before, so here goes:

When looking at the libraries used, I see alot jars without versioning (sure you can investigate their manifest, but often those aren't correct, i.e. with luck you get the version). Some of the jars are alterered, that is, they contain patched files to support correct functionality (see saxon-8.8, they contain org.orbeon packages and zero saxon packages). And, some of these are renamed to reflect transitive dependencies which leads to more confusement.

In my opinion, this is wrong. Upgrading becomes hard when you're not sure what version you are using.

Another observation is that I find 'conflicting' solutions/dependencies, where all in all, the whole lot becomes bloated? The lib directory is (rounded up) 50Mb.

For example:
- fop vs itext
- dom4j vs jdom
- saxon vs xalan

This leads to the next question, what is minimal required to run Orbeon? Meaning, what do I need to have for a clean and mean Orbeon (without all examples and their required libraries). Less is more?

Any feedback?

Regards,
Mike
Reply | Threaded
Open this post in threaded view
|

Re: Maven integration and Orbeon dependency mess

Alessandro Vernet
Administrator
Mike,

On Tue, May 13, 2008 at 1:25 AM, Mike Ahlers
<[hidden email]> wrote:

>  When looking at the libraries used, I see alot jars without versioning (sure
>  you can investigate their manifest, but often those aren't correct, i.e.
>  with luck you get the version). Some of the jars are alterered, that is,
>  they contain patched files to support correct functionality (see saxon-8.8,
>  they contain org.orbeon packages and zero saxon packages). And, some of
>  these are renamed to reflect transitive dependencies which leads to more
>  confusement.
>
>  In my opinion, this is wrong. Upgrading becomes hard when you're not sure
>  what version you are using.
If you have your own Java code, our recommendation is to use the
"separate deployment" method, so you don't have to worry to much about
what jars are used by Orbeon Forms. This way you can look at the
orbeon.war as one single package, and don't need to worry about the
version of each library we use. Or is it another problem that you
would like to solve?

Alex
--
Orbeon Forms - Web 2.0 Forms, open-source, for the Enterprise
Orbeon's Blog: http://www.orbeon.com/blog/
Personal Blog: http://avernet.blogspot.com/
Twitter - http://twitter.com/avernet


--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: Maven integration and Orbeon dependency mess

Erik Bruchez
Administrator
In reply to this post by Mike Ahlers
> I am not sure if this has been brought up as topic before, so here  
> goes:
>
> When looking at the libraries used, I see alot jars without  
> versioning (sure
> you can investigate their manifest, but often those aren't correct,  
> i.e.
> with luck you get the version).

Mmmh, I don't see a single JAR without proper versioning. I thought we  
took great care to name the JARs to list the proper versions, contrary  
to the original JAR files which in most cases don't have versions at  
all (and to most open source projects I know of, which certainly don't  
follow this practice).

In fact, our own JAR files (orbeon-*) are the exception, but then  
their version if that of the WAR file.

> Some of the jars are alterered, that is,
> they contain patched files to support correct functionality (see  
> saxon-8.8,
> they contain org.orbeon packages and zero saxon packages). And, some  
> of
> these are renamed to reflect transitive dependencies which leads to  
> more
> confusement.
>
> In my opinion, this is wrong. Upgrading becomes hard when you're not  
> sure
> what version you are using.
Say we have to patch Saxon. How would you name the JAR file?

The names we picked are a poor man's way of bundling JAR files  
together. We do this for eXist, MSV, and xerces. Until we have an OSGi-
based infrastructure, this seemed like a not-so-bad mechanism.

What's the solution in your opinion? We are interested, because we  
don't know of any good one.

> Another observation is that I find 'conflicting' solutions/
> dependencies,
> where all in all, the whole lot becomes bloated? The lib directory is
> (rounded up) 50Mb.
>
> For example:
> - fop vs itext
> - dom4j vs jdom
> - saxon vs xalan
>
> This leads to the next question, what is minimal required to run  
> Orbeon?
> Meaning, what do I need to have for a clean and mean Orbeon (without  
> all
> examples and their required libraries). Less is more?
The other day I downloaded a trial of Adobe Photoshop Elements, the  
"lightweight" Photoshop. It was 1.2 GB. In comparison, 50 MB doesn't  
seem like a big issue, even if you deploy your app on your cell  
phone ;-)

But you do raise a good question. I think we established recently that  
JDOM was not needed at this moment. Xalan shouldn't be used either  
anymore, unless you explicitly use the oxf:xalan processor. For the  
rest, it depends on which functionality of Orbeon Forms you use. If  
you don't use the oxf:xsl-fo transformer, the FOP JAR is likely to be  
unneeded, etc.

-Erik

--
Orbeon Forms - Web Forms for the Enterprise Done the Right Way
http://www.orbeon.com/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: Maven integration and Orbeon dependency mess

Mike Ahlers
In reply to this post by Mike Ahlers
Hi Erik,

Well, as a solution you could migrate the build process to adopt Maven. Alot open source projects are doing this (almost all Apache's for example). Recently, Cocoon is embracing this as well (look at their 2.2 release for example, of how they deal with modules).

A good example would be the embedded eXist. Current OPS bundle comes with eXist 1.1.1 while 1.2 is out. Sure you can wait till a new release for OPS comes, with hopefully an upgraded eXist. Or, you can 'open up' the lib folder and replace the jars by hand. Thing is, Maven could this for you automatically and alot more.
Another example is that you can keep the footprint to source code alone, instead of carrying along the required jars in the csv or svn (or whatever scm you use). To extend the example of eXist, it comes embedded, which is easy for quick proof of concepting and developing in general. However, in production like settings (especially scalable ones) you certainly do not want to have exist running embedded, but isolated. This requires alot of handwork to strip it out, or carry it as unused luggage.

With Maven and some configuration you can add modules that could be part of the OPS, like examples included or mimimal, together with the needed configuration files. That all can be solved.

The solution of treating OPS as an 'isolated' WAR is appealing, but is yet another entity that needs to be managed. It is good practise to keep a single website having a single WAR file. In real life, business applications run in an application server (like JBoss, Websphere, GlassFish, name the lot), with an apache server in front acting as proxy. Alot of virtual hosting is done as well. Having more WAR files means more configuration, and raises the bar of using OPS, and adds to the EAR deployment.

As for naming and versioning Jar files, that too is standarized. Maven uses a repository for that. As for Orbeon specific libraries, you can easily group them and still have unique names to identify their purposes. This one of the reasons or solutions that Maven brings. I could go and if you want me, I'll do. Heck, I am willing to contribute.

Oh, have I mentioned that Maven integrates nicely with Contineous Integration tools such as Continuum?

Anyways, this topic addresses alot more issues, but in the end of the day, it benefits us all, at least thats my opinion.

Regards,
Mike
Reply | Threaded
Open this post in threaded view
|

Re: Re: Maven integration and Orbeon dependency mess

Alessandro Vernet
Administrator
Mike,

On Wed, May 14, 2008 at 5:55 AM, Mike Ahlers
<[hidden email]> wrote:
> Well, as a solution you could migrate the build process to adopt Maven. Alot
> open source projects are doing this (almost all Apache's for example).
> Recently, Cocoon is embracing this as well (look at their 2.2 release for
> example, of how they deal with modules).
> [...]

You have a lot of good ideas, but it is unlikely that we will have the
time here to experiment with Maven in the near future. So if this is
something that you consider dear to your heart, I would recommend you
give it a shot yourself, and of course this is something that we would
be interested in incorporating in the code if you wish to contribute
back your changes. If there is anything we can do to help with this,
please do post here and we'll do our best to help.

Alex
--
Orbeon Forms - Web 2.0 Forms, open-source, for the Enterprise
Orbeon's Blog: http://www.orbeon.com/blog/
Personal Blog: http://avernet.blogspot.com/
Twitter - http://twitter.com/avernet


--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: Re: Maven integration and Orbeon dependency mess

Erik Bruchez
Administrator
In reply to this post by Mike Ahlers
Mike,

Thanks for your feedback.

I am personally very sensitive to issues of dependencies. Ten years  
ago, I wrote for a company an entire build system which handled  
packages, repository, and dependencies, for a large product written in  
Java and C++.

But I would say that we at Orbeon are wary of adding complexity to the  
build system. Maven certainly does add complexity: you have to have a  
repository, configuration files, executable, and you have to learn a  
new tool. Also, I may be misinformed, but I remember reading  
discussions about how Maven 2 was causing lots of trouble. I also have  
some experience with Ivy and wow, does that add complexity.

A few thoughts below about what problems are worth solving:

* Distribution size: not an issue at all for me: 50 MB is small.

* Extra baggage for things you don't need: see above. Just a few megs  
don't matter.

* Checking JAR files into CVS or SVN: again, not a problem. The  
benefit is that everything is right there where you need it, and  
without any setup.

* Updates to third-party software: unfortunately we often have to  
patch those, including eXist. Maven wouldn't solve this problem  
really: we still have to take eXist, fix it up, and integrate it,  
whether in our CVS or in a Maven repository.

Regarding eXist, you are right in that in production, you will likely  
deploy eXist as a separate WAR, or even as a separate server entirely.  
So it does not matter at all that Orbeon Forms comes with its own  
(possibly outdated at some point) version of eXist. Just point Orbeon  
to your separate eXist and you are all set. I wouldn't care about the  
extra baggage: again it's only a few megs.

Really for me the biggest issue worth fixing is JAR conflicts. This is  
the Java developer's hell. But I would rather handle this with  
something like OSGi (also JSR 291, see also JSR 277) and bundles/
modules which, at runtime, clearly define which piece of code depends  
on which other piece of code, and isolates components from each other.  
Whether you then use Maven or something else to store these modules is  
not very relevant. But Maven alone doesn't solve the issue of conflict  
between your version of eXist vs. my own, patched version of eXist, if  
they use the same Java packages. Something like OSGi does.

Am I making sense?

-Erik


On May 14, 2008, at 5:55 AM, Mike Ahlers wrote:

>
> Hi Erik,
>
> Well, as a solution you could migrate the build process to adopt  
> Maven. Alot
> open source projects are doing this (almost all Apache's for example).
> Recently, Cocoon is embracing this as well (look at their 2.2  
> release for
> example, of how they deal with modules).
>
> A good example would be the embedded eXist. Current OPS bundle comes  
> with
> eXist 1.1.1 while 1.2 is out. Sure you can wait till a new release  
> for OPS
> comes, with hopefully an upgraded eXist. Or, you can 'open up' the lib
> folder and replace the jars by hand. Thing is, Maven could this for  
> you
> automatically and alot more.
> Another example is that you can keep the footprint to source code  
> alone,
> instead of carrying along the required jars in the csv or svn (or  
> whatever
> scm you use). To extend the example of eXist, it comes embedded,  
> which is
> easy for quick proof of concepting and developing in general.  
> However, in
> production like settings (especially scalable ones) you certainly do  
> not
> want to have exist running embedded, but isolated. This requires  
> alot of
> handwork to strip it out, or carry it as unused luggage.
>
> With Maven and some configuration you can add modules that could be  
> part of
> the OPS, like examples included or mimimal, together with the needed
> configuration files. That all can be solved.
>
> The solution of treating OPS as an 'isolated' WAR is appealing, but  
> is yet
> another entity that needs to be managed. It is good practise to keep a
> single website having a single WAR file. In real life, business  
> applications
> run in an application server (like JBoss, Websphere, GlassFish, name  
> the
> lot), with an apache server in front acting as proxy. Alot of virtual
> hosting is done as well. Having more WAR files means more  
> configuration, and
> raises the bar of using OPS, and adds to the EAR deployment.
>
> As for naming and versioning Jar files, that too is standarized.  
> Maven uses
> a repository for that. As for Orbeon specific libraries, you can  
> easily
> group them and still have unique names to identify their purposes.  
> This one
> of the reasons or solutions that Maven brings. I could go and if you  
> want
> me, I'll do. Heck, I am willing to contribute.
>
> Oh, have I mentioned that Maven integrates nicely with Contineous
> Integration tools such as Continuum?
>
> Anyways, this topic addresses alot more issues, but in the end of  
> the day,
> it benefits us all, at least thats my opinion.
>
> Regards,
> Mike
> --
> View this message in context: http://www.nabble.com/Maven-integration-and-Orbeon-dependency-mess-tp17203540p17230401.html
> Sent from the ObjectWeb OPS - Users mailing list archive at  
> Nabble.com.
>
>
> --
> You receive this message as a subscriber of the [hidden email]  
> mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> OW2 mailing lists service home page: http://www.ow2.org/wws
--
Orbeon Forms - Web Forms for the Enterprise Done the Right Way
http://www.orbeon.com/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws
Reply | Threaded
Open this post in threaded view
|

Re: Re: Maven integration and Orbeon dependency mess

Mike Ahlers
Hi Erik and Alex,

Well, everything new does look complex? Does it? Maven is in fact a replacement for ant, and both use java. So, I wouldn't say you would need 'new' tools or executables. Sure, you have to install maven, but for ant you'd need a jar too! If something that can grow complex it would be configuring the build scripts for ant, especially regarding the classpath (been there done that), and dealing with multiple artifacts. Maven is far more better arranged. Ivy or anthill, is all wrapped around Ant. I have no experience with those, but knowing it is based on Ant says enough (for me). I forgot to mention Codehaus, which is another good reference. They do all their open source projects in maven. In fact, maven is incubated there.

You say you've read problems with Maven 2, well, I can tell you those are over! At start there were some child-diseases and lack of plugins, the first are solved and the other is in abundance (and standarized). And if needed, easy to write yourself.

Yes, dependencies are sensitive. Especially when you're having multiple projects and usually have copies of jars all over around. Thats where the repository comes in. You don't need to have a repository, it already is there! Maven uses a remote repository (or plural, using mirrors) and a local one. The local one is on your local development environment, and all your projects use that (when using Maven). Thats just one advantage of Maven, a neat organized layout of jars. The other advantages weigh heavy as well, but I am going to keep this post small. For more information I really need to refer to the maven site itself.

Osgi is the way to go, I agree here. But thats just a different specification which could co-exist easily (judging by glance).

You mention, oh, its just a few Mb's. But it is a few Mb's nonetheless. Not only in size, but likely in memory as well. Speaking of source, binaries should not belong in there, they are derived from it. The only setup is a single file called pom.xml, compared to the build.xml. Upgrading third party software is changing a single digit (or two when its major)!

What I can do, is to provide an example, of how Maven would deal with the Orbeon build. Not only the pom.xml but also the different layout of source files (which has a small impact compared to the situation now). For that matter, I could setup a mirror, with sources matching the standard followed by Maven and the build would deliver the same artifact, namely the ops-3.6.0.war. From there we could discuss it further?

If I were to proceed on this, I'd require feedback regarding the current source layout when looking at:
http://cvs.forge.objectweb.org/cgi-bin/viewcvs.cgi/ops/
As in, what would be the core and what is not needed?

Regards,
Mike
Reply | Threaded
Open this post in threaded view
|

Re: Re: Re: Maven integration and Orbeon dependency mess

Erik Bruchez
Administrator
Hi Mike,

Really, we appreciate your contribution, but we just want to make sure  
that we are actually solving problems.

So if you can do this, we'd be happy to look at it. If you go ahead  
with this, let us know what information you need from us.

-Erik

On May 15, 2008, at 9:19 AM, Mike Ahlers wrote:

>
> Hi Erik and Alex,
>
> Well, everything new does look complex? Does it? Maven is in fact a
> replacement for ant, and both use java. So, I wouldn't say you would  
> need
> 'new' tools or executables. Sure, you have to install maven, but for  
> ant
> you'd need a jar too! If something that can grow complex it would be
> configuring the build scripts for ant, especially regarding the  
> classpath
> (been there done that), and dealing with multiple artifacts. Maven  
> is far
> more better arranged. Ivy or anthill, is all wrapped around Ant. I  
> have no
> experience with those, but knowing it is based on Ant says enough  
> (for me).
> I forgot to mention Codehaus, which is another good reference. They  
> do all
> their open source projects in maven. In fact, maven is incubated  
> there.
>
> You say you've read problems with Maven 2, well, I can tell you  
> those are
> over! At start there were some child-diseases and lack of plugins,  
> the first
> are solved and the other is in abundance (and standarized). And if  
> needed,
> easy to write yourself.
>
> Yes, dependencies are sensitive. Especially when you're having  
> multiple
> projects and usually have copies of jars all over around. Thats  
> where the
> repository comes in. You don't need to have a repository, it already  
> is
> there! Maven uses a remote repository (or plural, using mirrors) and  
> a local
> one. The local one is on your local development environment, and all  
> your
> projects use that (when using Maven). Thats just one advantage of  
> Maven, a
> neat organized layout of jars. The other advantages weigh heavy as  
> well, but
> I am going to keep this post small. For more information I really  
> need to
> refer to the maven site itself.
>
> Osgi is the way to go, I agree here. But thats just a different
> specification which could co-exist easily (judging by glance).
>
> You mention, oh, its just a few Mb's. But it is a few Mb's  
> nonetheless. Not
> only in size, but likely in memory as well. Speaking of source,  
> binaries
> should not belong in there, they are derived from it. The only setup  
> is a
> single file called pom.xml, compared to the build.xml. Upgrading  
> third party
> software is changing a single digit (or two when its major)!
>
> What I can do, is to provide an example, of how Maven would deal  
> with the
> Orbeon build. Not only the pom.xml but also the different layout of  
> source
> files (which has a small impact compared to the situation now). For  
> that
> matter, I could setup a mirror, with sources matching the standard  
> followed
> by Maven and the build would deliver the same artifact, namely the
> ops-3.6.0.war. From there we could discuss it further?
>
> If I were to proceed on this, I'd require feedback regarding the  
> current
> source layout when looking at:
> http://cvs.forge.objectweb.org/cgi-bin/viewcvs.cgi/ops/
> As in, what would be the core and what is not needed?
>
> Regards,
> Mike
> --
> View this message in context: http://www.nabble.com/Maven-integration-and-Orbeon-dependency-mess-tp17203540p17257051.html
> Sent from the ObjectWeb OPS - Users mailing list archive at  
> Nabble.com.
--
Orbeon Forms - Web Forms for the Enterprise Done the Right Way
http://www.orbeon.com/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws