Embedded eXist configuration

Hi Guys

I've developed a fairly extensive form solution using Orbeon, a bit like the DMV example, that uses eXist for storing forms. The solution is slow and I've increased memory to 2gb, removed the widget processing etc. no joy, anyway that's a separate issue, the real problem I have is that eXist will not flush its cache to disk even though I tell it do this every few seconds.

Why is this a problem? Well for some reason the records in memory can be corrupted if something fails.. dropped connection etc. A user can spend a couple of hours filling in a form and then...zap... gone for good!!!

I've been running the system now for a few days and already this has happened twice. After investigation, it seems the cache only seems to be flushed to disk when I stop tomcat... this can't be right... can it?  This is the db-connection detail of my current exist-conf.xml file -  

<db-connection database="native" files="exist-data"
        pageSize="4096" cacheSize="256M" free_mem_min="10">

        <!-- Settings for the database connection pool:
            - sync-period
                defines how often the database will flush its
                internal buffers to disk. The sync thread will interrupt
                normal database operation after the specified number of
                milliseconds and write all dirty pages to disk.
            - min: minimum number of connections to keep alive.
            - max: maximum number of connections allowed.
            sync-period="10000"
        -->
        <pool min="1" max="20" sync-period="10000"/>
</db-connection>

Changing the sync-period does nothing. Is this a problem with eXist or the embedded implementation? I realise I could set up an external eXist database to test but before I go that route is there anything obvious I'm not doing etc?

Any help would be greatly appreciated as my client is losing confidence in the solution, as am I :(

Many thanks
Lee Bamford



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Lee,

You don't say what version of OPS you are using. Are you using 3.0.1?
We have since updated eXist to a more recent snapshot, and the XForms
engine has had numerous enhancements since then.

Also, try not using eXist in embedded mode, but try deploying eXist as
a separate servlet instead. When using XML:DB to access eXist, the
embedded mode is triggered with the use of this URL:

   xmldb:exist:///

If you use a URL such as:

   xmldb:exist://localhost:8888/ops/exist-xmlrpc

then you are no longer in embedded mode. This also allows you to
update to newer versions of eXist easily, as that gives you the
possibility to even deploy eXist in another web app or on another
server if you wish so.

Note that we are more and more accessing eXist directly through its
REST API directly from XForms. We plan to upgrade the DMV forms
example to illustrate this for OPS 3.1. This has the benefit of much
simpler code.

For eXist-specific issues, you may have more luck with the exist-open
mailing-list:

   http://sourceforge.net/mail/?group_id=17691

But this is an important topic, so please be sure to report findings
here.

It would be great also if you communicated to the list other issues
you have encountered along the way. Our goal is to make sure that the
OPS XForms engine is the best XForms solution out there, and we want
to work on fixing the issues that users encounter.

Finally, if you are stuck, don't forget that Orbeon provides a range
of services, from consulting to support contracts.

-Erik

[hidden email] wrote:
 > Hi Guys
 >
 > I've developed a fairly extensive form solution using Orbeon, a bit
 > like the DMV example, that uses eXist for storing forms. The
 > solution is slow and I've increased memory to 2gb, removed the
 > widget processing etc. no joy, anyway that's a separate issue, the
 > real problem I have is that eXist will not flush its cache to disk
 > even though I tell it do this every few seconds.
 >
 > Why is this a problem? Well for some reason the records in memory
 > can be corrupted if something fails.. dropped connection etc. A user
 > can spend a couple of hours filling in a form and then...zap... gone
 > for good!!!
 >
 > I've been running the system now for a few days and already this has
 > happened twice. After investigation, it seems the cache only seems
 > to be flushed to disk when I stop tomcat... this can't be
 > right... can it?  This is the db-connection detail of my current
 > exist-conf.xml file -
 >
 > <db-connection database="native" files="exist-data"
 >         pageSize="4096" cacheSize="256M" free_mem_min="10">
 >
 >         <!-- Settings for the database connection pool:
 >             - sync-period
 >                 defines how often the database will flush its
 >                 internal buffers to disk. The sync thread will interrupt
 >                 normal database operation after the specified number of
 >                 milliseconds and write all dirty pages to disk.
 >             - min: minimum number of connections to keep alive.
 >             - max: maximum number of connections allowed.
 >             sync-period="10000"
 >         -->
 >         <pool min="1" max="20" sync-period="10000"/>
 > </db-connection>
 >
 > Changing the sync-period does nothing. Is this a problem with eXist
 > or the embedded implementation? I realise I could set up an external
 > eXist database to test but before I go that route is there anything
 > obvious I'm not doing etc?
 >
 > Any help would be greatly appreciated as my client is losing
 > confidence in the solution, as am I :(
 >
 > Many thanks
 > Lee Bamford

--
Orbeon - XForms Everywhere:
http://www.orbeon.com/blog/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Hi Erik

Sorry, I'm using 3.0.1 - the latest official release I believe. I would take
a newer version but stability is a concern and I'm not sure which version to
pick up... please advise.

I can look at using eXist externally if that is better. However, my advice
would be to put a note: use at own risk etc. on the existing embedded eXist
examples because I figured (naively) that I wouldn't have this kind of
issue.

Does using the REST require significant code changes? Are there any full
cycle examples? i.e. create, save, update etc?

A general point(s), I think the overall XForms solution is great and I'm
looking to develop many more solutions using this technology. However,
finding information is very difficult especially within the current forum -
a better 'categorised' knowledge base would help. The examples on the site
itself are in general quite good but if you deviate slightly from them then
you're in trouble - especially when it comes to eXist, uploads,
authentication & performance. I'm not a Java developer, my expertise is in
C, C++, php, ASP - I don't want to edit JAVA files, I don't even want to
look at JAVA files - and my understanding having read the information on the
Orbeon  site is that you don't need to be a JAVA expert to use the
framework... not too sure about that claim :)

There may be a need for consultation on bigger projects but first I must
prove the concept on smaller projects - otherwise it's a 'sledge hammer to
crack a nut' approach costs will spiral etc.

Many thanks
Lee Bamford

----- Original Message -----
From: "Erik Bruchez" <[hidden email]>
To: <[hidden email]>
Sent: Wednesday, June 28, 2006 9:47 AM
Subject: Re: [ops-users] Embedded eXist configuration


--------------------------------------------------------------------------------


>
> --
> You receive this message as a subscriber of the [hidden email]
> mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> ObjectWeb mailing lists service home page: http://www.objectweb.org/wws
>




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Lee,

 > Sorry, I'm using 3.0.1 - the latest official release I believe. I
 > would take a newer version but stability is a concern and I'm not
 > sure which version to pick up... please advise.

Yes, the next stable version will be 3.1. Hopefully in late July...

We reckon that there is a risk with nightly builds: most of the time
they work great, but sometimes not. A good strategy with those is to
use them as you develop, report potential issues with them, updating
regularly and making sure you can revert to a previous working nightly
build if you hit one that is broken.

This said, one of our goals is to avoid destabilizing the
code. Sometimes something break, but we fix it as soon as we can. In
fact we use "unstable" builds in several ongoing projects at the
moment. We still call those builds "unstable" so that people don't
have expectations that are too high.

 > I can look at using eXist externally if that is better. However, my
 > advice would be to put a note: use at own risk etc. on the existing
 > embedded eXist examples because I figured (naively) that I wouldn't
 > have this kind of issue.

Well, pretty much any open source software will be at your own risk
unless a company provides a warranty for it ;-)

eXist itself has not yet reached version 1.0. As great as it is, it
will have bugs. As a user of free software, one great way of
contributing, even if you don't write any code, is to try the software
and report bugs.

Our examples are currently just that, examples. Over time, we hope to
show larger examples that are closer to what users work on. Currently,
the closest such thing is the "DMV Forms" example.

 > Does using the REST require significant code changes?

There are code changes, but for the better: you use xforms:submission
to perform all the CRUD operations: create, read, update and delete,
and you do so by using the regular HTTP methods: POST, GET, PUT, and
DELETE. This means that in most cases you can avoid writing XPL, and
your code and architecture become simpler.

 > Are there any full cycle examples? i.e. create, save, update etc?

This will come in the updated "DMV Forms" example that will ship with
3.1. Unfortunately, the example is not updated yet.

 > A general point(s), I think the overall XForms solution is great and I'm
 > looking to develop many more solutions using this technology.

Great!

 > However, finding information is very difficult especially within the
 > current forum - a better 'categorised' knowledge base would
 > help. The examples on the site itself are in general quite good but
 > if you deviate slightly from them then you're in trouble -
 > especially when it comes to eXist, uploads, authentication &
 > performance.

Thanks for a honest comment. We are aware of shortcomings in the
documentation. We had a decent tutorial for 2.8, but we haven't
updated it for 3.0 yet.

As a baby step towards something better, we plan a slight
reorganization of the examples for 3.1 (including the update to the
DMV Forms mentioned above). We also hope to allow developers to get
started faster with OPS, by providing a simple application skeleton
and a short "getting started" guide.

 > I'm not a Java developer, my expertise is in C, C++, php, ASP - I
 > don't want to edit JAVA files, I don't even want to look at JAVA
 > files - and my understanding having read the information on the
 > Orbeon site is that you don't need to be a JAVA expert to use the
 > framework... not too sure about that claim :)

When we write applications with OPS for customers, we usually don't
write a single line of Java code (except when it comes to implementing
new features in OPS or fixing bugs). So we are quite convinced that
this claim stands!

If it doesn't, then we regard it as a failure: we want developers as
much as possible to write applications with XForms and the other XML
technologies. Java code should ideally kick in only for very specific
tasks (writing a specific component for accessing a back-end system,
etc.). The great thing with REST and services is that you can write
your services, such for example as a persistence service, with any
technology you like: PHP, Java, .NET, Ruby, etc.

 > There may be a need for consultation on bigger projects but first I
 > must prove the concept on smaller projects - otherwise it's a
 > 'sledge hammer to crack a nut' approach costs will spiral etc.

Sounds good :-)

Best,

-Erik

--
Orbeon - XForms Everywhere:
http://www.orbeon.com/blog/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws