oberon forms/xrx architecture for demographic/historical data analysis and research?

Hello all,

I am new to OF. I am an experienced relational database analyst and web developer (pre-ajax vintage "classic" html) with past experience of sgml applications using omnimark. I am seeking appropriate open source architecture/tools to develop a flexible and easy to use research tool to be used by a small team of university researchers (demographers and historians). They need to interrogate an evolving and diverse set of data we are capturing about convicts who were transported to Van Diemans Land in Australia in the first half of the 19th Century. Data sources are diverse, and coming to me in drips and drabs e.g. convict conduct registers, indents, births, deaths and marriages etc..  

I want to build a tool which enables our research team to engage with the database as it evolves, participate in people matching, and annotate/document their research findings. The total convict population is about 75,000 and they arrived by ship in groups of 200-400. So each shipload of convicts represents a natural "blocking" factor (for matching of diverse data sources to convicts). XML seems an appropriate choice for the evolving database as we can easily incorporate new datasets. The tool needs to enable them to easily search/select/save groups of convicts (and in future, their families), access aggregates/statistics about the population of interest, and "drill-down" to the individual cases. Smooth and responsive interface flow to and from populations/individuals is important (its really a "prosopography" research project).

Of all the multitude of web development frameworks on offer, I find "XRX" most appealing due to its elegance and simplicity i.e. xforms at the client, with rest interface to xquery enabled datastore server side. The "Range" example in "Xforms Controls" gives me a useful starting point for building an interface to prepared reports and stats, which I can deliver as xml files, about the various convict groups.

Whilst I find the possibilities very very exciting (e.g. access to on the fly charts) I am very worried that ultimately Oberon Forms/Exist XML database will fail to perform adequately with the scale of data involved or lack of UI maturity of xforms. Obviously it is up to me to design an interface and XML database which provides the necessary search functionality in suitably sized presentation chunks to ensure good performance.

I can see that the learning curve for all this is quite non-trivial and it is tempting to stick to mature relational databases and simply allow the team to search/export flat file data to their traditional tools of (desktop) analysis such as Excel, FM Pro and SPSS. But I cannot help but think XRX/XML is "appropriate" technology for the kind of collaborative research we are attempting to undertake.

I am sorry to ramble on so vaguely, but I am seeking some general guidance as to whether, given the apparent emphasis in OF on (short) "business forms", any attempt to build such a domain-specific research data interrogation tool is unwise? Would experienced OF developers be so kind as to offer a few words of encouragement if you think such a RIA data anlysis tool, properly designed/architected, is buildable with this technology? If not, your words of warning could save me very much time and grief.

Any comments would be very much appreciated.

cheers,
ess

Hi ess,

Whilst I am not an expert with Orbeon, I am however experienced with enterprise architectures. So, lets get back at the prime requirements:

- Ease to produce different formats (presentation of data)
- Flexibel data, as it will evolve (dynamic data domain)
- Performance
(- ignoring others)

Obviously, the first stands clear. XML is the way to go, question is, how to store it, and how to manipulate it. As there are other 'engines' out, like Cocoon and Chiba (and surely other smaller ones), Orbeon is most matured regarding to XForms, at least to my opinion. But basically, all you need is a stylesheet to transform the XML into any other specification (html, iso-excel, etc) and a set of controls. So, XForms would be a good candidate.

Storing XML and query-performance is something different. Especially when having alot data (many files) or just huge chunks of single XML files could be a problem. Like SQL, there are many XML databases out there (for the record, OO-databases might be a solution as well, given high through put and suits read-only/reporting type of applications). As for native XML databases, eXist is an open source XML with good reputation or Tamino for commercial (non-open source) ones. While there might be hybrid solutions (SQL databases supporting XML) it boils down to budget, scalability and performance. I'd go native XML here. Either way, with a flexible data domain, xpath/xquery is your friend and key.

Regardless of choice, keep in mind that if scalability is something to keep in mind that you have to deal with it architecture wise. I talk about layers here and perhaps other types of application servers. Keep your database isolated versus embedded.

If client side execution is important (render performance), then there are other options to examine, like XUL (Mozilla) and MXML (flash/flex/air) to mention a few.

Hopefully I gave answers or guidelines and not more questions...

Success,
Mike