Orbeon Forms community mailing list

Concurrent users

Classic

List

Threaded

7 messages Options

Tom Grahame

Concurrent users

Hi,

Could anyone give me some numbers and types of users they concurrently support using Orbeon.

I'm having trouble getting my Orbeon apps to scale to more than twenty or so concurrent users with page load times of less than 3 seconds. I've tweaked according to performance recommendations and implemented tricks like delayed submissions.

I realise this is an open ended question with many variables and as such precision isn't necessary; a general feel of the sorts and sizes of audiences people are satisfactorily catering for would be great.

Thanks,

Tom

Tim Pizey-2

Re: Concurrent users

Hi Tom,

I have just written some destruction tests.
The program has evolved and so the graphs given below are difficult to
interpret.
I am running both the test and tomcat from within eclipse so there may
well be interaction.
I am running on Ubuntu 10.4.
The code for the test is at
http://code.google.com/p/dsn-chassis/source/browse/trunk/manta/src/test/java/org/cggh/chassis/manta/loadtest/LoadTest.java

A baseline graph might be straight Apache serving a straight HTML file:
https://spreadsheets.google.com/oimg?key=0AibFqy9K5V7OdFZQcVRraDFEWG1kOWRaQjNOVzlQQ3c&oid=2&zx=c4tvsk-kz59qc
which has an outlier at 33ms but is mostly under 10ms.

A complex Orbeon form, on localhost with three hits per rate and a
recovery between hits gave:
https://spreadsheets.google.com/oimg?key=0AibFqy9K5V7OdG1yTTc0UElTQW5qVE12T29MSjJTaWc&oid=2&zx=2set0q-8env7u
which seems to show something like garbage collection episodes, but
gives less than 0.5 seconds up to 400 hits per second.
The same periodic episodes are shown here, with a reduced rest:
https://spreadsheets.google.com/oimg?key=0AibFqy9K5V7OdENUckpEZm15SXY4RnZCQzUwZXgtSHc&oid=2&zx=530dw3-wpa7pc

However I think the recovery period was excessive above
(though the test, set to average 50 hits per rate, crashed our 1gig
Amazon EC2 instance at a rate on 192 hits per second.)

The same complex form, with no recovery period, ie with a steadily
increasing hit rate, gave a more realistic performance:
https://spreadsheets.google.com/oimg?key=0AibFqy9K5V7OdFNsdkFhN3ZpSzJZb1N6Tk40cjlRZFE&oid=2&zx=wi916a-op1jf9
which seems to say that above a hit rate of 10 hits per second the
performance regrades rapidly, leading to no response.

https://spreadsheets.google.com/oimg?key=0AibFqy9K5V7OdC05MXN2bUc5eHZDd2RIbUtEcmpPbXc&oid=2&zx=pxqb66-2n3qgz
shows a steady sub 10ms response all the way to 1000 hits per second
for a very simple (no xpath) page.

For completeness, and for the sake of colleagues on this list, here is a feed:
https://spreadsheets.google.com/oimg?key=0AibFqy9K5V7OdGNGVkR0Q0tlZ3poR0Z0bURWZmpzN3c&oid=2&zx=svprp8-h2sjz3
and an atom entry
https://spreadsheets.google.com/oimg?key=0AibFqy9K5V7OdHlBYUNmOTRNcURPMlRLWU55MjlHeGc&oid=2&zx=kxxsoo-uis126

I hope this is of some help to you.

regards
Tim Pizey

On 29 June 2010 11:28, Tom Grahame <[hidden email]> wrote:

>
> Hi,
>
> Could anyone give me some numbers and types of users they concurrently
> support using Orbeon.
>
> I'm having trouble getting my Orbeon apps to scale to more than twenty or so
> concurrent users with page load times of less than 3 seconds. I've tweaked
> according to performance recommendations and implemented tricks like delayed
> submissions.
>
> I realise this is an open ended question with many variables and as such
> precision isn't necessary; a general feel of the sorts and sizes of
> audiences people are satisfactorily catering for would be great.
>
> Thanks,
>
> Tom
> --
> View this message in context: http://orbeon-forms-ops-users.24843.n4.nabble.com/Concurrent-users-tp2271993p2271993.html
> Sent from the Orbeon Forms (ops-users) mailing list archive at Nabble.com.
>
>
> --
> You receive this message as a subscriber of the [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> OW2 mailing lists service home page: http://www.ow2.org/wws
>
>

--
Tim Pizey
Centre for Genomics and Global Health <http://cggh.org>
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Web: http://pizey.net/~timp
Tel: +44 (0)1865 287532

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws

Alessandro Vernet

Re: Concurrent users

Administrator

In reply to this post by Tom Grahame

Tom,

Yes, this is very much an open question :). See this page on
"performance and sizing":

http://wiki.orbeon.com/forms/doc/developer-guide/admin/performance-and-sizing

Based on this, you can determine what really needs to be optimized in
your case: page loads, Ajax requests, or both. If you have a
contention problem, you'll want to address it first. Otherwise, the
next step is in general to run the app with a single user, and
perform, after heating the VM, the operation you want to optimize with
debug information turned on, as described on:

http://wiki.orbeon.com/forms/doc/developer-guide/xforms-logging#TOC-Development-configuration

Then you can analyze what is happening during a single transaction and
see where the time is being step. Granted, this can look fairly hard
core, but it is quite interesting as well :).

Alex

On Tue, Jun 29, 2010 at 3:28 AM, Tom Grahame
<[hidden email]> wrote:

--
Orbeon Forms - Web forms, open-source, for the Enterprise -
http://www.orbeon.com/
My Twitter: http://twitter.com/avernet

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws

--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet

Tom Grahame

Re: Concurrent users

In reply to this post by Tim Pizey-2

Tim,

Thank you for your contribution. I am indeed finding it difficult to interpret the first two curves, however the third looks remarkably similar to my own, admittedly cruder, test.

To complete my understanding I would be grateful if you could provide me with:
Your definition of 'rate', at least for the third curve, and
A rough approximation of how complex your form is, in terms of numbers of xpath statements for example.

Many thanks,

Tom

Tim Pizey-2

Re: Re: Concurrent users

Hi Tom and all,

Whilst my previous approach was producing some interesting graphs, and
was stressing the machine to breaking point, getting repeatable values
or explaining the results was pretty much impossible, as both rate and
concurrency were varying.

I was only controlling the time between requests, so the number of
concurrent requests could reach the thousands.
I will return to something like this when trying to defend the machine
against DoS attacks, but to address the question of how many separate
users can we support I have changed the methodology.

METHODOLOGY

The url under test is a complex form: a separated into xforms model
file, xforms input file, an xsd and 65 vocabulary files.
It contains multiple relevancy clauses and invokes one widget.

The code can be browsed here:
http://code.google.com/p/dsn-chassis/source/browse/trunk/manta/#manta/war/WEB-INF/resources/apps/common

The tests were carried out on an elderly 3gHz P4 with 3g of memory.
The tests were to localhost, so there was no network latency.

The test runner ab, the Apache Benchmarking tool, is not written in
java and is fairly small so is not thought to have interacted with the
system under test, in the way that running my earlier tests from
within Eclipse appeared to.
Using ab I am able to make 100 requests with a varying concurrency:

ab -c 1 -n 100 -e questionnaire_1536_1.csv
http://localhost:8080/manta/questionnaire/
ab -c 2 -n 100 -e questionnaire_1536_2.csv
http://localhost:8080/manta/questionnaire/
ab -c 3 -n 100 -e questionnaire_1536_3.csv
http://localhost:8080/manta/questionnaire/
ab -c 4 -n 100 -e questionnaire_1536_4.csv
http://localhost:8080/manta/questionnaire/
ab -c 5 -n 100 -e questionnaire_1536_5.csv
http://localhost:8080/manta/questionnaire/
ab -c 6 -n 100 -e questionnaire_1536_6.csv
http://localhost:8080/manta/questionnaire/
ab -c 7 -n 100 -e questionnaire_1536_7.csv
http://localhost:8080/manta/questionnaire/
ab -c 8 -n 100 -e questionnaire_1536_8.csv
http://localhost:8080/manta/questionnaire/
ab -c 9 -n 100 -e questionnaire_1536_9.csv
http://localhost:8080/manta/questionnaire/
ab -c 10 -n 100 -e questionnaire_1536_10.csv
http://localhost:8080/manta/questionnaire/

For the first command ab issues 100 requests one at a time, waiting
for one to finish before making the next.
For concurrency 10 ab sets up 10 queues each of which issues
approximately 10 requests, though I believe
that the requests are sent by the first free queue.

In perl I knit the outputs together and then display on a graph, I
then change the amount of memory available to tomcat
and repeat:
JAVA_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m -Dfile.encoding=UTF8
-Djava.awt.headless=true"
JAVA_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m
-Dfile.encoding=UTF8 -Djava.awt.headless=true"
JAVA_OPTS="-Xms128m -Xmx1536m -XX:MaxPermSize=256m
-Dfile.encoding=UTF8 -Djava.awt.headless=true"

RESULTS

https://spreadsheets.google.com/ccc?key=tZBQvVLAQAJnELkXsd0IOiA#gid=1
https://spreadsheets.google.com/ccc?key=tAcw9MijfB3NQZ8ga6twcdQ#gid=1
https://spreadsheets.google.com/ccc?key=tzKEvapJHYzaH6vga6GlN_w#gid=1

Note that with 512mb ab returned no results with a concurrency of 7 or
more, as the maximum timeout of 70 seconds as exceeded.
One might interpret these as meaning that configured with 1.5g of
memory Orbeon can support 3 simultaneous users with a response time
of less than 5 seconds.

DISCUSSION

ab is not a browser. It only makes a request of the main url, no
asynchronous javascript calls are made and
this single call does not represent a user journey. Similarly users do
not initiate a new request, with no delay, after receiving a response.
I believe that in reality Orbeon will do much better than this.

I hope this helps.

yours
Tim Pizey

--
Tim Pizey
Centre for Genomics and Global Health <http://cggh.org>
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Web: http://pizey.net/~timp
Tel: +44 (0)1865 287532

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws

Alessandro Vernet

Re: Re: Re: Concurrent users

Administrator

Tim,

Wow, I never used ab, and didn't know about its existence! I'm glad to
learn about it; it doesn't look as powerful as JMeter, but I'm sure it
can do the trick for simple tests. Now one think you have to be
careful about is cookie handling. If you're simulating 10 users, you
should make sure that each request from the 10 users keeps its
JSESSIONID. Otherwise, Tomcat / Orbeon Forms will take every request
as coming from a new user, creating a new session, which will be less
efficient, and will eat up all the memory available if you do it long
enough. WIth JMeter, you create a Thread Group, and inside it you put
an HTTP Cookie Manager, but I am not sure how you would do this with
ab.

Alex

On Fri, Jul 9, 2010 at 2:36 AM, Tim Pizey <[hidden email]> wrote:

> Hi Tom and all,
>
> Whilst my previous approach was producing some interesting graphs, and
> was stressing the machine to breaking point, getting repeatable values
> or explaining the results was pretty much impossible, as both rate and
> concurrency were varying.
>
> I was only controlling the time between requests, so the number of
> concurrent requests could reach the thousands.
> I will return to something like this when trying to defend the machine
> against DoS attacks, but to address the question of how many separate
> users can we support I have changed the methodology.
>
> METHODOLOGY
>
> The url under test is a complex form: a separated into xforms model
> file, xforms input file, an xsd and 65 vocabulary files.
> It contains multiple relevancy clauses and invokes one widget.
>
> The code can be browsed here:
> http://code.google.com/p/dsn-chassis/source/browse/trunk/manta/#manta/war/WEB-INF/resources/apps/common
>
> The tests were carried out on an elderly 3gHz P4 with 3g of memory.
> The tests were to localhost, so there was no network latency.
>
> The test runner ab, the Apache Benchmarking tool, is not written in
> java and is fairly small so is not thought to have interacted with the
> system under test, in the way that running my earlier tests from
> within Eclipse appeared to.
> Using ab I am able to make 100 requests with a varying concurrency:
>
> ab -c 1 -n 100 -e questionnaire_1536_1.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 2 -n 100 -e questionnaire_1536_2.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 3 -n 100 -e questionnaire_1536_3.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 4 -n 100 -e questionnaire_1536_4.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 5 -n 100 -e questionnaire_1536_5.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 6 -n 100 -e questionnaire_1536_6.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 7 -n 100 -e questionnaire_1536_7.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 8 -n 100 -e questionnaire_1536_8.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 9 -n 100 -e questionnaire_1536_9.csv
> http://localhost:8080/manta/questionnaire/
> ab -c 10 -n 100 -e questionnaire_1536_10.csv
> http://localhost:8080/manta/questionnaire/
>
> For the first command ab issues 100 requests one at a time, waiting
> for one to finish before making the next.
> For concurrency 10 ab sets up 10 queues each of which issues
> approximately 10 requests, though I believe
> that the requests are sent by the first free queue.
>
> In perl I knit the outputs together and then display on a graph, I
> then change the amount of memory available to tomcat
> and repeat:
> JAVA_OPTS="-Xms128m -Xmx512m -XX:MaxPermSize=256m -Dfile.encoding=UTF8
> -Djava.awt.headless=true"
> JAVA_OPTS="-Xms128m -Xmx1024m -XX:MaxPermSize=256m
> -Dfile.encoding=UTF8 -Djava.awt.headless=true"
> JAVA_OPTS="-Xms128m -Xmx1536m -XX:MaxPermSize=256m
> -Dfile.encoding=UTF8 -Djava.awt.headless=true"
>
> RESULTS
>
> https://spreadsheets.google.com/ccc?key=tZBQvVLAQAJnELkXsd0IOiA#gid=1
> https://spreadsheets.google.com/ccc?key=tAcw9MijfB3NQZ8ga6twcdQ#gid=1
> https://spreadsheets.google.com/ccc?key=tzKEvapJHYzaH6vga6GlN_w#gid=1
>
> Note that with 512mb ab returned no results with a concurrency of 7 or
> more, as the maximum timeout of 70 seconds as exceeded.
> One might interpret these as meaning that configured with 1.5g of
> memory Orbeon can support 3 simultaneous users with a response time
> of less than 5 seconds.
>
> DISCUSSION
>
> ab is not a browser. It only makes a request of the main url, no
> asynchronous javascript calls are made and
> this single call does not represent a user journey. Similarly users do
> not initiate a new request, with no delay, after receiving a response.
> I believe that in reality Orbeon will do much better than this.
>
> I hope this helps.
>
> yours
> Tim Pizey
>
>
>
>
>
>
>
>
> --
> Tim Pizey
> Centre for Genomics and Global Health <http://cggh.org>
> The Wellcome Trust Centre for Human Genetics
> Roosevelt Drive
> Oxford
> OX3 7BN
> United Kingdom
> Web: http://pizey.net/~timp
> Tel: +44 (0)1865 287532
>
>
> --
> You receive this message as a subscriber of the [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> OW2 mailing lists service home page: http://www.ow2.org/wws
>
>

--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet

Tim Pizey-2

Re: Re: Re: Re: Concurrent users

Hi Alex,

On 10 July 2010 00:27, Alessandro Vernet wrote:

> Tim,
>
> Wow, I never used ab, and didn't know about its existence! I'm glad to
> learn about it; it doesn't look as powerful as JMeter, but I'm sure it
> can do the trick for simple tests. Now one think you have to be
> careful about is cookie handling. If you're simulating 10 users, you
> should make sure that each request from the 10 users keeps its
> JSESSIONID. Otherwise, Tomcat / Orbeon Forms will take every request
> as coming from a new user, creating a new session, which will be less
> efficient, and will eat up all the memory available if you do it long
> enough. WIth JMeter, you create a Thread Group, and inside it you put
> an HTTP Cookie Manager, but I am not sure how you would do this with
> ab.
>
> Alex

Sorry to say I made an error in my interpretation of the output from ab.
ab outputs a csv file which contains the output for the 'request' which
seems to mean the concurrent group making the request.

The fix for this has been to divide the value by the queue width.

ie for the command
ab -c 10 -n 100 -e questionnaire_1536_10.csv
http://localhost:8080/manta/questionnaire/

every value in questionnaire_1536_10.csv must be divided by 10.

For example:
previously I had
https://spreadsheets.google.com/ccc?key=0AibFqy9K5V7OdEk5WGdQMC0tNDZ0Q3p4eWRUSld5N2c&hl=en_GB#gid=1
which I interpreted as meaning that Orbeon was only able to support 5
simultaneous users with a response time of less than 5 seconds.
(This did ring warning bells, honest)
corrected the same data gives
https://spreadsheets.google.com/ccc?key=0AibFqy9K5V7OdGIxWEJhT3FzbGJMREVOZFJaM2Y1RHc&hl=en_GB#gid=1
which I interpret as meaning that Orbeon scales quite happily to 10
simultaneous users.

yours
Tim
--
Tim Pizey
Centre for Genomics and Global Health <http://cggh.org>
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Web: http://pizey.net/~timp
Tel: +44 (0)1865 287532

--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
OW2 mailing lists service home page: http://www.ow2.org/wws