Memory impact on performance

Hi,

when I first started using OPS I was getting out-of-memory errors and  
so I added -Xms128m -Xmx256m to my tomcat startup and everything  
looked good.
One project I have been working on recently was seeing response times  
from clicking on a link to the first change in the browser window  
being around 8 or 9 seconds.
I was looking at apache caching as a way to "hide" this problem, and  
Erik asked if I had increased OPS cache size.
I hadn't, and I wondered if I was to do so, would I need to increase  
my JVM memory again.

I didn't change the OPS cache size, but I went ahead and changed my  
JVM to -Xms384m -Xmx512m
And my response times went down to around 3 or 4 seconds.
This is very consistent, testing from a number of locations.

This raises a few questions

* I wonder what settings other OPS installations are running with?

* as time goes by (most in the first few hours), the memory used (as  
opposed to virtual memory used) (as shown in top and similar tools)  
continues to increase - does this mean it will ultimately reach  
saturation and the performance return to its previous poor level?

* As I add more projects to this server, will I again reach a point  
where performance starts to degrade?

* This project reads a lot of files from disk and then filters them  
with XSLT before building the resulting pages - if I change this to  
an XML database such as exist and use xquery/xslt, will this improve  
both performance and memory consumption, or will it simply move the  
problem over to exist?

Thanks in advance for anyone's insights
Colin




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Hi Colin,

On 10/18/06, Colin O'Brien <[hidden email]> wrote:
> * I wonder what settings other OPS installations are running with?

In development mode, most of the applications I work on run fine with
a heap of 386 or 512 MB. In production, you usually know what are the
most memory -hungry applications running on the machine (say a Java VM
and a database) , so we recommend to set the heap size be as large as
possible as long it it does not cause swapping. See 1.2 on the
Performance and Tuning page:

http://www.orbeon.com/ops/doc/integration-tuning

> * as time goes by (most in the first few hours), the memory used (as
> opposed to virtual memory used) (as shown in top and similar tools)
> continues to increase - does this mean it will ultimately reach
> saturation and the performance return to its previous poor level?

In production (and I do this in development too), we recommend to set
-ms and -mx to be the same. See 1.1 on the Performance and Tuning page
linked above.

> * As I add more projects to this server, will I again reach a point
> where performance starts to degrade?

Possibly. If the projects are running in different war files, you will
need to increase the heap size. If they are running in the same war
file, you will need to increase both the heap and cache size.

> * This project reads a lot of files from disk and then filters them
> with XSLT before building the resulting pages - if I change this to
> an XML database such as exist and use xquery/xslt, will this improve
> both performance and memory consumption, or will it simply move the
> problem over to exist?

Hard to say. If the files are large, and you only using a portion of a
file to process a request, and you can define what that portion is
with XQuery, then I would think you are better off storing those files
in eXist.

Alex
--
Blog (XML, Web apps, Open Source):
http://www.orbeon.com/blog/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Hi Alex,

thanks for your reply.

> Performance and Tuning page:
>
> http://www.orbeon.com/ops/doc/integration-tuning

Doesn't matter how often I think I've already read all the  
documentation, seems you can always come up with the answer in a doc  
page I haven't read - good planning - I think this page must be new  
for 3.5 - very useful.

So as you said, use as much VM memory as you can, without getting  
into swapping.

But - also re-evaluate as part of finishing each new project.

This server has now stabilized its VM memory consumption at about 95%  
of the peak it hit (judging by the values reported by top) with  
around 30mb (of 2gb) free - so it is probably pretty much as good as  
it is going to get in this respect for the time being.

Unless I expand the cache.

At the moment, all projects are being added to (actually referenced  
by) a single OPS instance.
That's interesting. Most of the file retrieved will be used, but  
sometimes only one of the files retrieved is actually displayed, and  
the rest are either filtered out, or used to build a next/previous  
linking sequence (though this particular problematic project is a  
little different, in that the whole filtered list needs to be  
presented down the side of the page and rollovers update the central  
detail on the page, so most of each file is not only  getting used,  
it's also being pumped out to the browser).

Best regards
Colin




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

On 10/19/06, Colin O'Brien <[hidden email]> wrote:
> Doesn't matter how often I think I've already read all the
> documentation, seems you can always come up with the answer in a doc
> page I haven't read - good planning - I think this page must be new
> for 3.5 - very useful.

Our secret is to update the doc after you ask the question, and
quickly push the updated doc online ;). But not this time. Most of
that page was written back January. Honest. Of course, there is room
for improvement: if you have in mind tips & tricks that have useful to
you and that can be mentioned there, just let us know.

Alex
--
Blog (XML, Web apps, Open Source):
http://www.orbeon.com/blog/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Hi Alex,

while I am thinking of this, there is one question that your  
documentation does not cover...

When (if at all) is it better to start running a second copy of  
tomcat on the same server?

(All of our servers are multi-processor if that makes any difference).

Thanks & regards
Colin




--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws

Hi Colin,

On 10/24/06, Colin O'Brien <[hidden email]> wrote:
> When (if at all) is it better to start running a second copy of
> tomcat on the same server?
>
> (All of our servers are multi-processor if that makes any difference).

If the VM and OS do a good job at using the CPUs available to run
multiple threads, having just one VM would be the best option. But
this assumption might not be true depending on your environment. In
particular, I remember that a few years ago we ran some tests and
Sun's VM didn't scale very well on Windows with a 2-CPU machine,
compared to JRockit or IBM's VM. I don't have any specific advice
here: testing seems the best way to go.

Alex
--
Blog (XML, Web apps, Open Source):
http://www.orbeon.com/blog/



--
You receive this message as a subscriber of the [hidden email] mailing list.
To unsubscribe: mailto:[hidden email]
For general help: mailto:[hidden email]?subject=help
ObjectWeb mailing lists service home page: http://www.objectweb.org/wws