ThreadPool is exhausted easily (Denial of Service attack)
Locked
 
|
After many requests to orbeon, all threads are consumed and tomcat hangs.
tomcat server.xml config: maxThreads="50" enableLookups="false" acceptCount="500" Press and hold F5 (yes, its not very representative of normal load, but its an easy DOS, and I am sure one of my users will try this out lol) Orbeon, eXist, and tomcat hang... (note, tomcat manager and eXist REST can both handle this refresh attack without hanging tomcat) catalina.out: SEVERE: All threads (50) are currently busy, waiting. Increase maxThreads (50) or check the servlet status Ideas? |
Re: ThreadPool is exhausted easily (Denial of Service attack)
|
Administrator
|
I would first try to heavily reduce the number of threads entering
Orbeon Forms. Keep just a few (say, 2 for a start), and let the other ones in line. Orbeon Forms requires quite a bit of memory to handle a single request, and adding threads starts putting a huge load on the garbage collector. So the net benefit of adding threads gets negative for both response times and throughput. See: http://www.orbeon.com/ops/doc/integration-tuning#application-server-threads -Erik On Jul 15, 2008, at 11:03 AM, lazx wrote: > > After many requests to orbeon, all threads are consumed and tomcat > hangs. > > tomcat server.xml config: > maxThreads="50" > enableLookups="false" > acceptCount="500" > > Press and hold F5 (yes, its not very representative of normal load, > but its > an easy DOS, and I am sure one of my users will try this out lol) > > Orbeon, eXist, and tomcat hang... (note, tomcat manager and eXist > REST can > both handle this refresh attack without hanging tomcat) > > catalina.out: > > SEVERE: All threads (50) are currently busy, waiting. Increase > maxThreads > (50) or check the servlet status > > > > Ideas? > -- > View this message in context: http://www.nabble.com/ThreadPool-is-exhausted-easily-%28Denial-of-Service-attack%29-tp18471429p18471429.html > Sent from the ObjectWeb OPS - Users mailing list archive at > Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] > mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws Orbeon Forms - Web Forms for the Enterprise Done the Right Way http://www.orbeon.com/ -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
|
|
I attempted setting maxThreads to 2... tomcat "adjustLimits" says this is too low and set maxThreads to 10. With maxThreads = 10, under load the orbeon, eXists, and tomcat all die.
Is there anything else I can do to improve performance? Memory settings are Xmx512M, Xms512M, PermSize=128M Thanks Erik!
|
Re: Re: ThreadPool is exhausted easily (Denial of Service attack)
|
Administrator
|
2 is too low? Did you try 4? I remember that some Tomcat connectors
needed actually 2 threads per request. By "die", do you mean hang? Are there exceptions? -Erik On Jul 15, 2008, at 1:18 PM, lazx wrote: > > I attempted setting maxThreads to 2... tomcat "adjustLimits" says > this is > too low and set maxThreads to 10. With maxThreads = 10, under load > the > orbeon, eXists, and tomcat all die. > > Is there anything else I can do to improve performance? > > Memory settings are Xmx512M, Xms512M, PermSize=128M > > Thanks Erik! > > > Erik Bruchez wrote: >> >> I would first try to heavily reduce the number of threads entering >> Orbeon Forms. Keep just a few (say, 2 for a start), and let the other >> ones in line. Orbeon Forms requires quite a bit of memory to handle a >> single request, and adding threads starts putting a huge load on the >> garbage collector. So the net benefit of adding threads gets negative >> for both response times and throughput. See: >> >> >> http://www.orbeon.com/ops/doc/integration-tuning#application-server-threads >> >> -Erik >> >> On Jul 15, 2008, at 11:03 AM, lazx wrote: >> >>> >>> After many requests to orbeon, all threads are consumed and tomcat >>> hangs. >>> >>> tomcat server.xml config: >>> maxThreads="50" >>> enableLookups="false" >>> acceptCount="500" >>> >>> Press and hold F5 (yes, its not very representative of normal load, >>> but its >>> an easy DOS, and I am sure one of my users will try this out lol) >>> >>> Orbeon, eXist, and tomcat hang... (note, tomcat manager and eXist >>> REST can >>> both handle this refresh attack without hanging tomcat) >>> >>> catalina.out: >>> >>> SEVERE: All threads (50) are currently busy, waiting. Increase >>> maxThreads >>> (50) or check the servlet status >>> >>> >>> >>> Ideas? >>> -- >>> View this message in context: >>> http://www.nabble.com/ThreadPool-is-exhausted-easily-%28Denial-of-Service-attack%29-tp18471429p18471429.html >>> Sent from the ObjectWeb OPS - Users mailing list archive at >>> Nabble.com. >>> >>> >>> -- >>> You receive this message as a subscriber of the [hidden email] >>> mailing list. >>> To unsubscribe: mailto:[hidden email] >>> For general help: mailto:[hidden email]?subject=help >>> OW2 mailing lists service home page: http://www.ow2.org/wws >> >> -- >> Orbeon Forms - Web Forms for the Enterprise Done the Right Way >> http://www.orbeon.com/ >> >> >> >> -- >> You receive this message as a subscriber of the [hidden email] >> mailing >> list. >> To unsubscribe: mailto:[hidden email] >> For general help: mailto:[hidden email]?subject=help >> OW2 mailing lists service home page: http://www.ow2.org/wws >> >> > > -- > View this message in context: http://www.nabble.com/ThreadPool-is-exhausted-easily-%28Denial-of-Service-attack%29-tp18471429p18474041.html > Sent from the ObjectWeb OPS - Users mailing list archive at > Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] > mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws Orbeon Forms - Web Forms for the Enterprise Done the Right Way http://www.orbeon.com/ -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
|
|
Editing /etc/tomcat55/server.xml...
<Connector port="8080" maxHttpHeaderSize="8192" maxThreads="4" enableLookups="false" redirectPort="8443" acceptCount="50" connectionTimeout="10000" disableUploadTimeout="true" /> And this is the error I (still) get: Jul 16, 2008 4:23:23 PM org.apache.tomcat.util.threads.ThreadPool adjustLimits WARNING: maxThreads setting (4) too low, set to 10 Setting maxThreads to 4, 6, 8 does not help. I guess I could hack tomcat, but that seems a little extreme. Is there anyway to minimize the number of threads orbeon uses in one of its configuration files?? And when I say "hang tomcat", this refers to every tomcat application (and tomcat itself) ceasing to work. This includes the database (eXist)... As for exceptions. Form builder and runner completely kill/crash tomcat and tomcat apps. Non form builder/runner pages only kill/crash orbeon (eXist and tomcat seem to continue working...) Is there a way to restrict the number of requests going to orbeon (or is this the purpose of the maxThreads option???) Thank you!
|
Re: Re: Re: ThreadPool is exhausted easily (Denial of Service attack)
|
Administrator
|
Yes the purpose of maxThreads is exactly to do this. However I have
found the following: http://www.mail-archive.com/tomcat-user@.../msg117716.html "10 is the hard-coded minimum set in Tomcat." That is just plain insane to me, and I have seen other people having this issue with memory-intensive apps. It seems that we should maybe implement this directly in Orbeon Forms. -Erik On Jul 16, 2008, at 1:34 PM, lazx wrote: > > Editing /etc/tomcat55/server.xml... > > <Connector port="8080" maxHttpHeaderSize="8192" > maxThreads="4" > enableLookups="false" redirectPort="8443" > acceptCount="50" > connectionTimeout="10000" disableUploadTimeout="true" /> > > And this is the error I (still) get: > > Jul 16, 2008 4:23:23 PM > org.apache.tomcat.util.threads.ThreadPool adjustLimits > WARNING: maxThreads setting (4) too low, set to 10 > > Setting maxThreads to 4, 6, 8 does not help. I guess I could hack > tomcat, > but that seems a little extreme. Is there anyway to minimize the > number of > threads orbeon uses in one of its configuration files?? > > And when I say "hang tomcat", this refers to every tomcat > application (and > tomcat itself) ceasing to work. This includes the database (eXist)... > > As for exceptions. Form builder and runner completely kill/crash > tomcat and > tomcat apps. Non form builder/runner pages only kill/crash orbeon > (eXist > and tomcat seem to continue working...) > > Is there a way to restrict the number of requests going to orbeon > (or is > this the purpose of the maxThreads option???) > > > Thank you! > > > > > Erik Bruchez wrote: >> >> 2 is too low? Did you try 4? I remember that some Tomcat connectors >> needed actually 2 threads per request. >> >> By "die", do you mean hang? Are there exceptions? >> >> -Erik >> >> On Jul 15, 2008, at 1:18 PM, lazx wrote: >> >>> >>> I attempted setting maxThreads to 2... tomcat "adjustLimits" says >>> this is >>> too low and set maxThreads to 10. With maxThreads = 10, under load >>> the >>> orbeon, eXists, and tomcat all die. >>> >>> Is there anything else I can do to improve performance? >>> >>> Memory settings are Xmx512M, Xms512M, PermSize=128M >>> >>> Thanks Erik! >>> >>> >>> Erik Bruchez wrote: >>>> >>>> I would first try to heavily reduce the number of threads entering >>>> Orbeon Forms. Keep just a few (say, 2 for a start), and let the >>>> other >>>> ones in line. Orbeon Forms requires quite a bit of memory to >>>> handle a >>>> single request, and adding threads starts putting a huge load on >>>> the >>>> garbage collector. So the net benefit of adding threads gets >>>> negative >>>> for both response times and throughput. See: >>>> >>>> >>>> http://www.orbeon.com/ops/doc/integration-tuning#application-server-threads >>>> >>>> -Erik >>>> >>>> On Jul 15, 2008, at 11:03 AM, lazx wrote: >>>> >>>>> >>>>> After many requests to orbeon, all threads are consumed and tomcat >>>>> hangs. >>>>> >>>>> tomcat server.xml config: >>>>> maxThreads="50" >>>>> enableLookups="false" >>>>> acceptCount="500" >>>>> >>>>> Press and hold F5 (yes, its not very representative of normal >>>>> load, >>>>> but its >>>>> an easy DOS, and I am sure one of my users will try this out lol) >>>>> >>>>> Orbeon, eXist, and tomcat hang... (note, tomcat manager and eXist >>>>> REST can >>>>> both handle this refresh attack without hanging tomcat) >>>>> >>>>> catalina.out: >>>>> >>>>> SEVERE: All threads (50) are currently busy, waiting. Increase >>>>> maxThreads >>>>> (50) or check the servlet status >>>>> >>>>> >>>>> >>>>> Ideas? >>>>> -- >>>>> View this message in context: >>>>> http://www.nabble.com/ThreadPool-is-exhausted-easily-%28Denial-of-Service-attack%29-tp18471429p18471429.html >>>>> Sent from the ObjectWeb OPS - Users mailing list archive at >>>>> Nabble.com. >>>>> >>>>> >>>>> -- >>>>> You receive this message as a subscriber of the [hidden email] >>>>> mailing list. >>>>> To unsubscribe: mailto:[hidden email] >>>>> For general help: mailto:[hidden email]?subject=help >>>>> OW2 mailing lists service home page: http://www.ow2.org/wws >>>> >>>> -- >>>> Orbeon Forms - Web Forms for the Enterprise Done the Right Way >>>> http://www.orbeon.com/ >>>> >>>> >>>> >>>> -- >>>> You receive this message as a subscriber of the [hidden email] >>>> mailing >>>> list. >>>> To unsubscribe: mailto:[hidden email] >>>> For general help: mailto:[hidden email]?subject=help >>>> OW2 mailing lists service home page: http://www.ow2.org/wws >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/ThreadPool-is-exhausted-easily-%28Denial-of-Service-attack%29-tp18471429p18474041.html >>> Sent from the ObjectWeb OPS - Users mailing list archive at >>> Nabble.com. >>> >>> >>> -- >>> You receive this message as a subscriber of the [hidden email] >>> mailing list. >>> To unsubscribe: mailto:[hidden email] >>> For general help: mailto:[hidden email]?subject=help >>> OW2 mailing lists service home page: http://www.ow2.org/wws >> >> -- >> Orbeon Forms - Web Forms for the Enterprise Done the Right Way >> http://www.orbeon.com/ >> >> >> >> -- >> You receive this message as a subscriber of the [hidden email] >> mailing >> list. >> To unsubscribe: mailto:[hidden email] >> For general help: mailto:[hidden email]?subject=help >> OW2 mailing lists service home page: http://www.ow2.org/wws >> >> > > -- > View this message in context: http://www.nabble.com/ThreadPool-is-exhausted-easily-%28Denial-of-Service-attack%29-tp18471429p18496030.html > Sent from the ObjectWeb OPS - Users mailing list archive at > Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] > mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws Orbeon Forms - Web Forms for the Enterprise Done the Right Way http://www.orbeon.com/ -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
Re: Re: Re: ThreadPool is exhausted easily (Denial of Service attack)
|
Administrator
|
In reply to this post by lazx
On Wed, Jul 16, 2008 at 1:34 PM, lazx <[hidden email]> wrote:
> As for exceptions. Form builder and runner completely kill/crash tomcat and > tomcat apps. Non form builder/runner pages only kill/crash orbeon (eXist > and tomcat seem to continue working...) With a maxThread set to 10 (since this is the minimum), what happens exactly? What do you mean by "kill/crash"? Do you have an error page coming back to the browser? An error in the logs? Does CPU utilization stay at 100% "for ever" without having a page ever returned to the browser? Alex -- Orbeon Forms - Web 2.0 Forms, open-source, for the Enterprise Orbeon's Blog: http://www.orbeon.com/blog/ Personal Blog: http://avernet.blogspot.com/ Twitter - http://twitter.com/avernet -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
|
|
Hi Alessandro,
When requesting many successive (you could also say excessive) refreshes from (for example) a form runner based xform listing page (ie the page that is used to list all of the forms that have been completed/saved), tomcat/eXist/orbeon all hang. No error is returned to the browser, and no error messages besides the one in catalina.out (the one about all the threads being used). To get more verbose logging in orbeon, I enabled: "XFormsModelSubmission, XFormsServer" in log4j.xml Log messages were produced in excess along with the many refresh/get requests - and the logging also stopped as orbeon/eXist/tomcat crashed. The last log message in orbeon.out seems to be random... so nothing useful in this area. I'll disable the "XFormsModelSubmission, XFormsServer" logging options and provide you with a couple crash simulations to attempt making this clearer. While I am at it, is there any other things that may be useful to you? Thanks!
|
|
|
In reply to this post by Alessandro Vernet
Some more log info:
catalina.out: INFO: Server startup in 4768 ms Begin flood of requests Jul 17, 2008 10:56:08 AM org.apache.tomcat.util.threads.ThreadPool logFull SEVERE: All threads (10) are currently busy, waiting. Increase maxThreads (10) or check the servlet status Tomcat/eXist/orbeon at this point do not respond to any further requests Below is the log of tomcat shutting down. Jul 17, 2008 10:56:47 AM org.apache.coyote.http11.Http11BaseProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-8080 Jul 17, 2008 10:56:48 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina Jul 17, 2008 10:56:48 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 8 instance(s) to be deallocated Jul 17, 2008 10:56:49 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 8 instance(s) to be deallocated Jul 17, 2008 10:56:50 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 8 instance(s) to be deallocated Jul 17, 2008 10:56:50 AM org.quartz.core.QuartzScheduler shutdown INFO: Scheduler exist_QuartzScheduler_$_NON_CLUSTERED shutting down. Jul 17, 2008 10:56:50 AM org.quartz.core.QuartzScheduler standby INFO: Scheduler exist_QuartzScheduler_$_NON_CLUSTERED paused. Jul 17, 2008 10:56:50 AM org.quartz.core.QuartzScheduler shutdown INFO: Scheduler exist_QuartzScheduler_$_NON_CLUSTERED shutdown complete. log4j:ERROR LogMananger.repositorySelector was null likely due to error in class reloading, using NOPLoggerRepository. Jul 17, 2008 10:56:51 AM org.apache.coyote.http11.Http11BaseProtocol destroy INFO: Stopping Coyote HTTP/1.1 on http-8080 Jul 17, 2008 10:56:51 AM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run SEVERE: Caught exception (java.lang.IllegalStateException) executing org.apache.tomcat.util.net.LeaderFollowerWorkerThread@17c76f6, terminating thread Jul 17, 2008 10:56:51 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: Failed shutdown of Apache Portable Runtime And here is something interesting... one of the last tomcat "Server Status" logs that I am able to get from the manager/status tomcat app: S 3686 ms 0 KB 0 KB 127.0.0.2 tomcatserver GET /orbeon_dev/fr/service/exist/crud/utilization_toolkit_project/patient_care_needs_assessment_tool/form/form.xhtml HTTP/1.1 S 7 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET /manager/status HTTP/1.1 S 4129 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET /orbeon_dev/fr/test_app/patient_care_needs_assessment_tool HTTP/1.1 S 3540 ms 0 KB 0 KB 127.0.0.2 tomcatserver GET /orbeon_dev/fr/service/exist/crud/test_app/test_form/form/form.xhtml HTTP/1.1 S 3931 ms 0 KB 0 KB 127.0.0.2 tomcatserver POST /orbeon_dev/fr/service/exist/search/test_app/test_form HTTP/1.1 S 4236 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET /orbeon_dev/fr/test_app/test_form HTTP/1.1 S 4053 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET /orbeon_dev/fr/test_app/test_form HTTP/1.1 S 3856 ms 0 KB 0 KB 127.0.0.2 tomcatserver GET /orbeon_dev/fr/service/exist/crud/test_app/test_form/form/form.xhtml HTTP/1.1 S 3896 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET /orbeon_dev/fr/test_app/test_form HTTP/1.1 R ? ? ? ? ? ? Could this problem be related to eXist vs. orbeon deadlock?? Orbeon uses a thread for each request, plus another thread is used if a database/eXist lookup/query is required. So when there is one thread remaining in the pool, orbeon will use it and hold it until eXist returns some data - BUT eXist is waiting for a thread, but will never get one because orbeon is holding it and waiting for eXist. Am I completely misunderstood about the way tomcat handles allocating threads to webapps? Note, tomcat/orbeon/eXist are all non-responsive, cpu usage is essentially idle and memory usage is stable. So, I am thinking this is not a bug in orbeon, but rather a resource problem. Is there anyway to allocate threads within tomcat (ie, 2 threads dedicate to eXist, 6 threads for orbeon). |
Re: Re: Re: Re: ThreadPool is exhausted easily (Denial of Service attack)
|
Administrator
|
I wonder if this can just be a deadlock situation. When everything
blocks, what is CPU usage like? Can you also try to change this property to false in properties.xml: <property as="xs:boolean" name="oxf.xforms.optimize-local- submission" value="false"/> (Not that I have hopes this will change anythying, but you never know.) -Erik On Jul 17, 2008, at 8:46 AM, lazx wrote: > > Some more log info: > > catalina.out: > > INFO: Server startup in 4768 ms > Begin flood of requests > Jul 17, 2008 10:56:08 AM org.apache.tomcat.util.threads.ThreadPool > logFull > SEVERE: All threads (10) are currently busy, waiting. Increase > maxThreads > (10) or check the servlet status > Tomcat/eXist/orbeon at this point do not respond to any further > requests > Below is the log of tomcat shutting down. > Jul 17, 2008 10:56:47 AM org.apache.coyote.http11.Http11BaseProtocol > pause > INFO: Pausing Coyote HTTP/1.1 on http-8080 > Jul 17, 2008 10:56:48 AM org.apache.catalina.core.StandardService stop > INFO: Stopping service Catalina > Jul 17, 2008 10:56:48 AM org.apache.catalina.core.StandardWrapper > unload > INFO: Waiting for 8 instance(s) to be deallocated > Jul 17, 2008 10:56:49 AM org.apache.catalina.core.StandardWrapper > unload > INFO: Waiting for 8 instance(s) to be deallocated > Jul 17, 2008 10:56:50 AM org.apache.catalina.core.StandardWrapper > unload > INFO: Waiting for 8 instance(s) to be deallocated > Jul 17, 2008 10:56:50 AM org.quartz.core.QuartzScheduler shutdown > INFO: Scheduler exist_QuartzScheduler_$_NON_CLUSTERED shutting down. > Jul 17, 2008 10:56:50 AM org.quartz.core.QuartzScheduler standby > INFO: Scheduler exist_QuartzScheduler_$_NON_CLUSTERED paused. > Jul 17, 2008 10:56:50 AM org.quartz.core.QuartzScheduler shutdown > INFO: Scheduler exist_QuartzScheduler_$_NON_CLUSTERED shutdown > complete. > log4j:ERROR LogMananger.repositorySelector was null likely due to > error in > class reloading, using NOPLoggerRepository. > Jul 17, 2008 10:56:51 AM org.apache.coyote.http11.Http11BaseProtocol > destroy > INFO: Stopping Coyote HTTP/1.1 on http-8080 > Jul 17, 2008 10:56:51 AM > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run > SEVERE: Caught exception (java.lang.IllegalStateException) executing > org.apache.tomcat.util.net.LeaderFollowerWorkerThread@17c76f6, > terminating > thread > Jul 17, 2008 10:56:51 AM org.apache.catalina.core.AprLifecycleListener > lifecycleEvent > INFO: Failed shutdown of Apache Portable Runtime > > And here is something interesting... one of the last tomcat "Server > Status" > logs that I am able to get from the manager/status tomcat app: > > S 3686 ms 0 KB 0 KB 127.0.0.2 tomcatserver GET > /orbeon_dev/fr/service/exist/crud/utilization_toolkit_project/ > patient_care_needs_assessment_tool/form/form.xhtml > HTTP/1.1 > S 7 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET /manager/status HTTP/ > 1.1 > S 4129 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET > /orbeon_dev/fr/test_app/patient_care_needs_assessment_tool HTTP/1.1 > S 3540 ms 0 KB 0 KB 127.0.0.2 tomcatserver GET > /orbeon_dev/fr/service/exist/crud/test_app/test_form/form/form.xhtml > HTTP/1.1 > S 3931 ms 0 KB 0 KB 127.0.0.2 tomcatserver POST > /orbeon_dev/fr/service/exist/search/test_app/test_form HTTP/1.1 > S 4236 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET > /orbeon_dev/fr/test_app/test_form HTTP/1.1 > S 4053 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET > /orbeon_dev/fr/test_app/test_form HTTP/1.1 > S 3856 ms 0 KB 0 KB 127.0.0.2 tomcatserver GET > /orbeon_dev/fr/service/exist/crud/test_app/test_form/form/form.xhtml > HTTP/1.1 > S 3896 ms 0 KB 0 KB 192.168.1.100 tomcatserver GET > /orbeon_dev/fr/test_app/test_form HTTP/1.1 > R ? ? ? ? ? ? > > > Could this problem be related to eXist vs. orbeon deadlock?? Orbeon > uses a > thread for each request, plus another thread is used if a database/ > eXist > lookup/query is required. So when there is one thread remaining in > the > pool, orbeon will use it and hold it until eXist returns some data - > BUT > eXist is waiting for a thread, but will never get one because orbeon > is > holding it and waiting for eXist. Am I completely misunderstood > about the > way tomcat handles allocating threads to webapps? > > Note, tomcat/orbeon/eXist are all non-responsive, cpu usage is > essentially > idle and memory usage is stable. So, I am thinking this is not a > bug in > orbeon, but rather a resource problem. Is there anyway to allocate > threads > within tomcat (ie, 2 threads dedicate to eXist, 6 threads for orbeon). > -- > View this message in context: http://www.nabble.com/ThreadPool-is-exhausted-easily-%28Denial-of-Service-attack%29-tp18471429p18511590.html > Sent from the ObjectWeb OPS - Users mailing list archive at > Nabble.com. > > > -- > You receive this message as a subscriber of the [hidden email] > mailing list. > To unsubscribe: mailto:[hidden email] > For general help: mailto:[hidden email]?subject=help > OW2 mailing lists service home page: http://www.ow2.org/wws Orbeon Forms - Web Forms for the Enterprise Done the Right Way http://www.orbeon.com/ -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
|
|
CPU usage is idle. Memory usage (for tomcat... process name) java.bin remains constant (no increase/decrease of memory, and swap is not touched). Sorry, doesn't help. What does help though is increasing the number of maxthreads. With maxthreads set to 50, tomcat/orbeon/eXist can handle quite a large amount of load (F5 depressed for 2 seconds or so... doubt this is the case for large forms though). Anyway, maybe I am being a little too demanding of the orbeon application. There would be a load balancer, static server, memcached in front of orbeon anyway. Will try a separate delployment of eXist to check out the orbeon vs. eXist deadlock theory. |
Re: Re: Re: Re: Re: ThreadPool is exhausted easily (Denial of Service attack)
|
Administrator
|
On Thu, Jul 17, 2008 at 12:13 PM, lazx <[hidden email]> wrote:
>> I wonder if this can just be a deadlock situation. When everything >> blocks, what is CPU usage like? > > CPU usage is idle. Memory usage (for tomcat... process name) java.bin > remains constant (no increase/decrease of memory, and swap is not touched). I was able to reproduce this by setting maxThreads="10". Looking at the stack trace, it looks like we have 9 threads waiting to be able to connect to a service. Because of the setting we have for maxThreads, those requests must be queued, hence the deadlock. Interesting one. I am not sure if this is really a bug per se (see what follows), but I am adding one as we at least need to do a better job documenting this: http://forge.objectweb.org/tracker/index.php?func=detail&aid=310595&group_id=168&atid=350207 A solution here, is to: 1) Limit the number of active threads with the balancer rather than Tomcat. 2) Make sure the calls to the services go directly to Tomcat rather than the load balancer. So the load balancer will queue up the requests (which is better anyway, rather than doing this at the Tomcat level), and calls to services will never be blocked. Alex -- Orbeon Forms - Web 2.0 Forms, open-source, for the Enterprise Orbeon's Blog: http://www.orbeon.com/blog/ Personal Blog: http://avernet.blogspot.com/ Twitter - http://twitter.com/avernet -- You receive this message as a subscriber of the [hidden email] mailing list. To unsubscribe: mailto:[hidden email] For general help: mailto:[hidden email]?subject=help OW2 mailing lists service home page: http://www.ow2.org/wws |
«
Return to Orbeon Forms community mailing list
|
1 view|%1 views
| Free forum by Nabble | Edit this page |