Orbeon Forms community mailing list

Idle connections not closed (Too many open files)?

Classic

List

Threaded

15 messages Options

pc3356

Idle connections not closed (Too many open files)?

This post was updated on .

Hi,

We're running 4.0.0 on Tomcat and periodically see an error from Tomcat, which prevents us from doing anything more on the server until the Tomcat instance is restarted:

28-Jun-2013 12:25:40 org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
	at java.net.ServerSocket.implAccept(ServerSocket.java:462)
	at java.net.ServerSocket.accept(ServerSocket.java:430)
	at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
	at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352)
	at java.lang.Thread.run(Thread.java:662)

This is seen in an Orbeon stack trace too:

too-many-open-files-trace.txt

It looks like this is the result of not calling closeIdleConnections/closeExpiredConnections from the ClientConnectionManager underlying the HttpClient inside Orbeon. (http://hc.apache.org/httpcomponents-client-dev/tutorial/html/connmgmt.html)

The number of open files climbs slowly over time and necessitates periodic service restarts until we can find a way to fix this issue.

Has anyone else encountered issues like this, and are there any known workarounds?

Many thanks,

Phil.

Alessandro Vernet

Re: Idle connections not closed (Too many open files)?

Administrator

Hi Phil,

What operating system is this on? Is this on Windows by any chance? It looks like this is happening when Orbeon Forms connects to the implementation of the persistence layer. Could it be related to this Windows issue (see link to a blog post below)?

http://blog.orbeon.com/2006/05/tcpip_05.html

Alex

--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet

pc3356

Re: Idle connections not closed (Too many open files)?

This post was updated on .

Hi Alex,

I don't know that it is related; we're seeing the problem when we run under RHEL, with Orbeon 4.0.0 on Tomcat 6.

Over time the number of open file descriptors steadily grows - we could increase the number of available files, but this will just move the problem rather than fix it.

I've tried setting the parameter, oxf.http.so-timeout, to 30 seconds in our properties override to see if that makes any difference, and I've left the server running with this configuration overnight to get a decent sample of results.

If that doesn't work, what are our other options? I looked into configuring a new Listener in the web.xml that periodically tries to close expired/idle connections, but I don't think I have access to the ClientConnectionManager used by the HTTPURLConnection class, which is the only place I can see it set up.

We are running a custom persistence layer using Jersey JAX-RS, and I'd be surprised if that was causing CLOSE_WAIT on connection, as other applications on the same platform don't seem to have this problem.

We'll keep an eye on it and post if we have any breakthrough.

Thanks,

Phil.

Andrey

Re: Idle connections not closed (Too many open files)?

In reply to this post by Alessandro Vernet

Hello!
I had a problem looking similar, at least it was "Too many open files", deploying Orbeon 3.9 as separate app on Debian Linux, Glassfish 3 server. I was not sure if Orbeon was a reason of problem, there was a lot of our own code deployed and I didn't check Orbeon log, server log only. I didn't find out how to fix it, but after my server configuration was changed, problem was gone. I had no this problem with JBoss 6 server both on Windows and Linux.

--
You received this message because you are subscribed to the Google Groups "Orbeon Forms" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].

pc3356

Re: Idle connections not closed (Too many open files)?

Andrey:

Do you know what the server configuration change involved? We don't have a lot of access to the servers here, and so it would be useful to know.

Alex:

I don't know how relevant this is, but the ThreadSafeClientConnManager currently in use in HTTPURLConnection is deprecated from 4.2 of Apache HTTPComponents, with advice to use PoolingClientConnectionManager instead. Maybe worth considering for the next release.

Thanks,

Phil.

Andrey

Re: Idle connections not closed (Too many open files)?

Main configuration change was: app server Glassfish 3.0.1 to JBoss 6.0.
But note, I am not sure Orbeon was a reason of my "Too many open files" problem. Together with server change lots of our application code refactorings were introduced.

Andrey

Alessandro Vernet

Re: Idle connections not closed (Too many open files)?

Administrator

In reply to this post by pc3356

Hi Phil,

If this is happening on Linux, it is most likely not because of the issue mentioned in that blog post. Can you see what "files" are open by the Java process running Tomcat? Do those correspond to TCP/IP connections, to temporary files, or something else? If TCP/IP connections, to which services?

Alex

--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet

pc3356

Re: Idle connections not closed (Too many open files)?

Hi Alex,

Thanks for the hint - I'll check it out and see how it looks,

Phil.

On 13 July 2013 05:12, Alessandro Vernet [via Orbeon Forms community mailing list] <[hidden email]> wrote:

Hi Phil,

If this is happening on Linux, it is most likely not because of the issue mentioned in that blog post. Can you see what "files" are open by the Java process running Tomcat? Do those correspond to TCP/IP connections, to temporary files, or something else? If TCP/IP connections, to which services?

Alex
--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet

If you reply to this email, your message will be added to the discussion below:
http://discuss.orbeon.com/Idle-connections-not-closed-Too-many-open-files-tp4656985p4657008.html

To unsubscribe from Idle connections not closed (Too many open files)?, click here.
NAML

pc3356

Re: Idle connections not closed (Too many open files)?

In reply to this post by Alessandro Vernet

Hi Alex,

Looking into it, it looks like almost all the open file descriptors are TCP sockets to the persistence layer in "CLOSE_WAIT" state.

We are using a custom persistence layer; is it possible we're not handling some incoming connections correctly? The API that Orbeon connects to uses Jersey JAX-RS; all the endpoints Orbeon connects to return JAX-RS Responses.

Thanks,

Phil.

On 13 July 2013 05:11, Alessandro Vernet <[hidden email]> wrote:

Hi Phil,

If this is happening on Linux, it is most likely not because of the issue
mentioned in that blog post. Can you see what "files" are open by the Java
process running Tomcat? Do those correspond to TCP/IP connections, to
temporary files, or something else? If TCP/IP connections, to which
services?

Alex

-----
--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet
--
View this message in context: http://discuss.orbeon.com/Idle-connections-not-closed-Too-many-open-files-tp4656985p4657008.html
Sent from the Orbeon Forms community mailing list mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "Orbeon Forms" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].

pc3356

Re: Idle connections not closed (Too many open files)?

After doing a bit more digging, we think this might be a side-effect of our exotic network set-up (which we're not able to do much about)

Is there any way to set the Keep-Alive header (e.g. to force it to turn off or set it to 1 second) within Orbeon? I can't find anything obvious in the code, and the DefaultConnectionKeepAliveStrategy used by the Apache DefaultHttpClient looks for this header, which I can't see anywhere.

Would changing this produce undesirable behaviour elsewhere, for example resource loading?

Thanks,

Phil.

pc3356

Re: Idle connections not closed (Too many open files)?

Before we go down more drastic routes of patching HttpClient to turn off Keep-alive altogether, I've tried adding some header info into our persistence/.../crud.xpl file:

We added:

<header>
    <name>Connection</name>
    <value>close</value>
</header>

Into all the GET "config" elements, and created a new "config" element in:

<p:choose href="#request"/>

(we're using a very slightly modified version of the default)

<config xsl:version="2.0">
    <header>
        <name>Connection</name>
        <value>close</value>
    </header>
</config>

Are these likely to have any effect in passing through the headers? Are these understood by all <p:input/> tags?

Thanks,

Phil.

pc3356

Re: Idle connections not closed (Too many open files)?

It turned out that adding the headers here made no difference.

We've ended up getting round the problem by creating our own copy of DefaultHttpClient that returns an instance of NoConnectionReuseStrategy when the createConnectionReuseStrategy is called.

This effectively tells the application to create a new connection for each request.

We suspect that our network here might have been doing something that Orbeon was handling incorrectly; something upstream was responding with Keep-Alive headers of its own, independent of anything we were sending through.

Cheers,

Phil.

On 18 July 2013 12:04, pc3356 [via Orbeon Forms community mailing list] <[hidden email]> wrote:

Before we go down more drastic routes of patching HttpClient to turn off Keep-alive altogether, I've tried adding some header info into our persistence/.../crud.xpl file:

We added:
<header>
    <name>Connection</name>
    <value>close</value>
</header>
Into all the GET "config" elements, and created a new "config" element in:
<p:choose href="#request"/>
(we're using a very slightly modified version of the default)
<config xsl:version="2.0">
    <header>
        <name>Connection</name>
        <value>close</value>
    </header>
</config>
Are these likely to have any effect in passing through the headers? Are these understood by all <p:input/> tags?

Thanks,

Phil.

If you reply to this email, your message will be added to the discussion below:
http://discuss.orbeon.com/Idle-connections-not-closed-Too-many-open-files-tp4656985p4657025.html

To unsubscribe from Idle connections not closed (Too many open files)?, click here.
NAML

Alessandro Vernet

Re: Idle connections not closed (Too many open files)?

Administrator

Hi Phil,

Good to know, and sorry for now having replied to this earlier, but I really wouldn't have been able to help. If this makes a difference to you, maybe there is a way to do this without patching HttpClient, but instead changing Orbeon Forms? The code you're most likely to want to change is in:

https://github.com/orbeon/orbeon-forms/blob/master/src/main/java/org/orbeon/oxf/resources/handler/HTTPURLConnection.java

And if you think this change is generic enough, and you want to add a property to set it up, maybe this is something we can even put in the codebase.

Alex

--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet

pc3356

Re: Idle connections not closed (Too many open files)?

Hi Alex,

Yes, this is where we looked.

It would be useful to update all of the code around this to make use of the latest set of Apache HttpComponents, as I think more properties are exposed.

I'll take a look today to see if I can work out what I think we would need to change in HTTPURLConnection, and what other configuration options this might give,

Many thanks,

Phil.

On 23 Jul 2013, at 07:11, Alessandro Vernet wrote:

> Hi Phil,
>
> Good to know, and sorry for now having replied to this earlier, but I really
> wouldn't have been able to help. If this makes a difference to you, maybe
> there is a way to do this without patching HttpClient, but instead changing
> Orbeon Forms? The code you're most likely to want to change is in:
>
> https://github.com/orbeon/orbeon-forms/blob/master/src/main/java/org/orbeon/oxf/resources/handler/HTTPURLConnection.java
>
> And if you think this change is generic enough, and you want to add a
> property to set it up, maybe this is something we can even put in the
> codebase.
>
> Alex
>
> -----
> --
> Follow Orbeon on Twitter: @orbeon
> Follow me on Twitter: @avernet
> --
> View this message in context: http://discuss.orbeon.com/Idle-connections-not-closed-Too-many-open-files-tp4656985p4657046.html
> Sent from the Orbeon Forms community mailing list mailing list archive at Nabble.com.
>
> --
> You received this message because you are subscribed to the Google Groups "Orbeon Forms" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
> To post to this group, send email to [hidden email].
>
>

Alessandro Vernet

Re: Idle connections not closed (Too many open files)?

Administrator

Hi Phil,

Sounds good, you'll let us know. And as you've seen, we use the venerable HTTPClient, which is part of the Apache's HTTP Component umbrella project, but maybe there is better way to do it.

Alex

--
Follow Orbeon on Twitter: @orbeon
Follow me on Twitter: @avernet