Cacheing

> Colin,
>
> > In the regular OPS cache, when I look at the statistics, on the  
> page I
> > am testing now, the log shows a Cache success rate of 88%.
> > Sounds pretty good, but nothing changed between one request and the
> > next, so I am wondering why it is not 100%?
>
> Good question. Without something like a "cache analyzer", it is
> difficult to provide an answer ;-) So much goes through the cache that
> there is probably no intuitive answer either. Does the number change
> if you make the cache larger or smaller?

> > If I look at the HTTP header returned to the browser, both
> > Last-Modified and Expires are set to the moment that the page was
> > generated. Each time it is "generated".
>
> > This raises a number of points
> >
> > * if OPS could achieve 100% success from its own cache, it could
> > look at the If-Last-Modified in the http request, and send back a
> > 304 not modified That's probably hoping for too much
>
> Yes and no, in the case a view only depends on static files, this
> should actually happen, as we have this code in the HTTP serializer:
>
> // Check If-Modified-Since (conditional GET) and don't return  
> content if condition is met
> if (!response.checkIfModifiedSince(lastModified, true)) {
>     response.setStatus(ExternalContext.SC_NOT_MODIFIED);
>     if (logger.isDebugEnabled())
>         logger.debug("Sending SC_NOT_MODIFIED");
>     return;
> }
>
> But there may be cases where this code is not triggered, for some
> reason.

>
> (There is also some "interesting" caching-related code in
> ServletExternalContext.setCaching() ;-)
>
> > * obviously, if you are building a truly dynamic website, then it  
> would
> > be desirable to tell the browser the content it has received has  
> already
> > expired.
> > But many sites/pages are not that dynamic, not completely.
> > But I don't see any way to influence what values are in the  
> response for
> > last-modified nor expires.
>
> Correct, this is meant to be handled automatically by the XPL caching
> mechanism and the HTTP serializer.
>
> > Hopefully I am missing something there.
> >
> > What I am thinking I am needing is some way to set a default  
> expiration
> > datetime (like "now + defaultvalue") and then some way to set a  
> value
> > for specific pages.
> > So a mostly dynamic site would have a default of 0 and set values  
> on the
> > few not so dynamic pages, while a mostly almost-static site with  
> a few
> > truly dynamic pages would have a default of a few hours say and then
> > apply an instant expire to the few dynamic pages.
>
> We have a hack that we have been using for 100% static sites. You can
> add these properties to properties.xml:
>
>   <property as="xs:dateTime"
>       name="oxf.http.force-last-modified"
>       value="2005-06-17T00:00:00"/>
>
>   <property as="xs:boolean"
>       name="oxf.http.force-must-revalidate"
>       value="false"/>
>
> This will set a single last modified date for all the pages of the web
> site that do not receive a "remote user". For those pages, you can
> also control whether you want the server to force the client to
> revalidate the cache when the page is needed again.
>
> On the contrary pages that receive a remote user will not be
> cacheable, with the idea that those may be dynamic pages.
>
> Again, this is a hack of sorts.

> Hi Erik
>
> I have looked into the memory questions and even improved the  
> performance a little.
> I am still interested in seeing what apache2 caching can do for for  
> this project.
>
> If I am reading correctly this apache2 document
> http://httpd.apache.org/docs/2.3/mod/mod_cache.html
> then it is expires and not last-modified that is important for this.
> It seems OPS is setting expires, but not in a particular consistent  
> way, and never for a future date, and I don't seem to have any  
> control over it.
> Similarly, OPS is setting the last-modified, as the current time -  
> unless the hack is used to specify a particular datetime, but this  
> datetime is not relative to the time the page is first served.
>
> Am I reading all this correctly, and is there anything (simple) we  
> can do to make OPS more usable with apqache2 caching?
>
> The optimum would seem to be for OPS not to set the expires, just  
> the last-modified, thus allowing the caching directives to set the  
> expires, and for there to be a way for the pipeline to indicate  
> when the page being served needs to be served as already expired.
> But then, I don't know why OPS is setting the expires, and there  
> may be a very good, conflicting reason for this?
>
> Thanks & regards
> Colin
>
>
>
> --
> You receive this message as a subscriber of the ops-
> [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> ObjectWeb mailing lists service home page: http://www.objectweb.org/ 
> wws

> Colin,
>
> > If I am reading correctly this apache2 document
> > http://httpd.apache.org/docs/2.3/caching.html then it is
> > expires and not last-modified that is important for this.  It seems
> > OPS is setting expires, but not in a particular consistent way, and
> > never for a future date, and I don't seem to have any control over
> > it.  Similarly, OPS is setting the last-modified, as the current
> > time - unless the hack is used to specify a particular datetime, but
> > this datetime is not relative to the time the page is first served.
>
> This is how the HTTP serializer works. First it tries to find a
> last-modified date for the resource produced. If it can't find one,

> The Resource Server processor (used to serve static .js, .css, etc.)
> works a little differently: it does not force revalidation. It still
> uses the Last-Modified date if found, but it sets an Expires header
> with the same heuristic used by web browsers. This was done with the
> idea that static resources in general would be cacheable by the web
> browser.
>
> In other words: static resources should be cacheable in a way similar
> to the case where they are served directly by Tomcat or by Apache. But
> dynamic resources, while they can be cached, always necessitate
> revalidation from the browser. At least, that's how things were
> intended to work...
>
> (If you us the oxf.http.force-last-modified hack, then we change
> things a little.)
>
> > Am I reading all this correctly, and is there anything (simple) we
> > can do to make OPS more usable with apqache2 caching?
>
> What are you trying to cache again? I fear that I have lost the exact
> use case.
>
> > The optimum would seem to be for OPS not to set the expires, just
> > the last-modified, thus allowing the caching directives to set the
> > expires, and for there to be a way for the pipeline to indicate when
> > the page being served needs to be served as already expired.
>
> Now I am wondering if the Expires header should in fact be set when we
> force revalidation, or if it should be set with the usual heuristic
> instead. I am not sure about that part. But can't you with Apache
> override the Expires header to experiment?

> Colin,
>
> > If I am reading correctly this apache2 document
> > http://httpd.apache.org/docs/2.3/caching.html then it is
> > expires and not last-modified that is important for this.  It seems
> > OPS is setting expires, but not in a particular consistent way, and
> > never for a future date, and I don't seem to have any control over
> > it.  Similarly, OPS is setting the last-modified, as the current
> > time - unless the hack is used to specify a particular datetime, but
> > this datetime is not relative to the time the page is first served.
>
> This is how the HTTP serializer works. First it tries to find a
> last-modified date for the resource produced. If it can't find one,

> The Resource Server processor (used to serve static .js, .css, etc.)
> works a little differently: it does not force revalidation. It still
> uses the Last-Modified date if found, but it sets an Expires header
> with the same heuristic used by web browsers. This was done with the
> idea that static resources in general would be cacheable by the web
> browser.
>
> In other words: static resources should be cacheable in a way similar
> to the case where they are served directly by Tomcat or by Apache. But
> dynamic resources, while they can be cached, always necessitate
> revalidation from the browser. At least, that's how things were
> intended to work...
>
> (If you us the oxf.http.force-last-modified hack, then we change
> things a little.)
>
> > Am I reading all this correctly, and is there anything (simple) we
> > can do to make OPS more usable with apqache2 caching?
>
> What are you trying to cache again? I fear that I have lost the exact
> use case.
>
> > The optimum would seem to be for OPS not to set the expires, just
> > the last-modified, thus allowing the caching directives to set the
> > expires, and for there to be a way for the pipeline to indicate when
> > the page being served needs to be served as already expired.
>
> Now I am wondering if the Expires header should in fact be set when we
> force revalidation, or if it should be set with the usual heuristic
> instead. I am not sure about that part. But can't you with Apache
> override the Expires header to experiment?

> Hi Erik,
>
> just wondering if you have had any more thoughts on the attached?
>
> A quick question (not sure how much it really helps, but might be quick
> way to get part way)
>
> you mentioned the addition of oxf.http.force-last-modified in
> properties.xml.
> And the example you gave was of a fixed/absolute time.
> Is it possible (or easy) to have it as a relative time - I would most
> likely set it to 12 or 24 hours, since as I said, most of the time the
> site content does not change (apart from use of a contact form) but
> things do come and go (for instance on a calendar/events page) so
> allowing for occasional updates would be a way to satisfy much of the
> requirement.
>
> Thanks again
> Colin
>
>
>
> On Nov 5, 2006, at 6:43 PM, Colin O'Brien wrote:
>
> Hi Erik
>
> just to recap, what I see as my current issue...
>
> I am using OPS to generate web content
> and one reason is that I am using OPS to give the site owner the ability
> to update some of the content
> which then requires presentation to be dynamic.
> In reality, the content changes infrequently,
> but as currently implemented, a lot of server work occurs for each request.
>
> As part of this dynamic page generation, various components might have
> to be assembled.
> I'm guessing this makes it difficult for OPS to work out the
> last-modified date.
>
> (As an aside, which may or not be relevant, in testing in Eclipse, I am
> seeing around 75/85% success in OPS caching.
> One time I edited an xslt file, and did a refresh of the page, and the
> cache % was unchanged.)
>
> Anyway, I am looking for a way to reduce the server workload and improve
> the responsiveness of our OPS-generated sites to visitors.
> The way we are exploring here is to use the caching capabilities in
> Apache 2.2...
>
> On Oct 30, 2006, at 9:25 AM, Erik Bruchez wrote:
>
>> Colin,
>>
>> > If I am reading correctly this apache2 document
>> > http://httpd.apache.org/docs/2.3/caching.html then it is
>> > expires and not last-modified that is important for this.  It seems
>> > OPS is setting expires, but not in a particular consistent way, and
>> > never for a future date, and I don't seem to have any control over
>> > it.  Similarly, OPS is setting the last-modified, as the current
>> > time - unless the hack is used to specify a particular datetime, but
>> > this datetime is not relative to the time the page is first served.
>>
>> This is how the HTTP serializer works. First it tries to find a
>> last-modified date for the resource produced. If it can't find one,
>
> one obvious question would be "how does it do this?"
> but I don't want you to spend all day writing emails.
> The next question would be, if OPS gets the date wrong,
> what about providing a way for the resource to tell OPS the right date
> to use.
> Every "object" in our sites has an element with date-last-modified.
>
>> then it uses "now" (the current time). Then it sets the Last-Modified
>> header with that value, Expires to the same value, and forces a
>> revalidation. This was designed this way in order to ensure that your
>> browser doesn't show cached pages in case they have changed on the
>> server.
>
> Sorry, my knowledge on this part is not detailed enough to under full
> what you mean by revalidation?
>
> This is obviously over-simplistic, even ignorant on my part, but it
> seems to me that the world developed a lot of ways to address this
> question of whether a document can be cached and/or has expired and you
> are short-circuiting that with what may be an over-simplistic approach.
> I'm sure that you had a very real situation in the past where this was
> the right answer, and that that you have seen many more situations than
> I have.
> Unfortunately it doesn't seem to apply in this case.
>
> The resource being returned by our server knows whether it can be cached
> or not.
>
>> The Resource Server processor (used to serve static .js, .css, etc.)
>> works a little differently: it does not force revalidation. It still
>> uses the Last-Modified date if found, but it sets an Expires header
>> with the same heuristic used by web browsers. This was done with the
>> idea that static resources in general would be cacheable by the web
>> browser.
>>
>> In other words: static resources should be cacheable in a way similar
>> to the case where they are served directly by Tomcat or by Apache. But
>> dynamic resources, while they can be cached, always necessitate
>> revalidation from the browser. At least, that's how things were
>> intended to work...
>>
>> (If you us the oxf.http.force-last-modified hack, then we change
>> things a little.)
>>
>> > Am I reading all this correctly, and is there anything (simple) we
>> > can do to make OPS more usable with apqache2 caching?
>>
>> What are you trying to cache again? I fear that I have lost the exact
>> use case.
>>
>> > The optimum would seem to be for OPS not to set the expires, just
>> > the last-modified, thus allowing the caching directives to set the
>> > expires, and for there to be a way for the pipeline to indicate when
>> > the page being served needs to be served as already expired.
>>
>> Now I am wondering if the Expires header should in fact be set when we
>> force revalidation, or if it should be set with the usual heuristic
>> instead. I am not sure about that part. But can't you with Apache
>> override the Expires header to experiment?
>
> As to overriding the Expires answer, if there is a way, I haven't found it.
> Neither by setting values in httpd.conf nor by using http-equiv.
>
> One simple solution might be add to properties.xml so that the expire
> behavior can be controlled
> - off or on/like-browser or on/always-expire or on/resource-defines
>
> And/or
>
> There is already a way to for the resource to communicate to the
> serializer when I value is required.
> That is "http-equiv".
> If a value is present, the serializer could use it, if none, carries on
> as now.
> This could be used for "expires" and/or for "last-modified".
>
> In my research this past week, I've begun to feel last-modified is
> actually what is more important in this process.
> Expires derives from last-modified, and it would still be relevant to be
> able to specify at the global, site and resource levels what the
> last-modified to expires offset is.
> But it is last-modified that the browser is going to use in its request,
> and so it is last-modified that the cache will use if it passes the
> request on to OPS.
>
> Thanks & regards
> Colin
>
> PS just in case you are wondering - if there is a change that
> necessitates the site to expire, then there are means available in
> apache to clear the cache and it would be easy to repopulate it.
>
>

> Colin,
>
> I was thinking that you could do this with Apache mod_expires. For
> example, we have been using this kind of configuration:
>
>  ExpiresActive on
>  ExpiresByType application/x-javascript "access plus 2000 minutes"
>
> Have you tried such an approach, and would it not solve your problem?
>
> Best,
>
> -Erik
>
> -Erik
>
> Colin O'Brien wrote:
>> Hi Erik,
>> just wondering if you have had any more thoughts on the attached?
>> A quick question (not sure how much it really helps, but might be  
>> quick way to get part way)
>> you mentioned the addition of oxf.http.force-last-modified in  
>> properties.xml.
>> And the example you gave was of a fixed/absolute time.
>> Is it possible (or easy) to have it as a relative time - I would  
>> most likely set it to 12 or 24 hours, since as I said, most of the  
>> time the site content does not change (apart from use of a contact  
>> form) but things do come and go (for instance on a calendar/events  
>> page) so allowing for occasional updates would be a way to satisfy  
>> much of the requirement.
>> Thanks again
>> Colin
>> On Nov 5, 2006, at 6:43 PM, Colin O'Brien wrote:
>> Hi Erik
>> just to recap, what I see as my current issue...
>> I am using OPS to generate web content
>> and one reason is that I am using OPS to give the site owner the  
>> ability to update some of the content
>> which then requires presentation to be dynamic.
>> In reality, the content changes infrequently,
>> but as currently implemented, a lot of server work occurs for each  
>> request.
>> As part of this dynamic page generation, various components might  
>> have to be assembled.
>> I'm guessing this makes it difficult for OPS to work out the last-
>> modified date.
>> (As an aside, which may or not be relevant, in testing in Eclipse,  
>> I am seeing around 75/85% success in OPS caching.
>> One time I edited an xslt file, and did a refresh of the page, and  
>> the cache % was unchanged.)
>> Anyway, I am looking for a way to reduce the server workload and  
>> improve the responsiveness of our OPS-generated sites to visitors.
>> The way we are exploring here is to use the caching capabilities  
>> in Apache 2.2...
>> On Oct 30, 2006, at 9:25 AM, Erik Bruchez wrote:
>>> Colin,
>>>
>>> > If I am reading correctly this apache2 document
>>> > http://httpd.apache.org/docs/2.3/caching.html then it is
>>> > expires and not last-modified that is important for this.  It  
>>> seems
>>> > OPS is setting expires, but not in a particular consistent way,  
>>> and
>>> > never for a future date, and I don't seem to have any control over
>>> > it.  Similarly, OPS is setting the last-modified, as the current
>>> > time - unless the hack is used to specify a particular  
>>> datetime, but
>>> > this datetime is not relative to the time the page is first  
>>> served.
>>>
>>> This is how the HTTP serializer works. First it tries to find a
>>> last-modified date for the resource produced. If it can't find one,
>> one obvious question would be "how does it do this?"
>> but I don't want you to spend all day writing emails.
>> The next question would be, if OPS gets the date wrong,
>> what about providing a way for the resource to tell OPS the right  
>> date to use.
>> Every "object" in our sites has an element with date-last-modified.
>>> then it uses "now" (the current time). Then it sets the Last-
>>> Modified
>>> header with that value, Expires to the same value, and forces a
>>> revalidation. This was designed this way in order to ensure that  
>>> your
>>> browser doesn't show cached pages in case they have changed on the
>>> server.
>> Sorry, my knowledge on this part is not detailed enough to under  
>> full what you mean by revalidation?
>> This is obviously over-simplistic, even ignorant on my part, but  
>> it seems to me that the world developed a lot of ways to address  
>> this question of whether a document can be cached and/or has  
>> expired and you are short-circuiting that with what may be an over-
>> simplistic approach.
>> I'm sure that you had a very real situation in the past where this  
>> was the right answer, and that that you have seen many more  
>> situations than I have.
>> Unfortunately it doesn't seem to apply in this case.
>> The resource being returned by our server knows whether it can be  
>> cached or not.
>>> The Resource Server processor (used to serve static .js, .css, etc.)
>>> works a little differently: it does not force revalidation. It still
>>> uses the Last-Modified date if found, but it sets an Expires header
>>> with the same heuristic used by web browsers. This was done with the
>>> idea that static resources in general would be cacheable by the web
>>> browser.
>>>
>>> In other words: static resources should be cacheable in a way  
>>> similar
>>> to the case where they are served directly by Tomcat or by  
>>> Apache. But
>>> dynamic resources, while they can be cached, always necessitate
>>> revalidation from the browser. At least, that's how things were
>>> intended to work...
>>>
>>> (If you us the oxf.http.force-last-modified hack, then we change
>>> things a little.)
>>>
>>> > Am I reading all this correctly, and is there anything (simple) we
>>> > can do to make OPS more usable with apqache2 caching?
>>>
>>> What are you trying to cache again? I fear that I have lost the  
>>> exact
>>> use case.
>>>
>>> > The optimum would seem to be for OPS not to set the expires, just
>>> > the last-modified, thus allowing the caching directives to set the
>>> > expires, and for there to be a way for the pipeline to indicate  
>>> when
>>> > the page being served needs to be served as already expired.
>>>
>>> Now I am wondering if the Expires header should in fact be set  
>>> when we
>>> force revalidation, or if it should be set with the usual heuristic
>>> instead. I am not sure about that part. But can't you with Apache
>>> override the Expires header to experiment?
>> As to overriding the Expires answer, if there is a way, I haven't  
>> found it.
>> Neither by setting values in httpd.conf nor by using http-equiv.
>> One simple solution might be add to properties.xml so that the  
>> expire behavior can be controlled
>> - off or on/like-browser or on/always-expire or on/resource-defines
>> And/or
>> There is already a way to for the resource to communicate to the  
>> serializer when I value is required.
>> That is "http-equiv".
>> If a value is present, the serializer could use it, if none,  
>> carries on as now.
>> This could be used for "expires" and/or for "last-modified".
>> In my research this past week, I've begun to feel last-modified is  
>> actually what is more important in this process.
>> Expires derives from last-modified, and it would still be relevant  
>> to be able to specify at the global, site and resource levels what  
>> the last-modified to expires offset is.
>> But it is last-modified that the browser is going to use in its  
>> request, and so it is last-modified that the cache will use if it  
>> passes the request on to OPS.
>> Thanks & regards
>> Colin
>> PS just in case you are wondering - if there is a change that  
>> necessitates the site to expire, then there are means available in  
>> apache to clear the cache and it would be easy to repopulate it.
>
>
> --
> Orbeon Forms - Web Forms for the Enterprise Done the Right Way
> http://www.orbeon.com/
>
>
> --
> You receive this message as a subscriber of the ops-
> [hidden email] mailing list.
> To unsubscribe: mailto:[hidden email]
> For general help: mailto:[hidden email]?subject=help
> ObjectWeb mailing lists service home page: http://www.objectweb.org/ 
> wws