Igor Sysoev wrote:
> On Mon, Jul 13, 2009 at 04:02:06PM -0400, Jim Ohlstein wrote:
>
>
>> Igor Sysoev wrote:
>>
>>> On Mon, Jul 13, 2009 at 03:33:21PM +0400, Maxim Dounin wrote:
>>>
>>>
>>>
>>>> Hello!
>>>>
>>>> On Fri, Jul 10, 2009 at 07:39:46PM -0400, Jim Ohlstein wrote:
>>>>
>>>>
>>>>
>>>>> I'm using the fastcgi cache for static files (images, javascript,css)
>>>>> and just found multiple lines in the error log like this one:
>>>>>
>>>>> 2009/07/10 10:22:54 [crit] 22476#0: ngx_slab_alloc() failed: no memory
>>>>> in cache keys zone "one"
>>>>>
>>>>> So I increased the memory available for the zone and reloaded nginx. It
>>>>> took over five hours to go through the cache but these are the relevant
>>>>> entries:
>>>>>
>>>>> 2009/07/10 12:11:03 [notice] 21038#0: start cache manager process 32730
>>>>> 2009/07/10 12:11:04 [notice] 21038#0: cache manager process 22480 exited
>>>>> with code 0
>>>>>
>>>>> and finally
>>>>>
>>>>> 2009/07/10 17:43:27 [notice] 32730#0: http file cache:
>>>>> /usr/local/nginx/cache 11638.289M, bsize: 4096
>>>>>
>>>>> My questions are:
>>>>>
>>>>> Is that simply the total (11638.289MB or 11.4GB) of all of the file
>>>>> sizes, or is that the actual disk space consumed taking into account
>>>>> total number of blocks used multiplied by the block size? The number
>>>>>
>>>>>
>>>> It's size on disk (i.e. number of blocks * block size), but for
>>>> files only (it doesn't take directories into account).
>>>>
>>>>
>>> Just note: nginx rounds a file size to the bsize.
>>> bsize is f_bsize from statfs() or f_frsize from statvfs().
>>> I'm not sure that bsize matches always a filesystem allocation unit.
>>>
>>>
>> OK, Thanks.
>>
>> In trying to tune this, if I set fastcgi_cache_min_uses to 2, does that
>> mean that a file will only be written to the cache the second time that
>> it is requested? Google translate did not give me a clear answer to this
>> from the Russian documentation. I think that I could improve efficiency
>> greatly if I didn't cache files on the first request.
>>
>
> Yes, a file will be cached only on the second request made during period
> set in by "inactive" option of fastcgi_cache_path directive.
>
>
>> I would know
>> better if I could get some statistics. I know the last time I asked the
>> answer was "not yet". Do you have any idea when this might be
>> implemented even on a rudimentary basis?
>>
>
> You may log $upstream_cache_status.
>

Can you give me an example of how to do this?

I have tried various permutations of:

location ~ (jpg|jpeg|png|gif|ico|js|css)$ {
fastcgi_pass unix:/tmp/my.sock;
fastcgi_cache one;
fastcgi_cache_key unix:/tmp/my.sock.1$request_uri;
fastcgi_ignore_headers Cache-Control Expires;
fastcgi_cache_valid 200 302 1d;
fastcgi_cache_valid 301 7d;
fastcgi_cache_valid any 10m;
fastcgi_cache_min_uses 2;
include /usr/local/nginx/conf/fastcgi_params;
fastcgi_param SCRIPT_NAME myscript;
fastcgi_buffers 64 8k;
access_log logs/my-cache.access.log;
log_format '$remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent"' '
$upstream_cache_status ';
}

The logs give me all above with exception of "$upstream_cache_status".
Every entry ends with the user agent.

>
>>>>> could be quite different given what I estimate are nearly one million
>>>>> mostly small files in the cache at this point.
>>>>>
>>>>> When I next upgrade nginx (I'm running 0.8.4), and I attempt a "graceful
>>>>> upgrade" will it have to go through this entire process again?
>>>>>
>>>>>
>>>> Yes. Binary upgrade doesn't preserve shared memory zones, so
>>>> cache will be rescanned again.
>>>>
>>>> Note that nginx uses cache even before it was completely scanned,
>>>> so it shouldn't be a problem.
>>>>
>>>>
>> Yes, but it uses a fair bit of system resources scanning a large cache.
>> Nothing much to be done about it though.
>>
>
> I tried to minimize impact of cache scanning according to this comment:
>
> /*
> * if processing 100 files takes more than 200ms,
> * it seems that many operations require disk i/o,
> * therefore sleep 200ms
> */
>
>
>
Jim