Igor Sysoev wrote:
> On Tue, Oct 06, 2009 at 03:56:46PM +0400, Igor Sysoev wrote:
>
>> On Tue, Oct 06, 2009 at 03:46:48PM +0400, Igor Sysoev wrote:
>>
>>> On Sat, Oct 03, 2009 at 09:54:59AM -0400, Jim Ohlstein wrote:
>>>
>>>> Igor Sysoev wrote:
>>>>> On Fri, Oct 02, 2009 at 01:07:24PM -0400, Jim Ohlstein wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm seeing odd disk usage statistics for my cache. The cache is
>>>>>> primarily of small files (images, javascript, and css) and sits on a
>>>>>> 128GB SSD using ext2 file system mounted with "noatime" option. The
>>>>>> cache is all that the the drive is being used for at this time.
>>>>>>
>>>>>> The attached graphs were generated by Munin using df. The data from du
>>>>>> correlates closely with df when run manually.
>>>>>>
>>>>>> As you can see, periodically, with no set interval, disk usage rises
>>>>>> gradually, and then drops precipitously. I don't see anything out of the
>>>>>> ordinary in the error log. The cache serves about 2.8-3 million requests
>>>>>> per day, up from ~2.2-2.4 million one month ago.
>>>>>>
>>>>>> The cache config is:
>>>>>>
>>>>>> fastcgi_cache_path /falcon/cache levels=1:2 keys_zone=one:1024m
>>>>>> inactive=1d max_size=80g;
>>>>>>
>>>>>> where the SSD is mounted as /falcon.
>>>>> As I understand /falcon is used solely for nginx cache ?
>>>> This is correct.
>>>>
>>>>> Could you monitor inode usage also ?
>>>> See attached graphs.
>>>>
>>>>> Did you restart nginx in the times of the drops or not ?
>>>> No.
>>>>
>>>>> Currently I have no idea why it may be.
>>> I saw the similar pattern, when nginx is restarted (not reHUPed) at the cliff
>>> growing start point: then cache loader loads many old inactive cache items
>>> and sets close inactivity timers for the items. Therefore cache size grows
>>> lineary and at some time all these inactive items are purged almost
>>> simultaneously causing precipitous drop.
>>>
>>> Have you restarted nginx at the growing start points ?
>
> This may be not a restart, but an online upgrade too.
>
>
I had error log level set for "crit" so I can't be sure. The binary was
built on September 29 @ 12:39 which was after the the climb started but
it's possible there was a restart earlier. I generally don't restart
except when upgrading or rebooting but it is possible.
I will set error logging to "notice" and upgrade the binary today and
watch what happens.
--
Jim Ohlstein
