All files from this thread

File Name	File Size		Posted by	Date
localhost-df-day.png	16.5 KB	open | download	Jim Ohlstein	10/02/2009	Read message
localhost-df-month.png	20.1 KB	open | download	Jim Ohlstein	10/02/2009	Read message
localhost-df_inode-day.png	16.3 KB	open | download	Jim Ohlstein	10/03/2009	Read message
localhost-df_inode-month.png	19.6 KB	open | download	Jim Ohlstein	10/03/2009	Read message

On Tue, Oct 06, 2009 at 03:56:46PM +0400, Igor Sysoev wrote:

> On Tue, Oct 06, 2009 at 03:46:48PM +0400, Igor Sysoev wrote:
>
> > On Sat, Oct 03, 2009 at 09:54:59AM -0400, Jim Ohlstein wrote:
> >
> > > Igor Sysoev wrote:
> > > > On Fri, Oct 02, 2009 at 01:07:24PM -0400, Jim Ohlstein wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >> I'm seeing odd disk usage statistics for my cache. The cache is
> > > >> primarily of small files (images, javascript, and css) and sits on a
> > > >> 128GB SSD using ext2 file system mounted with "noatime" option. The
> > > >> cache is all that the the drive is being used for at this time.
> > > >>
> > > >> The attached graphs were generated by Munin using df. The data from du
> > > >> correlates closely with df when run manually.
> > > >>
> > > >> As you can see, periodically, with no set interval, disk usage rises
> > > >> gradually, and then drops precipitously. I don't see anything out of the
> > > >> ordinary in the error log. The cache serves about 2.8-3 million requests
> > > >> per day, up from ~2.2-2.4 million one month ago.
> > > >>
> > > >> The cache config is:
> > > >>
> > > >> fastcgi_cache_path /falcon/cache levels=1:2 keys_zone=one:1024m
> > > >> inactive=1d max_size=80g;
> > > >>
> > > >> where the SSD is mounted as /falcon.
> > > >
> > > > As I understand /falcon is used solely for nginx cache ?
> > >
> > > This is correct.
> > >
> > > > Could you monitor inode usage also ?
> > >
> > > See attached graphs.
> > >
> > > > Did you restart nginx in the times of the drops or not ?
> > >
> > > No.
> > >
> > > > Currently I have no idea why it may be.
> >
> > I saw the similar pattern, when nginx is restarted (not reHUPed) at the cliff
> > growing start point: then cache loader loads many old inactive cache items
> > and sets close inactivity timers for the items. Therefore cache size grows
> > lineary and at some time all these inactive items are purged almost
> > simultaneously causing precipitous drop.
> >
> > Have you restarted nginx at the growing start points ?

This may be not a restart, but an online upgrade too.


--
Igor Sysoev
http://sysoev.ru/en/