Welcome! Log In Create A New Profile

Advanced

Re: Serving static file dynamically ...

Marcus Clyne
March 25, 2009 02:07PM
Hi Brice,

Firstly, I'm not talking about a full-blown database that's serving
files, but a lightweight front to the files (as PBMS is). PBMS is just
an HTTP front to a storage engine for MySQL, and doesn't deal with SQL
or anything like that. It's really like serving the content directly
out of a large file, in a pretty similar way to many caches do.

Ordinarily, the filesystem will be much faster than even the lightest of
fronts to a db, but if you have milllions of files, then each file will
have metadata associated with it (which takes up space - usually at
least 4KB) and the filesystem has to cope with all the files, and many
filesystems struggle when you start getting to large numbers of files,
and it slows things down.

If you have billions of files, then you couldn't even serve them off a
normal 32-bit fs, because you'd run out of inodes (I believe).

For thousands of files, or tens of thousands, you'd be fine, though, and
the filesystem will definitely be quicker than PBMS.

When I did my tests with PBMS, I created 2M objects, and put them in the
database with MySQL, then served them using PBMS. My benchmarks showed
that the number of req/s I could serve was similar to Apache 2 at best,
and about 20% slower at worst (it depended on the index of the file).
That might not seem great, but that was serving from 1 file vs 2M. I
tried creating files in a hierarchical structure, and after I got to
around 400k (? I can't quite remember), my system almost completely
stalled, so I stopped trying to add files.

In most scenarios, the filesystem will be quicker, but not always.

Cheers,

Marcus.


Brice Leroy wrote:
>
> On Mar 24, 2009, at 1:55 PM, Marcus Clyne wrote:
>
>> Michael Shadle wrote:
>>> not to mention that this is only useful if the OP is storing files in
>>> the database to begin with, I believe.
>>>
>>> if it's filesystem, then X-Accel-Redirect is the way to go.
>>>
>> Yes, if the filesystem is the storage mechanism, then I'd agree. If
>> you have a very large number of files, though, storing them in the DB
>> can be more efficient than using the filesystem (depending on
>> platform, file number, directory hierarchy etc).
>
> How can the filesystem can be slower than DB to serve huge video files
> ? That's completely on the opposite to my culture ! Can you explain me
> how this is possible ?
> My situation will be serving thousand(maybe more later) of different
> big files(between 200MB and 4GB) to different users.
>
> Brice
>
>
Subject Author Posted

Serving static file dynamically ...

Brice Leroy March 24, 2009 02:05PM

Re: Serving static file dynamically ...

Jean-Philippe Moal March 24, 2009 02:37PM

Re: Serving static file dynamically ...

Brice Leroy March 24, 2009 02:47PM

Re: Serving static file dynamically ...

Marcus Clyne March 24, 2009 02:49PM

Re: Serving static file dynamically ...

Marcus Clyne March 24, 2009 03:51PM

Re: Serving static file dynamically ...

mike March 24, 2009 04:12PM

Re: Serving static file dynamically ...

Marcus Clyne March 24, 2009 04:55PM

Re: Serving static file dynamically ...

Brice Leroy March 24, 2009 05:35PM

Re: Serving static file dynamically ...

mike March 24, 2009 05:56PM

Re: Serving static file dynamically ...

Brice Leroy March 24, 2009 06:17PM

Re: Serving static file dynamically ...

Marcus Clyne March 25, 2009 02:07PM

Re: Serving static file dynamically ...

Brice Leroy March 25, 2009 02:49PM

Re: Serving static file dynamically ...

Marcus Clyne March 25, 2009 03:34PM

Re: Serving static file dynamically ...

Brice Leroy March 25, 2009 05:28PM

Re: Serving static file dynamically ...

Juan Fco. Giordana March 27, 2009 03:03PM

Re: Serving static file dynamically ...

Yordan Georgiev March 27, 2009 03:26PM

Re: Serving static file dynamically ...

Brice Leroy March 27, 2009 03:39PM

Re: Serving static file dynamically ...

Adam Zell March 25, 2009 03:02PM

Re: Serving static file dynamically ...

Cliff Wells March 24, 2009 04:09PM

Re: Serving static file dynamically ...

Marcus Clyne March 24, 2009 04:50PM

Re: Serving static file dynamically ...

Brice Leroy March 24, 2009 05:41PM

Re: Serving static file dynamically ...

Marcus Clyne March 25, 2009 02:11PM

Re: Serving static file dynamically ...

Brice Leroy March 24, 2009 03:26PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 278
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready