revirii wrote:
> Hi,
>
> we want to establish some service that will deliver static files
> (images) with a size between 1-120 KB. The service should handle (at the
> moment) ~130.000.000 requests per month, so not that few and not
> extremly many requests. The to-be-served files have a size of several
> hundred GBs in total. Some of the files (5-10%) are asked more
> frequently and the bigger part less frequently.
>
> Let's say we have 16 GB Ram available for the server. We could simply
> avoid using some caching mechanism and read the files from disk. For my
> scenario: is it possible to establish some reasonable caching? No
> decision regarding the to-be-used technology is made so far. An
> associate favours varnish (with some webserver as backend) and some
> least recently used strategy, i'd give nginx+memcached a try. Any
> recommendations?
>

I'm investigating Kyoto Cabinet/Kyoto Tycoon for a caching strategy at
the moment and I can recommend it, though the use case is lots of small
text files. If you don't know, Kyoto Cabinet is a file based "key-value"
database. There are various flavours of Cabinet, one of which is a
"directory" database, so if you create a Cabinet with a '.kcd' extension
then the resulting database just stores each object as a unique file in
a directory.

Kyoto Tycoon then is a HTTP server through which you can access (one or
many) databases via GET and POST. You can access the databases directly,
but one of the benefits of using Tycoon is auto-expiration - whenever
you set a value you can give an expiry time and the file will be removed
after that.

Honestly not an expert in this sort of thing, but the setup I'm looking
at is:

[nginx] -> [4 x Tornado] -> [16 x Kyoto Tycoon] -> [4096 x Kyoto
Cabinet]

Tornado being the Python webserver which, as well as actually retrieving
the data from a remote source, makes the decision of which database to
store items based on a hash of the url. I'm thinking of a single server
(box), but since it's all HTTP I suppose you could have a fan out among
servers if required.

Again, not an expert, and this is all exploratory at this point, so take
with a pinch of salt.




_______________________________________________
nginx mailing list
nginx@nginx.org
http://nginx.org/mailman/listinfo/nginx