Welcome! Log In Create A New Profile

Advanced

Re: Centralized logging for multiple servers

Gabriel Ramuglia
April 17, 2009 01:53PM
Hi.

the first thing you have to deal with is multicast. It's designed so
that you can multicast your data to multiple servers without having to
specify several destinations and send the traffic to each separately.
However, if your switches / routers don't support multicast, it will
automatically fall back to broadcasting, even if you only have one
server as the destination for the data. My web hosting provider
quickly cut off the server that was broadcasting so much data to
everyone on the subnet. Broadcasts, since they go to everyone, can
slow down all the servers on your network as they all have to decide
what to do with the packets they're receiving. you can override this
behavior to singlecast if you only have one destination server, but
not if you have more than one.

secondly is the problem of getting it up and running in the first
place, setting up your various variables, compiling and installing,
which is not particularly simple.

Then you have to decide, how is my data going to get into spread?
You're going to need a program that sends the data into spread. Maybe
you'll use a perl script, that seems to be popular. One way to do that
is to pipe your log file output to a perl script. But what happens
when the perl script unexpectedly dies? The application that is piping
out to the script fails. You'll have to notice that this has occurred,
and kill your hosting software (nginx, apache, squid, whatever) as
well as the perl script, and restart the perl script and then your
hosting software, in that order. Your perl script can die for all
sorts of reasons, not least of which that it lost contact with the
spread server for too long, or it's queue of messages to send across
the pipe got too long. Yeah, definitely what you want when you've got
a backlog of 10,000 requests to send across is to lose them all when
your logging program crashes.

Ok so another method is to have your hosting platform log to a file as
normal, but have your perl script attach to the file with something
like tail -f | whatever.pl. That can work, but it suffers the same
problems with dying unexpectedly, the only difference being that when
the perl script dies, logging to the file continues and the hosting
platform doesn't unexpectedly die.

the other issue is performance. the receiving spread server has to be
able to process all these incoming "messages" as they're called from
all your servers, do something with them, and be available to receive
more messages, without crashing. Again, this is probably a perl
script. And, again, I had no real trouble creating a huge load on the
receiving spread server when it was receiving real time log data from
just one server. I have 20. If I was lucky, I could have gotten it
doing 2 or 3 servers of realtime log data. 20 was not going to happen.

The whole thing with spread is that, in theory, it is designed to be
robust, but in my experience it is far from it, the whole operation
seemed quite fragile. It looked like the amount of effort I was going
to have to put in to write programs to make sure that spread was
working properly, and to work around potential failure conditions in
an elegant way, was obscene. I'm sure spread has a number of good
uses, but I could not recommend it for centralized logging. It does
look interesting for a program called whack-a-mole, which is designed
to help you set up your servers in high availability, but that
requires a lot fewer messages flying around than log files would be.

SCP'ing or rsyncing a file from your source server to your centralized
logging server is a lot more robust. Those transfer programs have a
number of protections to make sure the file got there intact, can do
compression, whatever. And you don't have to transfer your log files
line-by-line with scp or rsync, you can dump huge amounts of data
across, and then have your central log server process them at it's
leisure. If it can't keep up with peak demand, it can catch up when
the site isn't as busy, so you don't have an end-of-the-world scenario
if the centralized log server can't keep up with the generation of
real time logs.

On Fri, Apr 17, 2009 at 4:54 AM, Zev Blut <zblut@cerego.co.jp> wrote:
> Hello Gabriel,
>
> I'd like to know why.
Subject Author Posted

Centralized logging for multiple servers

Kingsley Foreman April 12, 2009 11:26PM

Re: Centralized logging for multiple servers

Anton Yuzhaninov April 13, 2009 02:47PM

Re: Centralized logging for multiple servers

Gena Makhomed April 13, 2009 03:31PM

Re: Centralized logging for multiple servers

Igor Sysoev April 15, 2009 09:12AM

Re: Centralized logging for multiple servers

Glen Lumanau April 15, 2009 09:23AM

Re: Centralized logging for multiple servers

Dave Cheney April 15, 2009 09:32AM

Re: Centralized logging for multiple servers

Kingsley Foreman April 15, 2009 09:35AM

Re: Centralized logging for multiple servers

Glen Lumanau April 15, 2009 09:36AM

Re: Centralized logging for multiple servers

Dave Cheney April 15, 2009 10:06AM

Re: Centralized logging for multiple servers

mike April 15, 2009 11:44AM

Re: Centralized logging for multiple servers

W. Andrew Loe III April 16, 2009 10:37PM

Re: Centralized logging for multiple servers

Gabriel Ramuglia April 16, 2009 11:41PM

Re: Centralized logging for multiple servers

W. Andrew Loe III April 16, 2009 11:55PM

Re: Centralized logging for multiple servers

mike April 17, 2009 12:09AM

Re: Centralized logging for multiple servers

Gabriel Ramuglia April 17, 2009 06:32AM

Re: Centralized logging for multiple servers

mike April 17, 2009 11:43AM

Re: Centralized logging for multiple servers

Gabriel Ramuglia April 17, 2009 01:36PM

Re: Centralized logging for multiple servers

Gabriel Ramuglia April 17, 2009 08:07PM

Re: Centralized logging for multiple servers

Kon Wilms April 17, 2009 09:03PM

Re: Centralized logging for multiple servers

Gabriel Ramuglia April 18, 2009 05:22AM

Re: Centralized logging for multiple servers

Kon Wilms April 18, 2009 12:16PM

Re: Centralized logging for multiple servers

Gabriel Ramuglia April 18, 2009 03:03PM

Re: Centralized logging for multiple servers

Kon Wilms April 15, 2009 11:40AM

Re: Centralized logging for multiple servers

Gabriel Ramuglia April 17, 2009 01:53PM



Sorry, only registered users may post in this forum.

Click here to login

Online Users

Guests: 263
Record Number of Users: 8 on April 13, 2023
Record Number of Guests: 421 on December 02, 2018
Powered by nginx      Powered by FreeBSD      PHP Powered      Powered by MariaDB      ipv6 ready