Hey Guys,

Our Software was hit at a rate of 1k hits/s.
To compensate we had 6 servers to spread it out, each of them should
handle this rate easily, but the number of hits spiked to 5k per sec and
php-fpm
spawning went through the roof. We experienced that all servers couldn't
spawn any more child processes and
ran at 200 childs throwing around warnings like christmas presents.
Exactly at that moment our first sessions overlapped. This overlap
happened a few times after also, but always with the same session. So we
had 70 Users crawling in one session. Our sessionhandling is in memcache
handled over tcp. We can show that it is not some kind of malicious session
hijacking.
Also: the application wasn't accessible online and the first error was
reported 8
minutes after it went online. That's to fast to find an error and use it
maliciously in a closed environment.

We are pretty sure the communication between memcache and php-fpm is the
problem. All phenomens can be explained by that. But, we are not quite sure
why this happens.

First Idea: forking and distributing file or socket descriptors with the
fork. Phusion Passenger apparently has that problem with shared memcache
connection if the application spawner opens the connection and
the connection isn't re-established. But as far as i know, php-fpm doesn't
fork from childs.
Could it happen during extrem situationen that processes get forked by
childs?
Does the master php-fpm process handle any request? That would mean a
freakish accident, which
likelyhood to happen is to win the royal lottery...
Because you need an open memcache connection and exactly at that time you
would need
to fork that process.

Next idea: memory corruption or some kind of memory sharing between
processes. That's too deep into php-fpm and i don't know if anything
like this exists even. And that error wasn't happening for a
short amount time, coz the error propagates from user to user for around
2h. With a process handling 500 requests before it kills itself and that
hit rate,
there's no single process which could have survived that long.

I am up for any idea ... coz my best bet is in idea one. It explains
everything to cleanly.


--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

what is you process manager configuration ?

For high load website, you should avoid any process management and use only
a constant number of FPM child processes. For this, use the pm=static and
pm.max_children=200 (as you said that your server was able to handle no
more than 200 FPM children).

Forking new processes takes some time. For high loaded website, this should
be avoided if possible. But this can't especially be done in all
environments, it depends on many things. But this is a good start.

I don't know if this would fix your problem, but at least it would help you
with debugging while having a little bit better performances.


2013/10/19 Jan Hoersch <jhoersch@gmail.com>

> Hey Guys,
>
> Our Software was hit at a rate of 1k hits/s.
> To compensate we had 6 servers to spread it out, each of them should
> handle this rate easily, but the number of hits spiked to 5k per sec and
> php-fpm
> spawning went through the roof. We experienced that all servers couldn't
> spawn any more child processes and
> ran at 200 childs throwing around warnings like christmas presents.
> Exactly at that moment our first sessions overlapped. This overlap
> happened a few times after also, but always with the same session. So we
> had 70 Users crawling in one session. Our sessionhandling is in memcache
> handled over tcp. We can show that it is not some kind of malicious
> session hijacking.
> Also: the application wasn't accessible online and the first error was
> reported 8
> minutes after it went online. That's to fast to find an error and use it
> maliciously in a closed environment.
>
> We are pretty sure the communication between memcache and php-fpm is the
> problem. All phenomens can be explained by that. But, we are not quite
> sure why this happens.
>
> First Idea: forking and distributing file or socket descriptors with the
> fork. Phusion Passenger apparently has that problem with shared memcache
> connection if the application spawner opens the connection and
> the connection isn't re-established. But as far as i know, php-fpm doesn't
> fork from childs.
> Could it happen during extrem situationen that processes get forked by
> childs?
> Does the master php-fpm process handle any request? That would mean a
> freakish accident, which
> likelyhood to happen is to win the royal lottery...
> Because you need an open memcache connection and exactly at that time you
> would need
> to fork that process.
>
> Next idea: memory corruption or some kind of memory sharing between
> processes. That's too deep into php-fpm and i don't know if anything
> like this exists even. And that error wasn't happening for a
> short amount time, coz the error propagates from user to user for around
> 2h. With a process handling 500 requests before it kills itself and that
> hit rate,
> there's no single process which could have survived that long.
>
> I am up for any idea ... coz my best bet is in idea one. It explains
> everything to cleanly.
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "highload-php-en" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to highload-php-en+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

That is valid point. But as far as my understanding of php-fpm process
managing goes right now: child processes kill itself after a certain amount
of requests and respawn. therefore on high loaded websites, where you are
running on a limit anyway, it should make no difference but prevents
flapping of request times, coz you don't have to adjust your amount
processes.
in other words to speak: you will soften the blow, but nothing more.

We had dynamic process managing 50 processes on startup and 200 max. with 2
min and 75 max-spare.

the problem here is not getting the requests done, the problem is that
during getting the requests done. We have communication problem between
memcache and php-fpm but not on every request, just a few.
We are on the roof anyway, but that's an easy problem :)

On Saturday, October 19, 2013 11:19:47 AM UTC+2, Jérôme Loyet wrote:
>
> Hi,
>
> what is you process manager configuration ?
>
> For high load website, you should avoid any process management and use
> only a constant number of FPM child processes. For this, use the pm=static
> and pm.max_children=200 (as you said that your server was able to handle no
> more than 200 FPM children).
>
> Forking new processes takes some time. For high loaded website, this
> should be avoided if possible. But this can't especially be done in all
> environments, it depends on many things. But this is a good start.
>
> I don't know if this would fix your problem, but at least it would help
> you with debugging while having a little bit better performances.
>

--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

What do you mean by "session overlap"? Can you send any log or traces?


--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

We can show from the user-data, that 70 Users, which should have their own
respective sessionid, write into one user (the same on each 70)
Keep in mind, for 2 Million other users it just just fine.
We have access logs during that time, where you can see, that every user
comes from another IP, but we didn't log the sessionid.
So, they can change the respective Data for that User without a login (like
session hijacking)

currently i'm looking into caching of sessionids in memory of one process
from php-fpm

On Monday, October 21, 2013 10:59:18 AM UTC+2, Bike24 wrote:
>
> What do you mean by "session overlap"? Can you send any log or traces?
>

--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

As far as i see the code, the
function fpm_pctl_perform_idle_server_maintenance in fpm_env.c get's called
by master process and every child.
Before there is a check if the current process (which calls that function)
is a child, childrens get maked, therefore children spawn children if you
have high load, coz it's a simple fork we have that problem with open
session descriptors.

can somebody confirm, that this function gets called by a child as well?

--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

fpm_pctl_perform_idle_server_maintenance is only called from the master
process and is called every second.


2013/10/23 Jan Hoersch <jhoersch@gmail.com>

> As far as i see the code, the
> function fpm_pctl_perform_idle_server_maintenance in fpm_env.c get's called
> by master process and every child.
> Before there is a check if the current process (which calls that function)
> is a child, childrens get maked, therefore children spawn children if you
> have high load, coz it's a simple fork we have that problem with open
> session descriptors.
>
> can somebody confirm, that this function gets called by a child as well?
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "highload-php-en" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to highload-php-en+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>

--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

you were a bit faster than me :D wanted to notch you on this thread (git
blame is a nice function)

btp:
you refer to your sanity check in the heartbeat function. it doesn't get
called if parent_pid is the parent_pid.

On Wednesday, October 23, 2013 3:43:13 PM UTC+2, Jérôme Loyet wrote:
>
> fpm_pctl_perform_idle_server_maintenance is only called from the master
> process and is called every second.
>

--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Let's say the fpm_globals.parent_pid is really the parent pid, i would have
to see this pid inside the memory of each process and i can't find it.

we had a spawning rate of 32 during a few times during 18:00 and 18:07..the
rest of the time it's around 8-16. so...the master process decided 3
seconds in a row to double the spawn rate.

On Wednesday, October 23, 2013 3:43:13 PM UTC+2, Jérôme Loyet wrote:
>
> fpm_pctl_perform_idle_server_maintenance is only called from the master
> process and is called every second.
>

--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-en+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.