Here is what I found out: The cause is keepalive on vBulletin (nginx as proxy for apache) server that is used for authentication from the main web server I had troubles with. I debuged the application and found out the 3 second wait at website loading is always equal to keepalive settings on the vB server. I put 40 as keepalive timeout, then the website waited 40 seconds to load, etc... therefore I turned off keepalive on vB server and all loads instantly.

I do not understand this behavior though. :-? I will investigate further and switch completely to nginx + fastcgiphp or php-fpm.