It will need to spawn the pool, if you're on ultra-cheap-ultra-low-powered vps, this can be slow, because of slow cpu, low ram and very small disk cache portion which belongs to your vps, on a dedicated machine with 2 GB ram i dont notice any delays with spawning fpm pools on demand. Anyway i dont see much options for you with 128 Megs, even better configs can be slow, depending on the vps provider and hypervisor they use.

Additional suggestion to you - remove PHP extensions which are not needed

extension=hash.so
extension=session.so
extension=dom.so
extension=xml.so
extension=mysql.so
extension=mbstring.so
extension=zlib.so
extension=gd.so
extension=curl.so
extension=json.so
;extension=bcmath.so
extension=exif.so
extension=mcrypt.so
extension=simplexml.so
;extension=pdo_mysql.so
extension=sysvmsg.so
extension=tokenizer.so
extension=ctype.so
extension=mysqli.so
extension=iconv.so
extension=gmp.so
extension=openssl.so
extension=gettext.so
extension=zip.so

anything except that list is most likely just hogging your RAM )

Make sure to install a PHP opcode cache (not less than 32M), it will hog its own piece of RAM but will decrease RAM consumption for processing scripts, i.e. Wordpress 28 Mb to 4 Mb, eaccelerator still can swap to disk precompiled scripts, therefore (at my look) it will be the best choice (you will need master branch from GIT for PHP 5.4)