Hi,

First of all I would like to say that I am quite new to bokeh and nginx.

I have created a simple bokeh that reads in data, performs simple analysis and plots the results. However, the data import takes few seconds and when multiple users are trying to access the webapp at the same time they have to wait until the bokeh server process their request.

My user base has grown and I received quite few requests to address this. One way that I have been trying to apply is by using nginx as a load balancer.

I have used a set up that is similar to the one recommended on https://bokeh.pydata.org/en/latest/docs/user_guide/server.html#load-balancing-with-nginx 1

My docker-compose file looks as follows:

```
version: '3'

services:

upstream:
build: nginx
networks:
webapp-net:
ports:
- "80:80"

webapp0:
build: ../.
networks:
webapp-net:
ports:
- "5100:5100"
command: [
"/webapp/server.sh", "5100"
]

webapp1:
build: ../.
networks:
webapp-net:
ports:
- "5101:5101"
command: [
"/webapp/server.sh", "5101"
]

networks:
webapp-net:
```

My nginx configuration looks as follows:

nginx: nginx.conf

```
user nginx;
worker_processes auto;

events {
worker_connections 768;
}

http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
proxy_read_timeout 200;
types_hash_max_size 2048;

include /etc/nginx/mime.types;
default_type application/octet-stream;

access_log /dev/stdout;
error_log /dev/stdout;

# Increase POST body size
client_max_body_size 100M;

# Include additional configuration files
include /etc/nginx/sites-enabled/*;
}
```

nginx: sites-enabled/default

```
upstream backend_servers {
#least_conn; # Use Least Connections strategy
server webapp0:5100; # Bokeh Server 0
server webapp1:5101; # Bokeh Server 1
}

server {
#listen 80 default_server;
server_name _;

access_log /tmp/bokeh.access.log;
error_log /tmp/bokeh.error.log debug;

location / {
proxy_pass http://backend_servers;

proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host:$server_port;

proxy_buffering off;
}
}
```

The server.sh script is simply:

```
#!/usr/bin/env bash

bokeh serve --port $1 src/bokeh_server.py \
--allow-websocket-origin='*'
```

When I try to simulate simultaneous access to the webapp I can see that the requests are processed one by one:


reating network "docker-compose_webapp-net" with the default driver
Creating docker-compose_webapp1_1 ... done
Creating docker-compose_webapp0_1 ... done
Creating docker-compose_upstream_1 ... done
Attaching to docker-compose_webapp1_1, docker-compose_webapp0_1, docker-compose_upstream_1
webapp1_1 | 2019-08-27 15:37:21,787 Starting Bokeh server version 1.3.4 (running on Tornado 6.0.3)
webapp1_1 | 2019-08-27 15:37:21,788 Host wildcard '*' will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly
webapp1_1 | 2019-08-27 15:37:21,790 Bokeh app running at: http://localhost:5101/bokeh_server
webapp1_1 | 2019-08-27 15:37:21,791 Starting Bokeh server with process id: 7
webapp0_1 | 2019-08-27 15:37:21,805 Starting Bokeh server version 1.3.4 (running on Tornado 6.0.3)
webapp0_1 | 2019-08-27 15:37:21,806 Host wildcard '*' will allow connections originating from multiple (or possibly all) hostnames or IPs. Use non-wildcard values to restrict access explicitly
webapp0_1 | 2019-08-27 15:37:21,809 Bokeh app running at: http://localhost:5100/bokeh_server
webapp0_1 | 2019-08-27 15:37:21,809 Starting Bokeh server with process id: 8
webapp0_1 | 2019-08-27 15:37:31,425 302 GET / (172.21.0.3) 0.93ms
webapp0_1 | 2019-08-27 15:37:31,498 302 GET / (172.21.0.3) 0.63ms
webapp1_1 | 2019-08-27 15:37:33,632 200 GET /bokeh_server (172.21.0.3) 2174.15ms
webapp1_1 | 2019-08-27 15:37:33,634 302 GET / (172.21.0.3) 0.71ms
webapp1_1 | 2019-08-27 15:37:33,642 302 GET / (172.21.0.3) 1.11ms
webapp0_1 | 2019-08-27 15:37:35,062 200 GET /bokeh_server (172.21.0.3) 1425.93ms
webapp0_1 | 2019-08-27 15:37:35,064 302 GET / (172.21.0.3) 0.63ms
webapp0_1 | 2019-08-27 15:37:35,068 302 GET / (172.21.0.3) 0.61ms
webapp1_1 | 2019-08-27 15:37:36,270 200 GET /bokeh_server (172.21.0.3) 1204.42ms
webapp1_1 | 2019-08-27 15:37:36,273 302 GET / (172.21.0.3) 1.17ms
webapp1_1 | 2019-08-27 15:37:36,278 302 GET / (172.21.0.3) 0.95ms
webapp0_1 | 2019-08-27 15:37:37,443 200 GET /bokeh_server (172.21.0.3) 1170.69ms
webapp0_1 | 2019-08-27 15:37:37,444 302 GET / (172.21.0.3) 0.55ms
webapp1_1 | 2019-08-27 15:37:37,450 302 GET / (172.21.0.3) 1.64ms
webapp1_1 | 2019-08-27 15:37:37,454 302 GET / (172.21.0.3) 0.71ms
webapp0_1 | 2019-08-27 15:37:38,651 200 GET /bokeh_server (172.21.0.3) 1200.55ms
webapp0_1 | 2019-08-27 15:37:38,653 302 GET / (172.21.0.3) 0.56ms
webapp0_1 | 2019-08-27 15:37:38,659 302 GET / (172.21.0.3) 0.64ms
webapp1_1 | 2019-08-27 15:37:39,821 200 GET /bokeh_server (172.21.0.3) 1167.23ms
webapp1_1 | 2019-08-27 15:37:39,823 302 GET / (172.21.0.3) 0.64ms
webapp0_1 | 2019-08-27 15:37:39,828 302 GET / (172.21.0.3) 0.59ms
webapp0_1 | 2019-08-27 15:37:39,835 302 GET / (172.21.0.3) 1.10ms
webapp1_1 | 2019-08-27 15:37:41,038 200 GET /bokeh_server (172.21.0.3) 1208.31ms
webapp1_1 | 2019-08-27 15:37:41,040 302 GET / (172.21.0.3) 0.61ms
webapp1_1 | 2019-08-27 15:37:41,046 302 GET / (172.21.0.3) 0.83ms
webapp0_1 | 2019-08-27 15:37:42,265 200 GET /bokeh_server (172.21.0.3) 1223.21ms
webapp0_1 | 2019-08-27 15:37:42,267 302 GET / (172.21.0.3) 0.58ms
webapp1_1 | 2019-08-27 15:37:42,271 302 GET / (172.21.0.3) 0.57ms
webapp1_1 | 2019-08-27 15:37:42,277 302 GET / (172.21.0.3) 0.67ms
webapp0_1 | 2019-08-27 15:37:43,418 200 GET /bokeh_server (172.21.0.3) 1143.51ms
webapp0_1 | 2019-08-27 15:37:43,421 302 GET / (172.21.0.3) 0.85ms
webapp0_1 | 2019-08-27 15:37:43,427 302 GET / (172.21.0.3) 0.76ms
webapp1_1 | 2019-08-27 15:37:44,605 200 GET /bokeh_server (172.21.0.3) 1183.26ms
webapp1_1 | 2019-08-27 15:37:44,607 302 GET / (172.21.0.3) 0.58ms
webapp0_1 | 2019-08-27 15:37:45,797 200 GET /bokeh_server (172.21.0.3) 1189.02ms
webapp1_1 | 2019-08-27 15:37:47,017 200 GET /bokeh_server (172.21.0.3) 1212.78ms
webapp0_1 | 2019-08-27 15:37:48,200 200 GET /bokeh_server (172.21.0.3) 1178.95ms
webapp1_1 | 2019-08-27 15:37:49,380 200 GET /bokeh_server (172.21.0.3) 1172.22ms
webapp0_1 | 2019-08-27 15:37:50,553 200 GET /bokeh_server (172.21.0.3) 1170.79ms
webapp1_1 | 2019-08-27 15:37:51,710 200 GET /bokeh_server (172.21.0.3) 1154.01ms
webapp0_1 | 2019-08-27 15:37:52,815 200 GET /bokeh_server (172.21.0.3) 1099.60ms
webapp1_1 | 2019-08-27 15:37:53,935 200 GET /bokeh_server (172.21.0.3) 1117.51ms
webapp0_1 | 2019-08-27 15:37:55,137 200 GET /bokeh_server (172.21.0.3) 1194.86ms
webapp1_1 | 2019-08-27 15:37:56,344 200 GET /bokeh_server (172.21.0.3) 1203.75ms
webapp0_1 | 2019-08-27 15:37:57,565 200 GET /bokeh_server (172.21.0.3) 1218.49ms
webapp1_1 | 2019-08-27 15:37:58,756 200 GET /bokeh_server (172.21.0.3) 1184.73ms
webapp0_1 | 2019-08-27 15:37:59,943 200 GET /bokeh_server (172.21.0.3) 1185.49ms
webapp1_1 | 2019-08-27 15:38:01,129 200 GET /bokeh_server (172.21.0.3) 1180.08ms

```


I have already asked this question on the Bokeh forum, however, it seems it is more of a nginx related issue eather than bokeh (https://discourse.bokeh.org/t/nginx-bokeh-load-balancing-issue/3941).

Does anyone have any suggestions why the load balancing (asynchronous processing of requests) does not work?

You might think your running as asynchronous processing but it does not sound like it is:

https://www.codemag.com/Article/0102091/Handling-long-Web-Requests-with-Asynchronous-Request-Processing

---
nginx for Windows http://nginx-win.ecsds.eu/

Thank you for your reply itpp2012. Yes, it does seem so, however, it is not clear to me what is wrong with my configuration.

Nothing much, you need to add more backend servers to handle more requests, for example php-cgi is a single thread backend, if requests are handled fast one will suffice if not you add more backends. If 1 request takes 10 seconds you need 10 backends to handle that load for a 10 users, etc.
Or get the backend to handle requests via a pipe asynchronously, tcp handling has its limits.

---
nginx for Windows http://nginx-win.ecsds.eu/

What I expected to see is something as follows:

webapp1_1 | 2019-08-27 15:37:33,000 200 GET /bokeh_server (172.21.0.3) 2000ms
webapp0_1 | 2019-08-27 15:37:33,000 200 GET /bokeh_server (172.21.0.3) 2000ms
webapp1_1 | 2019-08-27 15:37:35,000 200 GET /bokeh_server (172.21.0.3) 2000ms
webapp0_1 | 2019-08-27 15:37:35,000 200 GET /bokeh_server (172.21.0.3) 2000ms

Where two server creating sessions simultaneously.

I have tried using 8 backend servers but I got almost identical performance as with one or two backends.



Edited 1 time(s). Last edit at 08/29/2019 04:34PM by TomasLaz.

Do you see them rotating in the nginx log ? (backend)

"when multiple users are trying to access the webapp at the same time they have to wait until the bokeh server process their request. "
With more than one backend it should work with more than one user but if the backend takes 'a long' time to respond you need to make the backend faster. Or implement a javascript which blocks submits until its finished.

---
nginx for Windows http://nginx-win.ecsds.eu/