We're using Nginx in a reverse proxy configuration (with TIBCO AMX-BPM as a backend). In 99% of the GTW-RPC calls this works fine, but once every 100 or so requests (repeating same activity in browser over and over) we see that the request payload that was sent by the browser is being 'moved around/wrapped around' in Nginx when looking in the client_body_temp files.

(The same issue is occurring when using Apache as reverse proxy as well, but there we didn't have the actual message bodies being logged, the errors at the backend were pointing to the same issue)

For example, this payload is being sent by browser:

Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Content-Length: 17189
Content-Type: text/x-gwt-rpc; charset=UTF-8
Cookie: JSESSIONID=m602ntlt30in1dzfd7ny3hxh6; s_fid=7908B30519ED0DEB-3144CC6E1F994BDF; s_cahi=%5B%5B%27Direct%27%2C%271551088986099%27%5D%5D; _ga=GA1.2.1729516751.1558944295; s_new_repeat=1563429121440-Repeat
Host: vm00004086.nl.eu.abnamro.com:8090
Origin: http://vm00004086.nl.eu.abnamro.com:8090
Referer: http://vm00004086.nl.eu.abnamro.com:8090/openspace/?locale=en_US&ldap=true
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
X-GWT-Module-Base: http://vm00004086.nl.eu.abnamro.com:8090/openspace/
X-GWT-Permutation: 5F7B9556913825225907FA19BA273057

payload (only kept relevant data to keep it readable) :
---
7|0|18|http://vm00004086.nl.eu.abnamro.com:8090/openspace/|35F26962680C2022276D5EE3DAB53951|com.tibco.openspace.services.shared.BusinessServiceGadgetService|updateBusinessService|com.tibco.bpm.web.client.model.types.PageData/2619873841|com.tibco.bpm.web.client.model.types.IBusinessServiceTemplate|java.lang.String/2004016611|pvm:001g70.m|{ "items":[{"$param":"Delete", "mode":"INOUT", "type":"BOOLEAN", "$value":false},
<....>
com.tibco.bpm.web.client.model.types.BusinessServiceTemplate/1666351301|nl.abnamro.dvb.launchDashboard/DVB_Dashboard|/nl.abnamro.dvb.launchDashboard/Process Packages/DVB_Dashboard.xpdl|MainDashboard|1.0.0.20190613125639319|openspaceGWTPull_DefaultChannel|1|2|3|4|3|5|6|7|5|8|9|10|0|A|11|0|11|0|0|0|11|0|0|12|A|0|13|14|15|16|17|18|
---


Received/processed payload by Nginx :
----
NTEGER", "$value":"30"}]}|http://vm00004086.nl.eu.abnamro.com:8090/bpmresources/1.0.0.20190723113809420/openspaceGWTPull_DefaultChannel/DVB_NewAnalysisbs/OrchestrationProcess/SituationNew/SituationNew.gwt.json|java.util.ArrayList/4159755760|pvm:0a106h|com.tibco.bpm.web.client.model.types.BusinessServiceTemplate/1666351301|nl.abnamro.dvb.launchDashboard/DVB_Dashboard|/nl.abnamro.dvb.launchDashboard/Process Packages/DVB_Dashboard.xpdl|MainDashboard|1.0.0.20190613125639319|openspaceGWTPull_DefaultChannel|1|2|3|4|3|5|6|7|5|8|9|10|0|A|11|0|11|0|0|0|11|0|0|12|A|0|13|14|15|16|17|18|
7|0|18|http://vm00004086.nl.eu.abnamro.com:8090/openspace/|35F26962680C2022276D5EE3DAB53951|com.tibco.openspace.services.shared.BusinessServiceGadgetService|updateBusinessService|com.tibco.bpm.web.client.model.types.PageData/2619873841|com.tibco.bpm.web.client.model.types.IBusinessServiceTemplate|java.lang.String/2004016611|pvm:001g70.m|{ "items":[{"$param":"Delete",

<....>

{"$param":"TotalTenantCount", "mode":"INOUT", "type":"I
----

As you can see, the start of the payload "7|0|18|http://vm00004086.nl.eu.abnamro.com:8090/openspace" is not at the beginning anymore.
Also the end of the message "mode":"INOUT", "type":"I" is continued at the start of the message being logged by Nginx (NTEGER", )

Current (test) config :


http {
include mime.types;
default_type application/octet-stream;
sendfile on;
gzip on;

log_format custom '$remote_addr $http_host $remote_user $timestr "$request" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_time $request_length';

chunked_transfer_encoding off;

server {
listen 10.240.48.246:8090;
if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})") {
set $year $1;
set $month $2;
set $day $3;
}
client_max_body_size 100M;
client_body_buffer_size 8k;
client_body_in_file_only on;
keepalive_timeout 0;

set_formatted_local_time $timestr "%Y-%m-%d %H:%M:%S";
location / {
proxy_pass http://127.0.0.1:8090;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 100M;
access_log logs/access_log_DVB.$year.$month.$day custom;
}
}



Does anyone have an idea how we could fix this ?

We are using Nginx 1.17.1



Edited 1 time(s). Last edit at 07/24/2019 06:35AM by sverzijl.

Thanks for all the responses.

After wireshark tracing, turns out to be a client side issue. Garbage packets/packets in wrong order are being sent by Chrome in rare occasions.
Switching to HTTP/2 which seems to not have this issue in Chrome.