Netscaler: Resolve large http POST packets against AAA protected LB vServers

Recently while working with two customers and setting up load balancing and external access to some of their applications, I’ve encountered issues with large POST packets (either large form submits or file uploads in the web app) stop working when AAA is activated on the load balancing (LB) vServer. In these cases the backend web applications have all been quite old and not exactly well-written coding-wise.

By default when AAA is activated on Netscaler, any large POST packet sent to the LB vServer, and subsequently to the backend, will be split up into two packets with the first one having a Content-Length header value of 0 and the secondary packet containing the correct Content-Length header value and the actual POST body. The backend code in this case is unable to handle the initial 0 length POST packet, and therefore crashes or bugs out before processing the second, real packet. See picture below of the WireShark capture showing the two packets.

IP adress 10.200.40.159 is a SNIP and 10.200.41.131 is the backend server hosting the web application.

 

The solution is to either have the backend software rewritten to handle above case, or to run a special Netscaler command which will disable the split-post-packet-into-two-packets behaviour. Article https://support.citrix.com/article/CTX225681 contains the relevant info. To resolve this through Netscaler, run the following command in shell on Netscaler, nsapimgr_wr.sh -ys arg1=0 -ys arg2=1 -ys arg3=16 -ys call=”set_sso_post_data_handler”. You have to run the command on all Netscaler nodes (if HA/Cluster) and you also have to put this command line in the file /nsconfig/rc.netscaler, otherwise the change will be lost on the next Netscaler reboot. See https://support.citrix.com/article/CTX122271 for more info regarding the rc.netscaler file.

After the change, you can see how from below picture that only one packet is sent.

One interesting thing to note is that the article https://support.citrix.com/article/CTX225681 doesn’t mention Netscaler version 12.1 as affected, but my guess is that nothing has changes between 12.0 and 12.1 regarding above (ie you still need to run above command to resolve it).

Feel free to email me at rasmus.kindberg@xenit.se, or leave a message here on this blog post, if you have any questions or comments.

Disclaimer: All information on this blog is offered "as is" with no warranty. It is strongly recommended that you verify all information and validate all scripts in isolated test environments before using them in production environments.