Xojo and load balancing with Nginx

Xojo and load balancing with Nginx

The following post is the results of some tests I did running a small cluster of Xojo apps on a CentOS server and load balancing them with Nginx. I am putting the summary in this post but there is additional discussion on the Xojo forum here and here.

……

OK – so the results are in. Nginx load balancing made a MAJOR difference in usability, if an app is likely at all to get temporarily locked up (processing a large file or something). In the end, this works pretty much as I hoped and allows you to run separate instances of standalone apps across different processors and still keep the connections xojo needs to maintain sessions.

Nginx Conf:

 upstream myproject {
        ip_hash;
        server 127.0.0.1:8080;
        server 127.0.0.1:8081;
        server 127.0.0.1:8082;
        #add a listing for any port you are running an instance on
  }

  server {
    listen 80;
    location / {
        proxy_buffering off;
        proxy_read_timeout 5m;
        proxy_pass http://myproject;
    }
  }

ip_hash; is required to maintain session, app fails to launch without it.
proxy_buffering off; needed to keep data flowing from app to client through proxy
proxy_read_timeout 5m; keeps connection open for app “pings”. If left at default the app will disconnect after 60 seconds. I tried several settings between 1-10 minutes and this was the lowest setting that worked consistently. The app remains open.

In order to start the app instances, I used upstart. I am on CentOS but this should also work on RHEL and possibly Fedora and many others. This is my method of choice because it will automatically respawn instances when they fail, allows easy starting and stopping of the whole group and can exec your app as a non-root user. If you are not familiar with upstart, here are my app.conf files for your reference:

Upstart, first app instance:

# /etc/init/myapp.conf

description 'First instance of app'
author 'john-joyce.com'
env LOG_FILE=/path/to/logfiles/myapp.log
env USER=<put desired username here>

start on runlevel [2345]
stop on runlevel [016]
respawn

script
   touch $LOG_FILE
   chown $USER:$USER $LOG_FILE
   exec su -s /bin/sh -c 'exec "$0" "[email protected]"' $USER -- /path/to/myapp --port=8080 >> $LOG_FILE 2>&1
end script

Upstart, additional instances:

# /etc/init/myapp1.conf

description 'Additional instance of app'
author 'john-joyce.com'
env LOG_FILE=/path/to/logfiles/myapp.log
env USER=<put desired username here>

start on starting myapp
stop on stopping myapp
respawn

script
   exec su -s /bin/sh -c 'exec "$0" "[email protected]"' $USER -- /path/to/myapp --port=8081 >> $LOG_FILE 2>&1
end script

This is just how I did it, if you are launching 20 instances you might want to iterate through it in the script rather than have a file for each instance, but this was easy for just a few instances. You just duplicate the second conf file for as many instances as you want and change the port.

Then when you do ‘initctl <command> myapp’ all of the instances will start, stop, restart.

Observations:

This worked surprisingly well and I would recommend this kind of setup for any app that is getting decent usage. In opening up and rendering an image file I was easily able to get the CPU up to 100% with just one browser, and as most of you know, xojo doesn’t spawn it’s own child process or anything so it is effectively locked up for anyone else trying to use it – even if you are running it on a 24 core server with plenty of memory. Just creating 2 additional instances makes a remarkable difference in performance in a multi-user situation.

-image-

Above you can see 1 instance of the app maxed out and unavailable (CPU 11) while the others are on CPUs 18,19. Instance 1 is totally unresponsive even though there are still 20+ idle cores. But the instances on 18 and 19 are unaffected and running fine. I have to say I was surprised how easy it was to overload a single instance of the app.

It should be noted that because the processes are grouped and routed by IP (same process receives all requests for any one IP) that users in an office using NAT might all appear as one IP to the server and all be routed to the same instance, bypassing the load-balancing for that office.

UPDATE

In order to avoid the above mentioned IP hashing issue, I eventually switched to Haproxy for load balancing and found it to do a fantastic job. You may want to check out my other post on configuring Haproxy if you have the same concerns.

Let me know if this helps anyone out there – hope it does.

John