9 Replies Latest reply on Aug 18, 2014 10:01 AM by Brendan Higgins

    WebService Nginx Timeouts on Create

    Graham Bird

      We have been reviewing an issue that we have come across and would like some advice on how to proceed and some verification on whether our interpretation of the system is correct.

       

      The system in its entirety is running RhoMobile (on Android) connecting to a Linux/Redis box which in turn is using Web Service calls to perform requests.

      The Linux box is running thin and nginx to host the rhoconnect project.

      RhoConnect is calling a RestClient.post call in the create method (I have removed the error handling from the following code to make it more readable):

        def create(create_hash)

          rest_result = RestClient.post("#{base}", create_hash.to_json, :content_type => :json)

        end

       

      In this case we have created 67 records of a single model type in RhoMobile (potentially with some other creates/updates of other model types).

      When the device(s) in the field sync, we are often (but not always) seeing the following nginx error (details changed to protect IPs etc.):

      2014/03/17 07:45:31 [error] 8301#0: *55204 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 123.123.123.123, server: , request: "POST /app/v1/mymodel HTTP/1.1", upstream: "http://unix:/tmp/thin_x_cluster.1.sock:/app/v1/mymodel", host: "456.456.456.456:9292"

       

      On the web service end of the system, the create for mymodel is creating the 67 records requested (this is always all 67 - never a subset of the creates). However, we then often receive the 67 creation requests again since the RhoMobile device has been told that they failed (due to the timeout).

      This leads me to the conclusion that the sync from the device is passing all 67 create requests through nginx at once. These requests are then being processed (relatively slowly due to it being 67 web service calls) before the whole collection can then return a successful sync response back through nginx to the RhoMobile device.

       

      We could look at increasing the timeout for nginx for the system, but I wanted to ask whether there was any other advice regarding this issue.

      The reason for my concern is that on this occasion we are creating 67 entries. It is always possible that we may have a larger number of records in the future and I want to consider any viable alternatives to just increasing the timeout.

       

      Kind regards,

      Graham Bird.

        • Re: WebService Nginx Timeouts on Create

          I suggest you should make sure that back-end is really slow and it takes many seconds just simply to create records there. If you add some timestamps logging to your model, then you got numbers.

           

          You also might try play with nginx options like

          location / {

               // ...

              proxy_connect_timeout 30;

              proxy_send_timeout    30;

              proxy_read_timeout    30;

              // ...

          }

           

          Increase above values to some reasonable number for your back-end.

          Also enable status page for nginx

          location /nginx_status {

               stub_status on;

               access_log   off;

               allow 127.0.0.1;

               allow SOME.IP.ADD.RESS;

               deny all;

          }

          1 of 1 people found this helpful
            • Re: WebService Nginx Timeouts on Create
              Graham Bird

              As a (hopefully temporary) measure we have increased various timeouts in Nginx.

              We will have to see whether this helps to alleviate the timeout errors we are seeing on the system.

               

              However, as mentioned, increasing timeouts is all very well for this project, but is not a particularly scalable solution to the issue.

              If creating 60 records requires a 1min timeout (as an estimate) then does this mean we would need a 5min timeout to create 300 records?

               

              I would hope to be able to control the sync in a better fashion to either reduce the time required or break up the collection into more manageable, smaller collections that will not break the timeouts.

               

              Many thanks for your advice Alexander as it has at least confirmed which Nginx timeouts we should be looking at in the interim.

               

              Kind regards,

              Graham.

                • Re: WebService Nginx Timeouts on Create
                  Graham Bird

                  Increasing the timeouts in Nginx has definitely reduced the Nginx errors mentioned above.

                  We are now seeing the following errors in the Nginx logs (although significantly less often than the timeouts we were seeing previously:

                  2014/03/21 07:57:20 [error] 19083#0: *4099 upstream prematurely closed connection while reading response header from upstream, client: 123.123.123.123, server: , request: "POST /app/v1/mymodel HTTP/1.1", upstream: "http://unix:/tmp/thin_x_cluster.1.sock:/app/v1/mymodel", host: "456.456.456.456:9292"

                   

                  From a quick search on this error, it looks like this might be the Thin client web requests themselves timing out (or failing) rather than Nginx.

                  Unfortunately, it seems that RhoConnect is not logging any information in the Thin logs at the point of this failure, so this may be very difficult to proceed with.

                   

                  I will continue to update this thread with information if we get any further with identifying/resolving the problem.

                   

                  Kind regards,

                  Graham.

                    • Re: WebService Nginx Timeouts on Create
                      Pietro Francesco Maggi

                      Hi Graham,

                      If you're dealing with operations taking a lot of time to execute on the backend system, it can be a better approach to setup RhoConnect to work asynchronously (maybe keeping query synchronous with Asynchronous Create, Update and Delete):

                      Rhomobile | Adapter Jobs with Resque

                       

                      ~Pietro

                        • Re: WebService Nginx Timeouts on Create
                          Graham Bird

                          The documentation is a little unclear as to how to set this up to work.

                          Primarily, if you set up the source adapter to run asynchronously (in settings/settings.yml) then where do you specify the Resque.enqueue statement? Do you put it inside the corresponding source adapter?

                          Will this mean that all of the create/update/query/delete statements for the specified source adapter will all be run asynchronously? If this is the case, it is overkill since it is only when we have multiple statements of the same type (e.g. create * 67) that we have an issue. A single query for a large number of records does not cause a problem as it is only contacting the (remote) DB once to get all of the records.

                          The other side of this is that surely the query step (used in the example in the documentation) cannot be asynchronous since the rhomobile client is waiting for a response set?

                           

                          I may be misunderstanding how the asynchronous calls will work so I apologise if that is the case.

                           

                          What I need really is a way to gather all of the create requests into an array (or hash?) and then pass this array through to the DB in one go instead of piecemeal.

                          I have no idea if this is possible or how to approach it if it is.

                           

                          Regards,

                          Graham.

                        • Re: WebService Nginx Timeouts on Create

                          Thin logs are in /var/log/thin directory.

                    • Re: WebService Nginx Timeouts on Create
                      Brendan Higgins

                      Hi Graham,

                       

                      Were you able to get any further with this issue?

                       

                      We are experiencing nearly the same issue in our development environment with 3 - 5 users max.  We have logs for our backend APIs and NGINX will time out long before our backend times out.

                       

                      API run times are:
                      81% < 0.5 second

                      91% < 1 second

                      97% < 5 seconds

                      99.8% < 30 seconds

                       

                      Our application yml has a timeout set at 90.

                      Our rhoconnect.conf has:

                         proxy_connect_timeout 60;

                         proxy_send_timeout    60;

                         proxy_read_timeout    60;

                      Our application model javascripts all have request timeouts set at 57.5 seconds:

                           req.setTimeout(57500, function() {

                                new rc.ServerTimeoutException(resp, "Request timeout error.");

                           });

                       

                      Yet we continue to get uncaught "upstream timed out (110: Connection timed out) while reading response header from upstream" from NGINX.

                       

                      It is very frustrating as we seem to be unable to catch this condition and it, obviously, causes issues with our application.

                       

                      Thanks,

                      Brendan