13 Replies Latest reply on Mar 1, 2013 7:07 PM by Mark Nongkhlaw

    Can AsyncHttp.post() send binary data?

    Jon Tara

      Can AsyncHttp.post() send binary data?


      I am having difficulty sending .jpg files to Dropbox with AsyncHttp.post(), using DropBox's REST API files_put.


      I've done some experiments, and added some logging to Rhodes.


      If I replace all null characters (0x00) in the .jpg with spaces, the post works.


      If I do not replace them, then Dropbox only receives a content length of 4. (JPEG has a null character in the 5th character always.) This happens if I omit the Content-Length header. However this header is required by Dropbox - it seems to work anyway, but truncates the file.


      If I include the Content-Length header, then operation times-out, because it only sends 4 bytes.


      I added logging to CURLNetRequest.http. It does not log the body length, so I added that:


      CURLcode CURLNetRequest::CURLHolder::perform()



          if ( !rho_conf_getBool("log_skip_post") )

              // [JT] RAWTRACE3("   Activate CURLNetRequest: METHOD = [%s] URL = [%s] BODY = [%s]", mStrMethod.c_str(), mStrUrl.c_str(), mStrBody.c_str());

              // [JT] Added length

              RAWTRACE4("   Activate CURLNetRequest: METHOD = [%s] URL = [%s] LENGTH = [%u] BODY = [%s]", mStrMethod.c_str(), mStrUrl.c_str(), mStrBody.length(), mStrBody.c_str());


              RAWTRACE1("   Activate CURLNetRequest: METHOD = [%s]", mStrMethod.c_str() );


      The length is 4.


      I also added logging to show sent headers. (Rhodes doesn't log this.)


      Somewhere between my call to AsyncHttp.post() and CURLNetRequest::CURLHolder::perform() the string got inappropriate interpreted as a c_str. I haven't been able to discover where that is.


      I 02/13/2013 12:37:59:591 b09ff000              Net| PUT request (Pull): https://api-content.dropbox.com/1/files_put/sandbox/xxxxxxxx%2Fxxxxxxx%20xxxx%2FUsers%2FJon%20Tara%2Fxxxxx%20xxxxx%2F001000_DSPNGM_%5BDay%200%5D%20-%20xxxxxxxx%201%2F001000_DSPNGM_08_xxxxx_130213123734_xxxx.jpg?overwrite=true
      T 02/13/2013 12:37:59:591 b09ff000              Net|Activate CURLNetRequest: METHOD = [PUT] URL = [https://api-content.dropbox.com/1/files_put/sandbox/xxxxxxxx%2FPxxxxxxx%20xxxx%2FUsers%2FJon%20Tara%2Fxxxxxx%20xxxxxx%2F001000_DSPNGM_%5BDay%200%5D%20-%20xxxxxxxx%201%2F001000_DSPNGM_08_xxxxx_130213123734_xxxx.jpg?overwrite=true] LENGTH = [4] BODY = [ÿØÿà]
      T 02/13/2013 12:37:59:591 b09ff000              Net| Send header: Authorization: OAuth oauth_version="1.0", oauth_signature_method="PLAINTEXT", oauth_consumer_key="xxxxx", oauth_token="xxxxx", oauth_signature="xxxxx"
      T 02/13/2013 12:37:59:591 b09ff000              Net| Send header: Content-Length: 14360



      Is binary data supported?

        • Re: Can AsyncHttp.post() send binary data?
          Mark Nongkhlaw

          Not sure if this is even remotely related, but I hope it provides another alternative :


            • Re: Can AsyncHttp.post() send binary data?
              Jon Tara

              Yes, I saw the announcement of the new Dropbox Sync API, and was initially pretty excited about it, despite the fact that we'd like to stick with a broadly cross-platform solution. (It's only available for iOS and Android, and it would need to be packaged as an extension for each platform individually.)


              It turns out, though, that the press misrepresented it a bit. They made it out to work similarly to the desktop Dropbox client. i.e. a directory in your filesystem is just automatically synced, and so you simply read and write local files using standard system file-access functions.


              That's not the case, though. You can only access files sequentially, front to back, and you have to call their file read function.


              There appears to be a bug or limitation in AsyncHttp where it is impossible to send binary data without encoding it. I've been adding some additional logging to Rhodes, and I have it narrowed down to the interface between the C and C++ code. I don't quite understand yet how Rhodes passes parameters down to C++ code yet, (there are a bunch of macros and functions for accessing parameters in C and C++ that I need to understand) so I haven't found the bug yet. The length of the data changes across that interface. My .jpg data is 12K bytes on one side of the interface, and 4 bytes on the other. Clearly, there is a strcpy being used inappropriately where a memcpy is needed.


              Ironically, the data is maintained in a length+data form on both sides of the interface. Ruby stores strings as length+data. \0 has no particular meaning in Ruby strings. As well, there's a layer of C++ code that uses STL strings, which, again, maintain strings as length+data, and \0 has no particular meaning. At the lowest level, libCurl is in C, but is specifically designed to let you pass a length and a pointer to a buffer. If the length is omitted, then libCurl will take the length of the data using strlen, and, of course, it will only get the length up to the first \0. But that is not coming into play here, because I am supplying a content-length, and verified that libCurl is being called WITH the length supplied. However, the length was already truncated up the call chain form there.

                • Re: Can AsyncHttp.post() send binary data?
                  Jon Tara

                  AsyncHttp.post() cannot be used to send binary data. There is a limitation when copying rhoparam in the SWIG interface. If there is a null character in the data, it will terminate the string.


                  AsyncHttp.send_file() only works because the data isn't sent to the low-level code - only a list of file names, and the files are opened by libCurl.


                  Unfortunately, this means that any web service that requires binary data with a simple POST or PUT (like dropbox /files_put or /chunked_upload is unusable with AsyncHttp if there is a null character in the data. Only dropbox /files method is usable. It uses multipart-mime and so can be used with AsyncHttp.send_file().

                    • Re: Can AsyncHttp.post() send binary data?
                      Mark Nongkhlaw

                      Thanks, Jon for the detailed analysis. Looks like Asynchttp cannot also be used to receive binary data via a get, because I've faced issues like data not seeding 100% from a JSON feed because there were extraneous characters in the data, and recently I've seen similar reports made by other people on these forums. But I do recall the very first demo app using RhoElements how they encoded a picture into a base64 string and used that for displaying the picture in the view. Wonder whether that technique can be used with Asynchttp. Of course, this might again necessitate a decoder at the recipient. Your thoughts?

                        • Re: Can AsyncHttp.post() send binary data?
                          Jon Tara

                          It appears that in general Rhodes cannot send binary data to underlying low-level code, period. There's a general-purpose structure call rhoparam that is used for passing parameters. A rhoparam can be a string, array, or hash. An array or hash value can also be a rhoparam.


                          rhoparams are copied (not passed by reference). Problem is, when they are copied, strings are treated a C strings, and so the copy of strings stops with the first null character.


                          I assume there is some code that does need to send binary data down to lower-level code, and in those cases I would assume they don't use rhoparams.

                              • Re: Can AsyncHttp.post() send binary data?
                                Mark Nongkhlaw

                                Its a great heads up. But I sure do hope they assign someone.


                                BTW, how do you create an issue ticket?

                                  • Re: Can AsyncHttp.post() send binary data?
                                    Jon Tara

                                    You can create a ticket on Github. You need to have a Github account. They're free.


                                    Not sure that anybody at RhoMobile is reading them, though.


                                    There was a separate, internal, issue system (not github) where users cannot post, but at least could track issues once they become "real" issues. However, that issue system seems no longer accessible by the public, and all links to those issues no longer work.

                                  • Re: Can AsyncHttp.post() send binary data?
                                    Jon Tara

                                    I've modified Rhodes experimentally to add a :filename parameter to asyncHttp.post(). This allows Rhodes to read the file in low-level code, both bypassing the limitation against embedded NULL characters, and also avoiding copying a potentially-large body content several times as the body is passed down to lower-level code.


                                    Most of my changes are in CurlNetRequest.cpp and AsyncHttp.cpp.


                                    I'd hoped to use libCURLs CURLOPT_READDATA or CURLOPT_READFUNCTION options to do this in the most efficient way possible. However, these appear non-functional in Rhodes. (Although asyncHttp does use CURLOPT_WRITEDATA, at least, to collect response headers and body.)


                                    So, instead, I create a String sized to the file size and use fread to read into the string. This does work to bypass the NULL character limitation. However, I'm still having a problem with large files, which occurred also before making this modification.


                                    This seems to be a second, separate asyncHttp.post() bug. You can't use it to send large files.


                                    This seems suspiciously related to the setting of CURL_MAX_WRITE_SIZE to 200*1024 (204,800) in Android and iOS builds. For other builds, it is set to 16384, which is the default for libCurl.


                                    This is not supposed to limit the size of data sent to the server. It is just an internal buffer that represents the maximum amount of data that Curl will hold internally at any one time for transmission. It will normally shuttle data through this buffer as needed, and should have no limited on the length of data sent.


                                    I have a hunch that setting this value this large is a work-around for some bug that causes curl (in the Rhodes environment) not to recognize that it has sent all of the supplied data if the data is larger than the buffer.


                                    I note that large files (larger than 204,800) can be successfully sent using asyncHttp.post(), but the operation times-out. I think maybe this value was bumped up as a workaround for this bug.

                                      • Re: Can AsyncHttp.post() send binary data?
                                        Mark Nongkhlaw

                                        We expect such an exhaustive analysis from the Rho team, but, no, maybe they're busy elsewhere. Thank God  your github issue post was noticed, though. Looks like they're using Pivotal Tracker. But I doubt you'll have access to that.


                                        I also notice that you and others (me included) are now burdened with having to double-post here and in Google groups :(

                                          • Re: Can AsyncHttp.post() send binary data?
                                            Robert Galvin

                                            Hi Mark & John


                                            I really appreciate your participation here. You are probably one of the most knowledgeable and experienced when it comes to Rho. Many of the users of Launchpad are still beginners in adopting Rho.


                                            You bring up a good point when it comes to the two communities problem. There are several reasons that I cannot go into here why we have not combined them. However now that the adoption has grown, I think we are overdue for combing the two. What I am looking to do is to put the Google groups into 'Archive' mode with access/search from Launchpad and then have both communities using Launchpad as the sole discussion area for Rho. 


                                            Rob Galvin

                                              • Re: Can AsyncHttp.post() send binary data?
                                                Jon Tara

                                                I wish you'd try to fix the usability issues here first. They are a significant impediment to use of this forum. It is a pain to work with.


                                                I suspect that if you put the Google Group in Archive mode, an unofficial one will just pop up. (At least you won't have to moderate it, though!)

                                                • Re: Can AsyncHttp.post() send binary data?
                                                  Mark Nongkhlaw

                                                  First off, let me take this opportunity to thank you Rho guys for this great product. Another great thing is we have 2 forums to bank on for support. As far as I'm concerned, I'm also a novice, if not a newbie, but its been a joyous learning experience. I started off with nil knowledge of Ruby, but with the help of Jon and others, I'm picking up fast :)


                                                  Now, coming to the forums merging issue, yes, I realize its not easy, some users still like the Google group, and they don't come here frequently, if at all. Jon and I were also latecomers here :)


                                                  Coming to the usability part, well, right now I'm using my SGS2, but its not easy to compose a post. Would be great if we had a cross-platform mobile interface if not an app for these forums.


                                                  Just an idea.