1 Reply Latest reply on Feb 21, 2015 10:49 AM by Jon Tara

    Streaming JSON parser?

    Jon Tara

      Is anyone aware of any streaming JSON parser ported as a Rhodes extension?


      Rhodes has built-in JSON parsing, but (as far as I know) it only supports converting a complete string into a Ruby structure. It does not support parsing a stream, and then deciding (e.g. via callbacks) what to do with the parts as they are recognized. (Rhodes XML parser does - it has a streaming option.)


      This is important if you should want to read a large chunk of JSON and either build a Ruby object or load it into a database. Without a streaming parser, you need to fit the whole thing into a big string first, and then to add insult to injury the parser will build a big Ruby object while the string still exists in memory!


      The built-in parser is fine for small objects, and I've used it for fairly large ones, too, but I've had the luxury of only working with recent iOS hardware, which has a good amount of memory.


      Now I have a project where I might be dealing with a very big JSON document that I will download. I'd rather not have to save it to a file first, and I'd rather not load the whole string and object in memory. I just want to pick off some hashes as they are streamed and add them to a Rhom object.


      oJ looks like the most performant thing out there for Ruby (faster even than `yajl-Ruby`). If nobody else has done it, I might port it to a Rhodes extension and then open-source it.


      yaJl-Ruby is a wrapper around the C library yaJl. oJ is a RubyGem that has a C extension built-in. It is not a wrapper around some stand-alone C library. I'm guessing part of it's performance advantage is that the C code must be specifically-tailored for Ruby.


      But first want to confirm that Rhodes has no built-in streaming JSON parser, and that somebody else hasn't already done the job and willing to share.


      It's not an immediate need, as I'll get-by for a while on what is built-in.


      Of course, if the RhoMobile team should agree what a great idea this is, and beat me to it, I'd be much obliged!

        • Re: Streaming JSON parser?
          Jon Tara

          Oh. YAML can stream, and JSON is valid YAML. (Hmmm.... Rhodes has YAML? I would assume, right? I only use it in rake right now, though.)


          But, performance-wise, Ruby YAML is way slower than any of the alternatives, including Ruby JSON. (According to benchmarks I have seen.)


          As far as streaming from HTTP, Network doesn't stream - I'd have to use native Ruby networking, which would need a Ruby thread to avoid blocking, and... no Ruby threads on Android! (Still...) But Network does support read to a file, which is not a really awful alternative, since it does have the advantage of reading the document at the fastest possible rate because no parsing and no database insertion. Then I could use standard Ruby File support which at worst I can read in fixed-size chunks and pass to a streaming parser.


          Maybe this is the best way, it gets all the time-consuming stuff out of the way or parsing and Rhom.


          Anyone else dealing with big chunks of JSON? Of course, that's a bad design, but we all encounter bad design that we didn't do and have to live with...