Archive

Monthly Archives: April 2010

Update: I ran tests again using newer version of protoc-gen-as3. They can be found here.

For last couple of days I have been investigating Google’s Protocol Buffers with intention to use it in Flex application. While in Flex world such idea instantly raises at least a couple of questions, let me explain our situation a little bit more.

We are looking into using Kaazing, which is quite low level and data format agnostic solution, to provide us with streaming capability. This means we have a choice of format we want to use for our messages (and streaming protocol, but that’s a topic for another post). We have never really considered XML because of message size, performance and Flash Player bugs (apparently fixed in version 10.1). Using AMF3 would give us a lot of developer friendliness and client-side performance, thanks to native FP support, but in our case would also add additional (and probably not insignificant) load on the server. As Protocol Buffers are already used to communicate between deeper layers of backend, we decided we could try to push PB messages up to the client, without having to translate them into another format. As that solution would also tick a couple of boxes on our nice-to-have list, we decided to give it a go.

During my investigation I concentrated on performance as this is very important factor for us. Our application will need to receive more than hundred messages per second and update view with minimal latency. As outbound traffic will not ever be as high, I was mostly interested in deserialization performance. To have some point of reference, I decided to test Protocol Buffers against AMF3, which in Flex world is a synonym for “as fast as you can get”. I used protoc-gen-as3 project, which seems to be most complete and is continuously maintained by it’s author.

Test application ran in total 150 tests per each format, 50 per each of three test classes. All tests classes have different size, so trend could be observed. Following listing shows definition for test class I refer to as Small in results section below. In terms of number of fields Medium is twice the size and Large thrice.

public class InstrumentPriceSmall
{
    public var numeric1 : int;
    public var numeric2 : int;
    public var text1 : String;
    public var text2 : String;
    public var date1 : ValueDate;
    public var date2 : ValueDate;
}

public class ValueDate
{
    public var year : int;
    public var month : int;
    public var day : int;
}

During each test an object was created, filled with data, serialized 10 000 times and then deserialized 10 000 times. Test application was compiled in release mode so that Protocol Buffers serialization and deserialization logic, which is completely coded in Action Script, could run with maximum efficiency. To minimize impact of garbage collector, System.gc() was called before running each test.

Results
In terms of over the wire size, PB is quite efficient for simple messages. However it’s advantage over AMF3 would quickly go away for objects that hold multiple references to the same value. This is fortunately not relevant for our use case, but could be for other scenarios.

Size
AMF 3 Protocol Buffers
Class: Small Medium Large Small Medium Large
Size:
(bytes)
198 290 371 31 80 123

On average deserialization times for PB are very close to AMF3. It also seems to scale quite well (although curve is steeper than AMF3). One thing I found surprising was that PB times were more stable (standard deviation smaller by factor of 2 to 6, depending on test object size). This is very desired feature because sometimes (like in our case) not only average timing is important, but also how far off best and worse case timings are.

Deserialization
AMF 3 Protocol Buffers
Class: Small Medium Large Small Medium Large
Average:
(ms)
444.92 955.06 1348.38 332.46 977.12 1457.12
Std dev:
(ms)
18.19 20.88 16.32 3.10 6.77 7.42

Serialization times look much less promising. Protocol Buffers are much slower, even 5 to 10 times depending on test object size.

Serialization
AMF 3 Protocol Buffers
Class: Small Medium Large Small Medium Large
Average:
(ms)
165.30 235.08 320.66 835.82 2385.44 3619.78
Std dev:
(ms)
22.22 17.15 16.57 17.89 63.02 81.80

I will keep my summary short. Small message size could mean short over the wire transfer time, deserialization times comparable with AMF3 could mean low processing overhead on the client. No need to translate each message to AMF3 could reduce server load. In our scenario Protocol Buffers could work.

EDIT: There’s one more thing that might be important. AMF3 serialization and deserialization is done by Flash Player on a separate thread while Protocol Buffers messages are processed by AVM on GUI thread. This means that large volumes of data might influence framerate and disrupt animations.