Choosing the best data representation format for Android

I recently did a PoC to evaluate which data representation format suits for small or mid sized data. Would like to share my observations of the same.

If you don't know about MessagePack and FlatBuffer. Visit the official sites to know more about them.

MessagePack

FlatBuffer

Comparison of sizes of different data representation formats (for small JSON files)

	Json	Message pack	Flat buffer
Without GZip (bytes)	958	722	1,028
With GZip (bytes)	309	324	493

Observation:

• Plain MessagePack is better than JSON, but upon GZipping JSON is better (in terms of size)
• Flat buffer takes more size as compared to JSON / MessagePack, but gives us the advantage of accessing the data without deserializing

Conclusion:

It is better to stick to JSON + gzip way of data representation and compression. It is more efficient than message pack and Flat buffers in terms of size. Also deserializing takes more time in Message Pack

Flat buffer eliminate the need of deserialization of data at the client end but we can observe any effect only if the data is being loaded from the device cache. In case of loading from the network the incremental benefit of around 15ms (average time to deserialize small data) is negligible compared to the efforts needed for replacement.

Supercharging the Android WebView

Techniques to selectively cache critical assets that drastically reduce the load time for common web views in your app. Learn how to achieve instant load times - making transitions between native and web screens feel seamless.

Background

Hybrid Android applications are mainstream these days. But Android WebViews have a limited access to any of the native resources. I would like to share some techniques which we implemented that improved the load time of our WebViews.

Performance of WebViews is very critical, especially when they are a part of the transactional flow (like checkout, payments). But there are limited set of capabilities that the WebViews come with. One of the main things that bothered us was the caching limitations and inconsistency. Cache control headers were not respected across multiple app launches. Also, loading a web asset (CSS, JS etc) even the first launch (assuming caching works well) of the WebView is also a costly operation in case of 2G networks. There are still a sizeable number of users in 2G and slow networks, especially in countries like India. If we have a mechanism to load these instantaneously without making a network call to fetch them, it will make the WebView load much faster.

Many times, in case of checkout related WebViews, there are multiple images which get displayed. Many WebViews use the same fonts across native and WebViews to have a consistent look. We can actually build capability in the WebView to load the images from device cache and load fonts from the already bundled fonts instead of loading these from the network. I’ll be explaining how these simple changes can enable you to do all these.

Android Implementation:

WebViewClient provided by Android has this magical method called shouldInterceptRequest, which can be overridden. What it basically allows you to do is intercept any request (to fetch HTML, CSS, JS, images, fonts etc) going from the WebView and gives you the ability to take some actions before network calls are made. Read more about it in Android Developers site.

shouldInterceptRequest returns a WebResourceResponse which is created by data loaded from the network. All you need to explicitly create appropriate WebResourceResponse is MIME type, encoding format and input stream(can be created from assets or files) of the resource. So, you can create a WebResourceResponse for CSS/JS request being made if you have it stored somewhere in the app (may be in assets or file system) and return it instead of making a network call to get it!

Sample code to create a WebResourceResponse:
Once we realised that we can to this, we went ahead and bundled all CSS, JS and stripes (yes, it did increase our APK size by around 200 kb) which we were using in our WebViews, also created a mapping file which maps urls to local assets. You can create a gradle task to automate this process of bundling and creating a mapping file. All the requests going from WebView are intercepted and we check if we have any local asset for the url intercepted. If so, we create a WebResourceResponse using it and return instead of making a network call. Along with this we had a common font which is being used in both native and WebViews, we started loading these fonts also from bundled fonts instead of making a network call to get these.

So far good…But the problem is, once we push the application to play store we can’t update the bundled assets whenever CSS, JS or sprites used in WebViews are updated/changed. All the bundled assets are basically useless now. To make sure that we have all the assets locally before a user reaches the WebView (in most of the hybrid app first few screens are usually native, at least in our case) we wrote an IntentService which downloads all the new web resources which are not present in file system or bundled assets and stores in the file system (You need an endpoint which lists all the web resources currently used in your WebViews, see below for sample), also delete the files which are no more required to clear the foot prints. Now our shouldInterceptRequest first checks if the resource is present in assets, if not checks in file system, if not makes a network call to fetch it…There we go…our own custom cache with very high hit rate!

A sample response of asset listing API:

Note that all the assets have a unique hash or version number appended at the end. This is very important to invalidate a file which is no longer used. In our intent service, we save files with the <resource name> + <hash> as the file name. Only required files are downloaded and unnecessary files are deleted. This way we can easily check if a local file corresponding to the url for which request is being made is present, we simply have to check if a file with such file name exist in our application’s files directory. Any other logic can be used to do this.

After we did this, the only network calls made in WebView were to fetch HTML of the page, some third party analytics JS files and images. All the in house JS files, CSS, fonts, stripes were loaded from local resources. This gave a massive improvement in load time of the WebViews. Now the majority of the time was spent in downloading the images.

We went ahead and added a check saying, if we have the image already in our device cache (we use fresco for loading and caching images) then load the images from the device cache instead of making a network call (We wrote a method to synchronously get Input Stream of an image from Fresco cache if it is available). So, finally what we load from network is HTML and third party JS files. This drastically reduced the load time of the WebView, especially in slower networks, as much as 80% improvements in some cases. See the comparison below

In case you need help in reading InputStream synchronously from Fresco pipeline.

Comparison:

Below is a comparison of WebView before and after we did these optimisations in a very slow network.

Before: Observe that fonts, sprites, CSS, JS, all taking considerable amount of time to load

After: Observe that fonts, sprites and images are taking less than 10ms to load!

The complete code of shouldInterceptRequest

Note that we are being defensive and call super.shouldInterceptRequest in case of any exception, which will make sure a network call is made in case of exception.

These small changes we did for our WebViews gave us a lot of benefits. We achieved a super fast loading of WebViews, lesses data usage, less hits to our CDN network to name a few of the benefits.

Hope this was useful and you will try this in your hybrid application!