Choosing the best data representation format for Android

I recently did a PoC to evaluate which data representation format suits for small or mid sized data. Would like to share my observations of the same.

If you don't know about MessagePack and FlatBuffer. Visit the official sites to know more about them.

MessagePack

FlatBuffer

Comparison of sizes of different data representation formats (for small JSON files)

	Json	Message pack	Flat buffer
Without GZip (bytes)	958	722	1,028
With GZip (bytes)	309	324	493

Observation:

• Plain MessagePack is better than JSON, but upon GZipping JSON is better (in terms of size)
• Flat buffer takes more size as compared to JSON / MessagePack, but gives us the advantage of accessing the data without deserializing

Conclusion:

It is better to stick to JSON + gzip way of data representation and compression. It is more efficient than message pack and Flat buffers in terms of size. Also deserializing takes more time in Message Pack

Flat buffer eliminate the need of deserialization of data at the client end but we can observe any effect only if the data is being loaded from the device cache. In case of loading from the network the incremental benefit of around 15ms (average time to deserialize small data) is negligible compared to the efforts needed for replacement.

Supercharging the Android WebView

Techniques to selectively cache critical assets that drastically reduce the load time for common web views in your app. Learn how to achieve instant load times - making transitions between native and web screens feel seamless.

Background

Hybrid Android applications are mainstream these days. But Android WebViews have a limited access to any of the native resources. I would like to share some techniques which we implemented that improved the load time of our WebViews.

Performance of WebViews is very critical, especially when they are a part of the transactional flow (like checkout, payments). But there are limited set of capabilities that the WebViews come with. One of the main things that bothered us was the caching limitations and inconsistency. Cache control headers were not respected across multiple app launches. Also, loading a web asset (CSS, JS etc) even the first launch (assuming caching works well) of the WebView is also a costly operation in case of 2G networks. There are still a sizeable number of users in 2G and slow networks, especially in countries like India. If we have a mechanism to load these instantaneously without making a network call to fetch them, it will make the WebView load much faster.

Many times, in case of checkout related WebViews, there are multiple images which get displayed. Many WebViews use the same fonts across native and WebViews to have a consistent look. We can actually build capability in the WebView to load the images from device cache and load fonts from the already bundled fonts instead of loading these from the network. I’ll be explaining how these simple changes can enable you to do all these.

Android Implementation:

WebViewClient provided by Android has this magical method called shouldInterceptRequest, which can be overridden. What it basically allows you to do is intercept any request (to fetch HTML, CSS, JS, images, fonts etc) going from the WebView and gives you the ability to take some actions before network calls are made. Read more about it in Android Developers site.

shouldInterceptRequest returns a WebResourceResponse which is created by data loaded from the network. All you need to explicitly create appropriate WebResourceResponse is MIME type, encoding format and input stream(can be created from assets or files) of the resource. So, you can create a WebResourceResponse for CSS/JS request being made if you have it stored somewhere in the app (may be in assets or file system) and return it instead of making a network call to get it!

Sample code to create a WebResourceResponse:
Once we realised that we can to this, we went ahead and bundled all CSS, JS and stripes (yes, it did increase our APK size by around 200 kb) which we were using in our WebViews, also created a mapping file which maps urls to local assets. You can create a gradle task to automate this process of bundling and creating a mapping file. All the requests going from WebView are intercepted and we check if we have any local asset for the url intercepted. If so, we create a WebResourceResponse using it and return instead of making a network call. Along with this we had a common font which is being used in both native and WebViews, we started loading these fonts also from bundled fonts instead of making a network call to get these.

So far good…But the problem is, once we push the application to play store we can’t update the bundled assets whenever CSS, JS or sprites used in WebViews are updated/changed. All the bundled assets are basically useless now. To make sure that we have all the assets locally before a user reaches the WebView (in most of the hybrid app first few screens are usually native, at least in our case) we wrote an IntentService which downloads all the new web resources which are not present in file system or bundled assets and stores in the file system (You need an endpoint which lists all the web resources currently used in your WebViews, see below for sample), also delete the files which are no more required to clear the foot prints. Now our shouldInterceptRequest first checks if the resource is present in assets, if not checks in file system, if not makes a network call to fetch it…There we go…our own custom cache with very high hit rate!

A sample response of asset listing API:

Note that all the assets have a unique hash or version number appended at the end. This is very important to invalidate a file which is no longer used. In our intent service, we save files with the <resource name> + <hash> as the file name. Only required files are downloaded and unnecessary files are deleted. This way we can easily check if a local file corresponding to the url for which request is being made is present, we simply have to check if a file with such file name exist in our application’s files directory. Any other logic can be used to do this.

After we did this, the only network calls made in WebView were to fetch HTML of the page, some third party analytics JS files and images. All the in house JS files, CSS, fonts, stripes were loaded from local resources. This gave a massive improvement in load time of the WebViews. Now the majority of the time was spent in downloading the images.

We went ahead and added a check saying, if we have the image already in our device cache (we use fresco for loading and caching images) then load the images from the device cache instead of making a network call (We wrote a method to synchronously get Input Stream of an image from Fresco cache if it is available). So, finally what we load from network is HTML and third party JS files. This drastically reduced the load time of the WebView, especially in slower networks, as much as 80% improvements in some cases. See the comparison below

In case you need help in reading InputStream synchronously from Fresco pipeline.

Comparison:

Below is a comparison of WebView before and after we did these optimisations in a very slow network.

Before: Observe that fonts, sprites, CSS, JS, all taking considerable amount of time to load

After: Observe that fonts, sprites and images are taking less than 10ms to load!

The complete code of shouldInterceptRequest

Note that we are being defensive and call super.shouldInterceptRequest in case of any exception, which will make sure a network call is made in case of exception.

These small changes we did for our WebViews gave us a lot of benefits. We achieved a super fast loading of WebViews, lesses data usage, less hits to our CDN network to name a few of the benefits.

Hope this was useful and you will try this in your hybrid application!

My experience with using Fresco (Android image management library)

I recently worked on Replacing UIL with Fresco in an Android App. I would like to share my experience in doing so and some of the learnings that might help you if you plan to use Fresco.

If you don’t know what Fresco is yet, please look at the github repository https://github.com/facebook/fresco and official site http://frescolib.org/

If you are new to Fresco, I recommend you read article by Facebook https://code.facebook.com/posts/366199913563917/introducing-fresco-a-new-image-library-for-android/. It clearly explains the motive behind Fresco, its implementation, different memory regions in Android than an App might use and the way  Fresco works.

Assuming that you have read about how Fresco and how it works I will only explain about my observations and about the features of Fresco which is already available in the above links.

Fresco has three levels of caching

• Bitmap cache
• Encoded memory cache
• Disk cache

On Android 4.x and lower, the bitmap cache’s data lives in the ashmem heap, not in the Java heap. This means that images don’t force extra runs of the garbage collector, slowing down your app.

Android 5.0 has much improved memory management than earlier versions, so it is safer to leave the bitmap cache on the Java heap.

In Android 4.x devices you will observe a lot less OutOfMemoryErrors after switching to Fresco.

Hard part

Note that if you want to replace UIL or Picasso with Fresco you will have to edit all the layout files using ImageViews which load images from network. 
Also note Drawees do not support the wrap_content value for the layout_width and layout_height attributes. http://frescolib.org/docs/using-drawees-xml.html#wrap-content. Keep these limitations in mind when you want to replace the existing image download library.


Below are some of the useful tips and code that might be useful if you are planning to use Fresco in your next project. These are the difficulties faced and some possible solutions

Bitmap Downloading 

One of the limitations of Fresco is that image request can be attached to only DraweeView and its subclasses. Right now there is no way to attach image request to ImageView or custom views extending ImageView. 

One way to handle this is to download the bitmap and set this bitmap to ImageView / custom view.

But this approach is not very direct. One way is to do this is to have a method in a helper class to return a CloseableReference of type CloseableImage (imageReference in the code below) in a callback ( don't try to read the bitmap and return bitmap ). Reason why CloseableReference is returned not the actual bitmap is because Fresco recycles the bitmap if there are no reference to bitmap. The underlining bitmap can me read from CloseableReference from Activity / Fragment.

Below is one of the ways you can do it

private static void downloadBitmap(ImageRequest imageRequest, Context context, final IBitmapDownloader iBitmapDownloader) {
 DataSource<CloseableReference<CloseableImage>> dataSourceImage = Fresco.getImagePipeline().fetchDecodedImage(imageRequest, context);
        DataSubscriber<CloseableReference<CloseableImage>> dataSubscriberImage = new BaseDataSubscriber<CloseableReference<CloseableImage>>() {
            @Override
            public void onNewResultImpl(DataSource<CloseableReference<CloseableImage>> dataSource) {
                CloseableReference<CloseableImage> imageReference = dataSource.getResult();
                if (imageReference != null) {
                    iBitmapDownloader.onSuccess(imageReference);
                }
                else{
                    iBitmapDownloader.onFailure(null);
                }
            }

            @Override
            public void onFailureImpl(DataSource dataSource) {
                iBitmapDownloader.onFailure(dataSource);
            }
        };
        dataSourceImage.subscribe(dataSubscriberImage, CallerThreadExecutor.getInstance());
}

IBitmapDownloader is a simple interface with callbacks.

In the Activity or Fragment maintain a map of urls and corresponding CloseableReference

Map<String, CloseableReference<CloseableImage>> imageRefMap = new HashMap<String, CloseableReference<CloseableImage>>();

When you get the callback in the Activity add the reference to imageRefMap. We can get the bimap from the CloseableReference. Next time you can check if the CloseableReference is present for a URL in imageRefMap, if present you can get the bitmap, no need to send the request down the pipeline again. This is mostly useful in case of Bitmap being used in RecyclerView or ViewPager.  

CloseableReference<CloseableImage> imageRef = imageReference.clone();
imageRefMap.put(imageUrl,imageRef);
CloseableImage image = imageRef.get();
final Bitmap bitmap = ((CloseableBitmap) image).getUnderlyingBitmap();

Don't forget to close the references in onDestroy of the activity so that bitmaps can be garbage collected. 

Iterator<Map.Entry<String, CloseableReference<CloseableImage>>> entries = imageRefMap.entrySet().iterator();
        while (entries.hasNext()) {
            Map.Entry<String, CloseableReference<CloseableImage>> entry = entries.next();
            CloseableReference<CloseableImage> closeableRef = entry.getValue();
            if(closeableRef != null){
                closeableRef.close();
            }
        }

Adding to Cache Manually 

If you want to add any bitmap to cache manually, this code can be used

try {
    CacheKey cacheKey = new SimpleCacheKey(url);
    ByteArrayOutputStream stream = new ByteArrayOutputStream();
    bitmap.compress(Bitmap.CompressFormat.JPEG, 100, stream);
    final byte[] byteArray = stream.toByteArray();
    Fresco.getImagePipelineFactory().getMainDiskStorageCache().insert(cacheKey, new WriterCallback() {
        @Override
        public void write(OutputStream outputStream) throws IOException {
            outputStream.write(byteArray);
        }
    });
} catch (IOException cacheWriteException) {
   
}

Checking in Disk Cache Synchronously 

In Fresco Disk cache check is asynchronous (http://frescolib.org/docs/caching.html#checking-to-see-if-an-item-is-in-cache). This is because Disk cache check is a costly operation and takes a lot of clock cycles, so the thread gets blocked. But if you are checking in Disk cache in non UI thread you can use this code to check 

CacheKey cacheKey = DefaultCacheKeyFactory.getInstance().getEncodedCacheKey(ImageRequest.fromUri(imageUrl));
        StagingArea stagingArea = StagingArea.getInstance();
        EncodedImage result = stagingArea.get(cacheKey);
        if(result != null){
            result.close();
            return true;
        }
return ImagePipelineFactory.getInstance().getMainDiskStorageCache().hasKey(cacheKey) || ImagePipelineFactory.getInstance().getSmallImageDiskStorageCache().hasKey(cacheKey);

Fresco has a very active community using it and backed by Facebook. More enhancements can be expected by the actively contributing community in the coming days.

Hope this was use useful! 

Bunkbook Attendance manager - A new way to maintain your attendance status

I guess all of you have faced a serious problem of maintaining your attendance of various subjects in the college. You might have used methods like writing the number of classes attended in the last page of the book, maintain a note on your mobile phone etc. 

Being an engineering students ( who bunk  a lot of classes ), I have faced this problem as we have to have a certain minimum percentage of attendance in all the courses. So I thought of making an android app that helps students to maintain their attendance status of all the courses and built this android app called Bunkbook Attendance manager.

Here is a brief note on the app:

Bunk book helps students to maintain their attendance (rather bunk! ) status of all the subjects. Bunk book analysis your bunks and gives interesting insights about your attendance and bunk status.  It enables users to create any number of subjects and track and update attendance of them. It gives periodic notifications based on user preferences reminding you to update your attendance.

Some of the screen shots of the application:

Webslam.me! a new way to online social networking

I have built a new social networking website called Webslam (webslam.me). The unique feature of the website is that it enables users to create virtual slam books online.

It helps users to maintain slam books online. Users can create slam books online with the questions of their choice and ask their friends to write in their slam book which is quite different from other online slam books site.



It also helps users to share their thoughts, photos and videos with their friends. It has full support for social news feed system which brings the users the latest updated from the people whom he/she follows. It also has so many other options like messaging, social connections and much more options coming soon.

Users can use either their google or facebook account to sign up with the website.

Read the help for any further information.

Here are some screenshots of the website:

The technologies used to builds the website are:
Codeigniter 2.1.4 ( An MVC framework for PHP )
MySQL ( Most popular open source relational database system )
Twitter Bootstrap ( CSS tool )
JQuery
HTML 5

Facebook Style Hovercard using JQuery and CSS

Facebook has a feature of displaying the hover card when you hover over a link of a user, page or a group. I always wondered how this could be done. I tried creating my own hover card feature like facebook and came up with this.

I used CSS to give facebook like feel to the hover card.

#hoverbox {
    background-color: white;
    border: solid 1px #BCBCBC;
    box-shadow: 0 4px 16px rgba(0,0,0,0.3);
    -webkit-box-shadow: 0 4px 16px rgba(0,0,0,0.3);
    -moz-box-shadow: 0 4px 16px rgba(0,0,0,0.3);
    color: #666;
    font-size: 13px;
    width: 350px;
    height: 230px;
    overflow: hidden;
    position: absolute;
    word-wrap: break-word;
    z-index: 1202;
    display: none;
    margin-top: -10px;
}

#hoverbox img#cover {
    width: 100%;
    height: 62%;
    float: left;
}

#hoverbox img#profile {
    width: 90px;
    height: 90px;
    position:absolute;
    bottom:20%;
    left:3%;
    z-index: 1234;
    background-color: #FFFFFF;
    border: 3px solid #FFFFFF;
    border-radius:1px;
}

#hoverbox_name{
    font-size: 17px;
    position: absolute;
    top:50%;
    left: 35%;
    color:#FFFFFF;
}

#hoverbox_bottom{
    width:100%;
    height:18%;
    position:absolute;
    bottom:0px;
    background-color: #eee;
}

#fbstyle_button{
    -moz-border-bottom-colors: none;
    -moz-border-left-colors: none;
    -moz-border-right-colors: none;
    -moz-border-top-colors: none;
    background-color: #EEEEEE;
    background-image: linear-gradient(#F5F6F6, #E4E4E3);
    border-color: #999999 #999999 #888888;
    border-image: none;
    border-style: solid;
    border-width: 1px;
    box-shadow: 0 1px 0 rgba(0, 0, 0, 0.1), 0 1px 0 #FFFFFF inset;
    color: #333333;
    cursor: pointer;
    display: inline-block;
    font: bold 11px 'lucida grande',tahoma,verdana,arial,sans-serif;
    margin: 0;
    overflow: visible;
    padding: 0.3em 0.6em 0.375em;
    position: relative;
    text-align: center;
    text-decoration: none;
    white-space: nowrap;  
}
a{
    font-size: 18px;
    margin-top:10px;
    margin-bottom:10px;
    text-decoration: none;
    color:#444;
    padding: 5px;
}  

The javsacript code is as below. The mouseenter event is caught when the user moves the mouse over the link or image and hover card is displayed at the position of the cursor. You can fetch the data from the database by making an ajax call if necessary.

            $(function(){
                $('#hoverbox').mouseleave(function(){
                    $('#hoverbox').hide();
                });
                
                $(".trigger").mouseenter(function(e){                  
                    //you can get picture addresses and the user_name from the database by making an ajax at this point.
                    //You can either store used id or the full link in an attribute in the hyperlink or photo.
                    var  cover_pic = "abc.jpg";
                    var  profile_pic = "xyz.jpg";
                    var  user_name = "Chethan";
                    var display_content =  '

' + user_name + '


';
                    
                    var pos = {
                        top: (e.clientY ) + 'px',
                        left: (e.clientX ) + 'px'
                    };
                    $('#hoverbox').css(pos).html(display_content).show(400);
                }).mouseleave(function(e) {
             if ($(e.relatedTarget).attr('id') != "hoverbox") {
                $('#hoverbox').hide();
            };
            });
            });            
       

Try Demo Download Script

Simple Database design to support multiple blogs in MySQL

MySQL is open source, very powerful and the most used database. In this post I explain how one can easily create a database in MySQL that can support multiple blog.

I have kept the design simple keeping only the elements absolutely necessary in order to keep the design simple to understand for those who are new to designing databases. The key idea here is to explain about the foreign keys in MySQL.

The central idea here is to maintain a proper relations among the tables. Relationship between the parent and child table is achieved by using foreign key constrain. It helps you in maintaining the structural integrity of the database, deletion of unnecessary data, joining of tables and gives you many more advantages.

The figure below explains the relationship among the tables.

You can download the script below.

Download the script