Whether it’s the app’s stability, a smoother user experience or better app performance (e.g. reduced loading times), we’re always looking for ways to improve our app at Whatnot. As an Android engineer on the Discovery team, I was tasked in late ’23 to review our feed-driven surfaces. This post focuses on auditing the network calls the app was making as well as looking at the bandwidth usage of the Android app. For a livestream shopping app where latency and performance sit at the core of our experience, this investment led to better app and scroll performance for our Android Whatnot users, which is especially important as Whatnot continues to grow internationally where there are many Android users.
Starting point
Android Studio comes with an inbuilt Network traffic inspector that allows you to review your network calls. For more granular information, we prefer to use a tool like Proxyman to capture and inspect network traffic with ease. Especially being able to filter out specific endpoints or inspect network requests originating from the third-party SDKs in your app.
Besides slimming down your network requests (e.g. request fewer fields in your API calls), the other thing that we can do is to ensure that we only request images that are appropriately sized for mobile devices.
The JPG format has been the de facto standard for the web for a while now and it is no surprise that we have been using this on Android as well. As new image formats were developed (e.g. WebP, HEIF, AVIF) we had options for alternative more efficient image formats. On the web, WebP is slowly gaining popularity and is supported by all major modern browsers. The main benefit of migrating from JPG to WebP is the reduction of the image file size while maintaining a reasonable picture quality.
With the help of Proxyman, we did a simple test. For the same show, we requested the show image but in different file formats: JPG, PNG and WebP.
On average, we measured a reduction of ~34% when using WebP which seems to be on par with the WebP Compression Study.
A smaller file size, means faster image loading times, a more responsive app and hopefully happier users.
The importance of internal dogfooding
Regularly using our own products internally helps us catch bugs early. Initially, when we released the changes internally, things seemed to be ok until one of our community moderators noticed that something was wrong with some of his listings. They were showing up correctly on the profile shop on his iPhone, but not on his Android device. On the Android app, the WebP images were being displayed sideways. Luckily this problem was found early and we were able to revert the change so it wouldn’t impact our users.
Finding the culprit
With all the benefits of WebP, it would be a shame not to be able to use it on Android.
Not knowing what could be the reason for the wrong image orientation, the first step was to see if we could find some hints by checking the image data.
Similar to the JPG image format, WebP images can also contain an Exchangeable Image File Format (EXIF) section that stores information about the image such as creation date, capturing device, image size, image rotation, color profiles, GPS locations, etc.
As the images were displayed correctly on iOS it got us wondering if our image server was malforming the images during the conversion process to WebP or if the image library we are using on Android, Coil, had a bug in its implementation.
Inspecting the EXIF data
There are plenty of tools we can use to review the EXIF data of an image, two popular ones are Exiv2 and ExifTool.
The first thing we did was to create some test material to replicate the problem our community moderator was seeing. We used an iPhone to take photos in 4 different orientations. By default these images were captured as JPGs and were uploaded to our image server. Next, we requested a WebP conversion via our image service and saved them locally.
This allowed us to have 4 original images and 4 converted images that we could inspect. Using the ExifTool on both sets of files resulted in this:
File Name : case1-portait.jpg
Orientation : Rotate 90 CW
File Name : case1-portait.webp
Orientation : Rotate 90 CW
File Name : case2-landscape-turned-left.jpg
Orientation : Horizontal (normal)
File Name : case2-landscape-turned-left.webp
Orientation : Horizontal (normal)
File Name : case3-upside-down-portrait.jpg
Orientation : Rotate 270 CW
File Name : case3-upside-down-portrait.webp
Orientation : Rotate 270 CW
File Name : case4-landscape-turned-right.jpg
Orientation : Rotate 180
File Name : case4-landscape-turned-right.webp
Orientation : Rotate 180
Looking at the results meant that the original image (JPG) contained proper orientation information and that the converted image (WebP) also contained orientation information.
As a sanity check, we tried to open up the WebP files from the image service on Safari, and MacOS’s Finder which both displayed the image correctly. Chrome, Firefox and our app (using Coil) all seemed to fail which was very suspicious.
Diving deeper into Coil
As previously mentioned, we use a 3rd party image library called Coil to handle all our image loading in the app. So the next step would be to review if there is an issue with the library itself. Fortunately, Coil is open source so we have easy access to the code. Placing many breakpoints in the code and stepping through the image loader functions revealed that the orientation reported by Coil was off for some files.
While the correct orientation was reported for JPG files, it failed to show any orientation information for our WebP files. Very odd, could the problem be somewhere else?
The AndroidX Exifinterface
While Coil helps you load images into your Composables/Views, it relies on the AndroidX Exifinterface to read the EXIF data. Like Android, this is open source so that anyone can download and review (or fix) the source code.
To make debugging easier, we created a test application that would directly interface with the ExifInterface.
One thing we noticed while looking at the source code of the ExifInterface was that Google did cover a test case with a WebP image containing EXIF data. Why did that one work?
Let’s get low level
At this point, we know that something is up between the Google WebP file and our own. Before we start inspecting the files, it’s a good idea to look up the official specifications of the image data container format. In the case of WebP, Google has published them here.
Since we’re dealing with binary files, we won’t be able to use a text editor. Instead, we need something more advanced such as a hex editor.
A hex editor allows you to inspect binaries in hexadecimal as well as text. On macOS, we can use a free tool called Hex Fiend to open the two files.
Hmm, wait a moment, the EXIF structure looks a bit different between the two files. Let’s take a look at what the EXIF structure is supposed to look like in the specs.
There should be an EXIF chunk header and a payload with the metadata. In both cases, we can see that the “EXIF” marker (in plain text) is present.
The Exif metadata section should start with a so-called TIFF header. This header contains information about the byte order (Endianness) of the payload. The first two bytes are 0x49 0x49 (II for Intel) if the byte order is little-endian or 0x4D 0x4D (MM for Motorola) for big-endian. The following two bytes are magic bytes 0x00 0x2A (42 😉).
Going back to our files we can see that the file generated by our image service contains an additional section that looks like this: 0x45 0x78 0x69 0x66 0x00 0x00 (“Exif\0\0”) before the TIFF header but why?
The EXIF APP1 Section
Checking on Google, it appears that our WebP files include an additional APP1 marker — uhm, isn’t this supposed to be standardized?
Reviewing the EXIF standards by the Camera & Imaging Products Association (CIPA), it becomes clear that the APP1 is a JPG-specific marker that precedes the EXIF data (and is not considered part of the EXIF data payload).
It’s important to note that the official WebP specifications do not mention the APP1 marker at all, meaning WebP files should have never contained this APP1 marker in the first place.
It’s very possible that the bug originates from image processing libraries (and that affects any frameworks relying on them), for example:
This confirms that our image service is copying EXIF data directly from the original JPG file into the newly created WebP files.
Images in the wild
So here we are, there are images out there because some image libraries didn’t completely adhere to the specifications and malformed WebP files that are now floating around the internet.
Our best guess is that Safari, MacOS’s Finder, ExifTool and Exiv2 all use a relaxed EXIF metadata reader while Firefox, Chrome and the (AndroidX) ExifInterface are more strict.
Fixing the problem
Going back to the ExifInterface source code we were able to track down the problem. It was failing to read the TIFF header and that resulted in not reading any of the EXIF data.
Knowing this, we created a patch that would scan the first 6 bytes of the payload. If the 6 bytes match the JPG EXIF APP1 marker, we adjust the offset and skip the number of bytes and the length of the marker before processing the rest of the data.
To validate the changes we updated the test application
Success!
We provided the patch to Google which was merged into the codebase at https://android-review.googlesource.com/c/platform/frameworks/support/+/2889627
What’s next
It’s rare to stumble upon a bug this low level, but it’s always helpful to know your debugging tool options. While the debugger in Android Studio is very advanced and should cover most of your day-to-day app development needs, external tools like Proxyman and Hex Editors can make your life much easier when it’s time to debug a problem.
The complicated part here is that the bug is likely caused by an external image generation framework (seemingly harmless) but that didn’t surface until a more strict EXIF library was trying to read the data.
Now that the AndroidX ExifInterface is patched, libraries like Coil will also be able to benefit from this change.
Since we have full control of our image service, we also decided to circumvent the issue on the backend by reading EXIF data from the JPG file and applying the rotation on the requested image. The benefit of this approach is that it will now work out of the box on Android, Chrome and Firefox plus it allows us to strip away all EXIF information for privacy reasons.
This article was previous posted on the Whatnot Engineering blog
