Today's blog post is from our engineers at Jana—the talented team behind our mCent app—all about the importance of testing new features of our mCent app.
Jana’s mission is to make the Internet free for the next billion. To help us reach more people, we recently launched gifting in mCent. mCent is Jana’s Android app that allows users to discover and try new apps without incurring any data charges.
What is Gifting?
Gifting allows members to share data earned in mCent between friends and family. Our hypothesis was that by making Internet more fluid and liquid, members would use their data to introduce others to mCent and also become more engaged in the app in the process.
We chose to build this feature for the following reasons:
Many members had unused free data sitting in their accounts and we wanted our members to put that airtime to work.
Making data more liquid meant that each dollar earned would be more valuable for our members.
As a company, Jana believes in testing and validating every feature that we ship. We wanted to measure how effective Gifting was at driving member growth, but we faced a testing challenge inherent in Gifting’s mechanics. The more users who had access to it, the better the feature was. A simple A/B test wouldn’t have measured the exponential effects of gifting when all members didn’t have access. For example, if you had the ability to send a gift to someone, but none of your friends could receive it, then you probably wouldn’t use the feature.
To deal with testing network effect type features, we could have used tools like snowball sampling. That would have taken us too long to build, so we prioritized shipping gifting to all of our users, as opposed to building an invite mechanism.
We started our test at 1% exposure, meaning 1% of our users had access to gifting. Gradually, we ramped exposure to 25% over two weeks. The main objectives for our test were the following:
Ensure that the feature was working, and fix all major bugs
Prove that gifting didn’t have adverse effects on our core metrics (retention, growth, ARPU)
Determine if Gifting was being used fraudulently.
We were explicitly not testing for any improvement to our metrics in the A/B test since Gifting would only reach it’s potential when many densely connected people had access to it. Measurement of Gifting’s impact would only come after we had fully launched the feature, which we did a few weeks later.
One of the best things about being a member of the Jana tech team is that everyday we get to see the immediate impacts of the features we’re launching. Gifting ended up being a huge hit among our members. mCent’s Facebook page was bombarded with members talking about the feature and our Google Play reviews were full of members writing about it.
Not surprisingly, we saw an increase in engagement in the days after we launched Gifting. To analyze the data, we looked at users that received gifts, users that sent gifts, and users that did neither on their first day. We learned that:
16% of users sent or received gifts on their first day after joining mCent
Users that used Gifting were more than 2X as likely to refer their friends to mCent
Users that sent or received gifts were over 10% more profitable for us, and generated more revenue than those that didn’t.
From a macro level, we saw record growth and DAU numbers after gifting was launched and never looked back.
Get your app discovered
By creating new features like Gifting, we’re taking another step toward our mission of making the Internet free for the next billion.