Adaptively plan to conquer dauntingly ambitious goals.
Adaptively plan to conquer dauntingly ambitious goals.
When replacing legacy apps actively used by millions of job seekers, where do you begin? Here's how we used adaptive planning to move even when the path was uncertain.
During my time at SEEK Asia, we had a lot of interesting challenges but one I will share today was specific to what we needed to achieve to improve the mobile app experiences for our JobsDB and Jobstreet brands.
JobsDB and Jobstreet are job boards which had competed against each other in various South East Asian markets for two decades. These two brands came together under one roof when they were acquired by SEEK and these two companies formed the foundation of SEEK Asia, SEEK’s South East Asian job board business. Almost a year later after the acquisition of Jobstreet, the latter of the two acquisitions, I joined SEEK Asia as CTO.
JobsDB and Jobstreet were expending a lot of financial resources battling each other and were at risk of being disrupted by new market entrants. This included job listing offerings from behemoths such as Google and Facebook, global job boards such as Indeed and LinkedIn and also some disruptive local market players.
The opportunity to join together these two dominant brands to face this competition could help them switch from using those war chests against each other to, instead, pool those financial resources together and combat these new threats.
Mobile apps battlefront
Drilling into the mobile app space which was a key battlefront where there was a big risk of losing market share overall. Both brands had apps which for both businesses had started life built by outsourced development shops and then later in-housed and iterated from there.
The foundation of these apps as a result was poor. Most of the feature-set was driven by years of whims with little to no measurement of what was actually resonating or being used. These apps rated very poorly on the app stores.
Adding to the complication there were separate codebases for iOS and Android as well so that meant 4 separate small teams each devoted to delivering a single brand and OS combination. The install base was huge — millions of users across South East Asia for each of the apps we were managing but the ratings and for all of them the user experience was very poor.
To migrate millions of users from one user experience to another without significant backwards steps, in terms of our position in the market, is a daunting task. The poor experience of the apps presented a risk of being disrupted by newer more focused apps. These new apps offered a better user experience which had already been proven in other geographies.
The multiple brands SEEK now held under one roof multipled by Operating System combinations, iOS and Android, also diffused our efforts, reducing the velocity of any potential improvement impact. There was currently a code base for each brand, on each operating system, totalling four to be managed.
We measured ourselves as a business in terms of our market share of people searching for jobs with our services and applying for jobs using our services. Assessing the apps and the codebases that supported them we knew the effort to bring ourselves to a position where we could compete against global players would be longer than building from scratch. And we hypothesised that a lot of the functionality our bloated apps provided was not actually serving the needs of job seekers well.
As part of making the case to embark on a significant change, we identified what we believed success would look like. We needed to maintain or improve our market share of search and job applications. We identified key elements of achieving this was to maintain or grow our significant install base of users.
We experimented with a variety of mobile application development frameworks and eventually settled on, a cross-platform development approach using React Native. Our testing showed we could deliver a user experience which was competitive with other new players in the market with only about a 20% overhead for each additional Operating System to support.
We also elected to build apps we could theme easily, build once and support the different brands. To a similar end, we also adopted an approach to support what initially would be different backends (systems which were being unified on a different timeline) but would allow for consolidation in the future.
What good mobile apps needed to achieve
That in turn identified further goals that were important to achieving our high-level goals:
We needed to successfully migrate users from the old apps to the new apps. This informed our MVP and the alpha and beta iterations that followed as we slowly released it to larger and larger groups of users and validated the apps were achieving our other goals.
Provide a great user experience that was efficient for our users (i.e. we needed to be careful that the volume of screens viewed or actions taken could be good usage or could be indicative of an inefficient experience and needed to be handled with more sophistication) and also lead to productive reuse.
Having high ratings for the apps on the app stores would be important to improve the prominence of the apps in those stores which would translate to more downloads.
Reduce/eliminate any crashes (challenging for mobile apps when there’s a rapidly evolving development ecosystem and a longtail of OS versions and devices).
And a variety of other qualitative goals that would influence the experience job seekers had using the app.
What may be evident to you from this sample above of what was a longer set of goals we had identified as critical to being successful in this challenging endeavour is that these goals would be relevant at different resolutions (big picture versus detail), timescales (current quarter versus longer term) and concerns (different qualitative aspects of the experience).
Not all would be relevant on day one as we started development but knowing each of these would be critical in the future did inform some earlier choices we made. For instance, we instrumented our apps for crash reporting even though our tolerance for crashes in the early days was higher than when we started releasing and over the course of the development we would raise the standard we strived for. This helped keep this dimension visible and not get too unhealthy.
Understanding how goals relate to each other and why that’s helpful
What we could see is there are clear relationships between these goals. For instance, reducing the crash rate correlated with the ratings the apps got. However, another lever for influencing the volume of ratings was how actively your app asked for the user to make a rating and the timing of when to ask etc.
Next, we visually mapped these relationships and analysed these maps to decide where effort could make the biggest impact on our overall goals. One interesting, less apparent area of investment which once identified seemed quite obvious — fast response to app reviews — especially taking action and communicating actions taken in response to constructive reviews could bolster rating scores and indirectly grow downloads.
To provide these fast responses to reviews we ensured multiple members of the team spent significant time reading and responding to reviews and using that feedback to inform the iterative development goals so that issues were always short-lived. Users often updated their lower score reviews with 5-star reviews when their issues were addressed.
For example, one chain of these relationships from most granular and near term to more macro and longer-term might look like this:
Users' concerns are addressed quickly (Respond to reviews within 24 hours, solve issues within a week)
⬇ contributes to ⬇
Prominence in app stores (Ratings are above 4.5*)
⬇ contributes to ⬇
Growing install base (Downloads above 10k per week)
⬇ contributes to ⬇
Most job searches (>45% of all job searches conducted)
⬇ contributes to ⬇
Most job applications (>30% share of job applications)
This is only one example of the many important chains of relationships which were important parts of us achieving our long-term goals. We maintained a visual map of these relationships which we updated when we learned things. We used our understanding of these relationships to set priorities as we went along.
We shared both the big picture map of relationships as well as specific causal chains and curated subsets of the big picture to help communicate WHAT was important and WHY it was important to the various people involved — from developers, designers, product managers, operational teams, marketing teams, leadership and many more who were involved in one way or another.
To be honest the big picture maps resonated only with some people but not others. I think the complexity of the diagram was a heavy cognitive load, especially for anyone who wasn’t directly involved in creating the map. The causal chains and curated subsets we made of the most relevant sections carved off from the big diagram proved more successful in connecting people with what we were trying to do and more importantly how they might be able to contribute.
What was the result?
In under two years, we moved millions (approaching 10 million) of job seekers on to the new experiences across thousands of devices, for multiple brands, for users in 7 countries meeting our search and job application goals, with 99.99% crash-free sessions, 4.6+ star ratings, performant screen rendering and a feature-set we knew was being used successfully, fit for purpose.
The example of this post is an example of just one of the sets of relationships between the goals we used. Visualisation of these relationships helped members of the team and other collaborators get on the same page about what we were trying to achieve at each step of the journey and why we believed it would help us progress towards our goals.
Comparing each of these chains allowed us to debate what the next most important investment needed to be based on our current situation and where we were heading. We also would update our diagrams when our hypotheses were invalidated and as such, they also could represent our best understanding at a point in time.
For diagramming goals and relationships we used Miro. The first drafts were just brainstorms of what concerns we needed to be aware of and using connectors to draw lines linking between related concerns. More macro concerns gravitated to the top of the page and objectives which might influence achieving the more macro goals were lower on the page.
Later drafts were closer to Result Maps as described by Stacey Barr’s PuMP framework. We could then use Horizon Planning to identify a sequence of goals and approximate time windows where it might be important to achieve these outcomes if we wanted to achieve our lofty long-term goals — something I will cover in a future post.
Have you been involved in achieving a very ambitious goal and used any form of dynamic or adaptive planning which helped you attain a positive result? What practices and artefacts did you use? What did you learn? Share your experiences in the comments.