The Limits to Improvement
4 June, 2021
In my previous job, I stumbled across an interesting problem. When conducting user research, I noticed that a certain section of our users couldn’t book a direct flight. At that time, I was leading the air products team responsible for helping users find the best flight deals. Our user research showcased that a significant number of users had to make multiple bookings to reach the desired destination. They had to book two flights separately using two weblinks and going through their user journeys one by one, consecutively.
It was taking a lot of time. Our users were unable to avail offers due to insufficient transaction value with one flight while struggling to find the relevant information and cancellation policies. Spending all that time exploring different possibilities and doing their own research took the fun out of their travel and made it quite cumbersome. Our vision as an organization was to make travel fun, and we wanted to come up with a solution that would make it so.
We looked into our flight data for the last 30 days to learn how many users who booked connecting flights had to wait at the airport for more than three to four hours. We found out that 10 to 12 percent of our users were experiencing that, which indicated a serious problem. To put it more concretely, if 20 000 tickets were booked on our platform, some 2 000 people were going through this.
To better understand the problem, we had to take a look at different things: flight schedules, prices, shortest travel time to an optimal price, etc. We did some research, collected some rudimentary data, and came up with a crude equation that stated that every 15 minutes of waiting equals 15 Indian rupees. We built an algorithm that could tell us the money value of time. For example, if we could help our users save time, how much money they would be willing to attribute to that, or how much time the users can forgo for a fair price. Obviously, some price-conscious users would be willing to trade their time for money.
Also, a portion of our users struggled to discover multi-airline flights. So to address all those problems, we built an API that would show users direct flights, but also other opportunities with connecting flights showcasing how much money they could save in return for waiting longer in transit.
However, we encountered a number of challenges we had a hard time dealing with. For example, terminal changes were frequent and abrupt. Also, terms and conditions were different for different airlines -- some would allow 20k baggage and others 10k, some would give 100 percent refund for cancelations, while others would refund zero. On top of that, we had some technical difficulties related to latency. When a user would try to fetch all the flights from the different airlines, all those permutations and combinations were slowing the API down.
When we finally launched the product in the pilot phase, a number of complaints were filed by users who were unaware of the challenges described above. Some of the problems we managed to remedy. Every time a user would try to book any multi-airline itinerary, a pop-up would appear with information of baggage and cancelation policies, and only after a user would check it off would they be able to move forward.
Our product was initially designed to be used in India, but we soon expanded it to international flights for Indian users. We introduced visa terms and conditions to address the main pain point of Indian travelers. By expanding to international flights, we managed to make a conversion jump of 6 to 7 percent in our flight consumer journey.
- We could have done user testing much better. If we had done it more extensively, many of the user complaints received in the pilot phase could have been circumvented.
- Over time, the product didn’t gain much traction. At best, ten percent of our audience was using it, which was in stark contrast to the forecasted 80 to 90 percent. Maybe we failed at marketing or were unable to explain our value proposition clearly. We did a concept validation and user testing that was promising, but things simply didn’t work out in the end. Though it was an average product with average chances for success, it didn’t work as planned.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Jean-Benoit Malzac, Lead Product Manager at Ava, employs a system of categorization to prioritize his team’s efforts.
Lead Product Manager at Ava
Xun Tang, Engineering Manager at Twitter, has written the playbook on what it means to be a successful member of her team.
Engineering Manager at Twitter Inc.
Jason Mawdsley, Director System Software at NVIDIA, explains how to determine the maturity of AI features and why validation and verification are the most important part of the process.
Director, System Software at nVidia
Heiko Reintsch, Head of Product at GetYourGuide, begs the question: should his company be focusing on improving what it produces currently, or should it be looking to expand its operations?
Head of Product at GetYourGuide
Apurva Mudgal, Director of Product and Partnerships at Haptik, explains what and how to measure in early-stage products, emphasizing how selective and/or biased interpretation of user data can lead to inaccurate conclusions.
Head of Consumer Product at Simpl
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.