Managing Expectations on a Highly Visible Project
Senior Software Engineer at Netflix
I started a brand new project, which was much in demand even before its official start. The problem we were trying to solve was exceedingly important to the business, and many teams were interested in using our project. The craze kicked off even before we began articulating the idea and coming up with the design.
We tried to improve the Netflix recommendation system and make it more real-time and take into account the browsing habits of our users. When not playing video content, our users would navigate the home page, engage with a few rows, play trailers, look for specific titles, using thumbs up/down to rate content, etc. All those activities were important because they could tell us what a user’s intent was. If we could further ramp up our recommendation and personalization system, it would make an even more satisfying user experience.
From a data perspective, we were collecting the data that would help us understand what users were doing in real-time and capture their interactions over a longer period of time. We could go back for a month or two to learn from their activities, personalizing the system accordingly. Apparently, that was critically important to different business units, and no wonder that a number of teams, including the search, recommendation, page construction, and ranking team, were interested. The challenge I found particularly difficult was how to manage expectations given the excitement surrounding the project from the start and furthermore, deliver the features timely without compromising on quality.
For starters, I had to understand what were the most common use cases. To do so, I talked to a number of consumers, meeting with them one-on-one. I would also meet with other teams to get precise requirements for the projects. I would ask them very specific questions such as, How will you query the data or How will your systems integrate with the new system? I also wanted to learn about their priorities. If a particular access pattern was important to them, was it more important than the other one? I wanted to understand and establish a clear order of priorities early on. That helped me plan my milestones so that I delivered the most important features first.
Next, I had to find a balance between the most important use cases. I wanted to make sure that the use cases I was targeting in the V0 or V1 release, included use cases coming from a variety of teams and users from whom I could get feedback. V1 should be good enough to address most use cases so that people could onboard in the system and help improve it by providing feedback.
We started to build a proof of concept early on, which was somewhat unique. We would begin with a smaller project that would be easy to build but could provide users with some concrete data about what the full-scale project would look like. We built it and demoed it to our users to see what they were thinking. When one is in the early stages of a project, most of the time, one talks hypothetically: I will do this or that. Not surprisingly, users may have different interpretations of what that should be. It’s important to be on the same page from the expectations standpoint from the start.
We build a proof of concept application in a few weeks time and then host a demo for a larger audience. We explained how things would look like, how fast it would be, what data could be captured, and made available to the consumers with acceptable latencies. We asked them for their feedback explicitly: Is this the direction you had in mind? How far is this from your expectations? Based on their feedback, we formalized what V0 should mean in terms of requirements, specifications, and timeline.
We also wanted to regularly show progress to our users. Since everything was taking place after the Covid pandemic hit, demoing through a screen was the only option. I missed going to users in person and collecting their feedback chatting in the hallway. We were demoing to a larger group at least once a month, frequently doing demos to different teams weekly.
Doing demos was a great opportunity to get users interested in the project. We also built a tool that could do some visual demos. If a demo cannot be visually interpreted, it would be less appealing. When someone is producing data as a project, they should find a way to make it more attractive. Building some UI or visualization tool that could take that raw data and present it visually is critical to draw interest. We enabled users to see what was being created to be visible on their Netflix screens.
By moving from the early idea to a more mature product, we also moved from managing expectations to managing adoption. You can build something great, but your effort will be futile if no one is using it.
We were trying out a new technology that was not mature and supported enough. Alternatives were simply not a good fit for our use case. We could have used three other available and well-supported technologies, but they were not a great fit for this particular project. We started using this new technology, and of course, as with any new technology, we expected a lot of hiccups, especially when operated on a large scale. If you are going with a less adopted solution, consider the failure domain and build tooling early on to mitigate failures. For example, when a catastrophic failure happens in your data storage, you should be able to build that state back. In this process, we helped improve the new technology around its stability at a larger scale.
We built the system for scale from day one. We didn’t launch this project to build it for 1 percent of users and then slowly increase to 10 and 50 percent. The reason is simple: we are building a data store that all users will use. If it gets adopted and people want to onboard, we don’t want to put them on hold for a quarter or two until we try a new system that can scale. We launched this system with 100 percent traffic from the early on. The scale is the most important requirement; if it doesn’t scale, it would be a failure no matter how good the data is.
Start your integrations early on. Most often, companies have multiple services and libraries that consume each other to produce a complete and cohesive system. If you are going to introduce a new system, talk to people who will be either producers or consumers of the system and make sure they can integrate. If you do this integration early on, you will create a feedback loop, which will help you understand which parts of the system you will not be able to integrate.
Be notified about next articles from Ramayan Tiwari
Senior Software Engineer at Netflix
Connect and Learn with the Best Eng Leaders
We will send you a weekly newsletter with new mentors, circles, peer groups, content, webinars,bounties and free events.