Creating From Scratch Without Any Prior Knowledge
10 September, 2021
I have been in this situation a great many times. I had to create a data lake using AWS or Google cloud platform. I got data from production and put it in a data warehouse, let’s say Snowflake or Ratchet. I would have to create a data streaming platform using Kafka and put some machine learning algorithms in production too. Then I had to develop the platform, train models, and publish predictions.
It all looks seamless, right? But where to start from when you are not familiar with the domain or technology?
I would reach out for help wherever possible. To start with, I would reach out to my peers -- either those within the company or outside. I wouldn’t necessarily stick to my narrow field of expertise only; great ideas and inspiration could come from anywhere. I would also not shy away from approaching the competition because, in today’s world, people who were competitors a year ago may end up working on the same projects in no time. I most often approach peers outside my company through LinkedIn. Also, looking for help through mentorship platforms such as Plato should also be an opportunity to be explored.
If one is using AWS or GCP, they can reach out directly to the company and ask them to help them out. Sometimes a free trial could be obtained, or one will be directed to a consultancy firm that implements their solutions and can help. For example, Google paid ten days of consultancy to the firm they were working with to help us onboard. For ten days, we could toy around with a new tool free of charge.
Further on, I would read whatever I could get my hands on: documentation, live content on YouTube, blogs from reputable tech companies, etc. Taken all together, this should be enough for one to get a basic understanding of the domain and start proliferating some ideas of their own. However, this is the most difficult moment. A person would acquire all of that great knowledge and then will have to take a leap and start creating something on their own. What makes this leap difficult is that in this phase, one needs to make some decisions. After studying different approaches, a person could be overwhelmed with multiple ideas, not knowing which one to choose. To choose the right one, I created a matrix with a number of variables: cost, time to implement, internal expertise, scalability, availability of skilled engineers, etc.
Finally, whenever I build something from scratch, I like to take a pragmatic approach. First, I do an MVP of the platform, after which I start building things incrementally. I find it useful to break down one large project into several smaller segments, each being contextualized. For example, if I need to do a large data platform, I would create it first for the accounting team, then the marketing team, or even one or only several use cases for the marketing team. If it works for one use case, I will keep adding other use cases. Only once I am certain of what I am building, I would start thinking about a roadmap.
- There is no way for you to try everything. In the best-case scenario, you should go with three approaches at most. In today’s engineering world, there are too many solutions for the same problem. People often get stuck in a try-mode and end up indefinitely trying out different things. You need to limit a period over which you will be trying new things, and once it is over, you will make the decision.
- Be pragmatic. For me, being pragmatic means start small and iterate. See for yourself if your solution works on a small scale and if so, scale it up.
- Read the documentation. We are using more and more tools, and all of those tools are coupled with documentation for a reason. Each time we have a problem, we go and Google it. If you read the documentation first, you will save yourself a lot of time. Read the documentation before you try to do the implementation. Act the same way as when you are assembling furniture -- you read a manual before you start screwing bolts.
- People are in general nice and want to help. So, don’t shy away from asking for help.
Scale your coaching effort for your engineering and product teams
Develop yourself to become a stronger engineering / product leader
Senthil Kumar, Sr Engineering Manager at Quotient Technology, explains the challenges he faced while managing an engineering team and how he collaborated with the product management team and other stakeholders for overcoming these challenges.
Sr Engineering Manager at Quotient Technology
Sachin Shah, CTO, shares how he created an elegant solution to improve the delivery process.
CTO at Self Employeed
Rohit Karanam, Director, Product Management at Teradata, discusses how to strike a balance between good enough and perfect when getting a product out the door fast becomes a priority.
Director, Product Management at Teradata
Gaurav Sharma, Engineering Manager at GlobalLogic, shares how he enabled cross-team collaboration and delivered valuable end-to-end functionality.
Engineering Manager at GlobalLogic
Bertrand Dubaut, Director of Engineering at Zivver, explains his strategies and plans to improve the office environment and how to work faster even when working remotely.
Director of Engineering at Zivver
You're a great engineer.
Become a great engineering leader.
Plato (platohq.com) is the world's biggest mentorship platform for engineering managers & product managers. We've curated a community of mentors who are the tech industry's best engineering & product leaders from companies like Facebook, Lyft, Slack, Airbnb, Gusto, and more.