Creating From Scratch Without Any Prior Knowledge

Problem

I have been in this situation a great many times. I had to create a data lake using AWS or Google cloud platform. I got data from production and put it in a data warehouse, let’s say Snowflake or Ratchet. I would have to create a data streaming platform using Kafka and put some machine learning algorithms in production too. Then I had to develop the platform, train models, and publish predictions.

It all looks seamless, right? But where to start from when you are not familiar with the domain or technology?

Actions taken

I would reach out for help wherever possible. To start with, I would reach out to my peers -- either those within the company or outside. I wouldn’t necessarily stick to my narrow field of expertise only; great ideas and inspiration could come from anywhere. I would also not shy away from approaching the competition because, in today’s world, people who were competitors a year ago may end up working on the same projects in no time. I most often approach peers outside my company through LinkedIn. Also, looking for help through mentorship platforms such as Plato should also be an opportunity to be explored.

If one is using AWS or GCP, they can reach out directly to the company and ask them to help them out. Sometimes a free trial could be obtained, or one will be directed to a consultancy firm that implements their solutions and can help. For example, Google paid ten days of consultancy to the firm they were working with to help us onboard. For ten days, we could toy around with a new tool free of charge.

Further on, I would read whatever I could get my hands on: documentation, live content on YouTube, blogs from reputable tech companies, etc. Taken all together, this should be enough for one to get a basic understanding of the domain and start proliferating some ideas of their own. However, this is the most difficult moment. A person would acquire all of that great knowledge and then will have to take a leap and start creating something on their own. What makes this leap difficult is that in this phase, one needs to make some decisions. After studying different approaches, a person could be overwhelmed with multiple ideas, not knowing which one to choose. To choose the right one, I created a matrix with a number of variables: cost, time to implement, internal expertise, scalability, availability of skilled engineers, etc.

Finally, whenever I build something from scratch, I like to take a pragmatic approach. First, I do an MVP of the platform, after which I start building things incrementally. I find it useful to break down one large project into several smaller segments, each being contextualized. For example, if I need to do a large data platform, I would create it first for the accounting team, then the marketing team, or even one or only several use cases for the marketing team. If it works for one use case, I will keep adding other use cases. Only once I am certain of what I am building, I would start thinking about a roadmap.

Lessons learned

• There is no way for you to try everything. In the best-case scenario, you should go with three approaches at most. In today’s engineering world, there are too many solutions for the same problem. People often get stuck in a try-mode and end up indefinitely trying out different things. You need to limit a period over which you will be trying new things, and once it is over, you will make the decision.
• Be pragmatic. For me, being pragmatic means start small and iterate. See for yourself if your solution works on a small scale and if so, scale it up.
• Read the documentation. We are using more and more tools, and all of those tools are coupled with documentation for a reason. Each time we have a problem, we go and Google it. If you read the documentation first, you will save yourself a lot of time. Read the documentation before you try to do the implementation. Act the same way as when you are assembling furniture -- you read a manual before you start screwing bolts.
• People are in general nice and want to help. So, don’t shy away from asking for help.