Five Reasons Why Your Data Platform Sucks

Salman Ahmed

Senior Director of Engineering at Twilio



A large number of companies in many industry sectors are continually embarking on building data products to understand and serve their customer base better. Unfortunately, some companies embark on building data products without proper business strategy, controls, and procedures in place.

Actions taken

It is important not to get mired into setting and managing unrealistic expectations around what data can and cannot do for you. Try to slay the following unrealistic expectations and architect solutions appropriately with a proper requirement hash-out first to avoid exponential pain later managing your "data lake".

Lessons learned

  1. "More data always leads to better insights." - You can collect an enormous amount of data and still not be able to turn it into competitive intelligence to transform your business. Invest upfront time in collecting business requirements and map them to well-designed architectures that help you find the nuggets of competitive intelligence.
  2. "Build the Data Platform now, enforce standards later." - Data growth without clear-cut data governance, retention, and usage policies leads to dollars spent with mediocre ROI and a huge headache down the line.
  3. "Security shortcomings will automagically obfuscate." - It is a bad idea to dump data into a data lake without thinking about sensitivities and classification upfront. If you don't have a security team, build one!
  4. "Data center space is free." - Data platforms require parallel processing compute clusters built from commodity hardware, which is different from how we have classically deployed legacy infrastructure. This means that energy consumption, cooling, software, hardware, and the systems assumptions needed to run these new systems are also different. Remember time is money and is finite, focus it on value-added proposition upfront!
  5. "Obliviate the data quality." - Data quality is critical. If you don't have it, you can't trust the results of your data queries. It's important to develop the new ROI metrics for data quality and tirelessly measure them.

Source: Four Reasons Your Data Platform Sucks

Be notified about next articles from Salman Ahmed

Salman Ahmed

Senior Director of Engineering at Twilio

Connect and Learn with the Best Eng Leaders

We will send you a weekly newsletter with new mentors, circles, peer groups, content, webinars,bounties and free events.


HomeCircles1-on-1 MentorshipBounties

© 2024 Plato. All rights reserved

LoginSign up