Identifying the “Right” Problem to Solve

Problem

When I worked at a content site, there were several projects my team inherited. Some of these solutions were built by multiple vendors for capturing customer registration data for a coupon or the like, offered by our sponsors. The captured data was then made available to sponsors - not very securely all the time. The vendors used different databases for each of these solutions and not necessarily the same fields.

After my team began working on these projects, the reason for pulling these from the vendors became apparent. The publishing team needed to change the UI on a regular basis, the data captured was not clean and the data transfers were not reliable. To address these issues, my team was releasing code on a weekly basis, which was very tiring, and still was not enough to keep up with business needs. We needed more work from a demoralized team, faster and with more accuracy.

Actions taken

Clearly, the process we inherited was not manageable and was causing challenges to business and engineering teams alike.

With a new problem, we tend to come up with a solution for the immediate issues at hand and we did the same. We started moving faster to keep up with business needs, with a dedicated group of people to work on it, even though each solution was very simple in nature and should not have required that many people on it. I was spending more time communicating on this group of projects than any other project my team was handling.

We needed to fix something, but at that time, with fires all around us, it was hard to notice the bigger issue.

The issue was that there were too many solutions, not standardized, and not built in a structured manner.

Once I realized that I decided to chase the bigger problem of the solution itself, besides the immediate problem of releases. I had a senior engineer join my team to help me redesign these solutions. We came up with a strategy that would move all of these solutions to a standard platform, without giving up the uniqueness of the HTML built by our sponsors.

We made a way to post the forms to a publishing system that would read through the HTML to identify the form fields within it. This system would enforce a few fields for the inner working of the system, but leave everything else for the individual registration form to decide. Besides the required fields, all other data was stored in key-value pairs and an admin tool allowed publishers to manage the fields that are on the form and define the format of files that were to be sent to the sponsors.

The data API layer was built with security in mind and encrypted some data at rest per HIPPA compliance. All form pages posted data over SSL - no choice was left for individual publishers and the file transfers were allowed from central sftp servers and the process for file transfers was standardized.

Once we implemented this solution, the engineering team managed the backend solution and allowed publishers to publish sponsors’ forms via the publishing tool. They could publish a change, review it on the staging server and publish it in real-time.

Publishers no longer had to wait a week for their changes to appear. They had full control of the UI for the forms and with limited restrictions on the field names, they had the flexibility to add form fields important for the sponsors.

By building a new product, my team solved three problems at once. Publishing could make UI changes in close to real-time. Secondly, my team became more motivated once we had a standardized system to work on and with a smaller frequency of releases. Thirdly, using SSL, data encryption and SFTP made the info secure, protecting our customers and establishing our companies credibility.

Lessons learned

• Anytime you have a problem, it is best to think outside of the box. It is easy to focus on solutions before us, but this often does not empower us to pinpoint the best approach to a problem. It’s best to visualize your end goal and work backward from there, asking yourself how you can reach that finish line.
• Standardization and structure are best. Once we standardized our data, our team could share and analyze this information more effectively. Our team was more enthusiastic, and it benefited our customers as well.