Improving a cloud-based developer environment

dbt Labs

Overview and goals

As a lead designer for my product team, I worked on multiple iterations of improvements to the web-based IDE for dbt Cloud. This case study covers rapid, iterative work that was implemented over a longer time period. The focus of this work was on making quality of life improvements to the IDE and improve metrics related to usage and user satisfaction, with the goal of making it a scalable and accessible way to work on dbt projects beyond the core open source product that is run locally.

Process

I started my role as the new lead designer on an existing project to update the IDE experience. This work started with one larger refresh, and then continued in subsequent related updates, to get it to where it is today. So my overall approach was based on building on an existing foundation of research, strategy, and design decisions, and planning ahead from the start to iterate and expand on that work.

Overall approach

In this case, I started as the new lead designer on this work, and picked up where that work left off, based on previous research and planning for the release. I worked with my PM and engineers to better understand how this functionality would work, what tools would be incorporated, and to plan out the best initial release that could be iterated on over time. I started by evaluating the current experience of the IDE, based on my own background as a data analyst. During my ramp up period, I focused on understanding business needs, reviewing analytics, and learning more about users' technical needs.

As the work continued, I started to gathered user feedback from multiple channels on each product release. I learned more about current data engineering and data industry practices in order to ask the right questions during my design work. I gathered information from our dbt Community Slack channel, and conducted surveys and interviews to get more information on what users knew and how they think about their work. I conducted user interviews with engineers and analysts to better understand engineering norms and how they collaborate with stakeholders. Internally, I worked with solution architects to understand users' common questions, and to shadow the training and onboarding process. I worked with sales reps to understand the background behind why organizations want our product.

All of this informed my design work over time, so that even when I didn’t have the bandwidth to get direct user feedback on my designs, I could still make informed decisions. It has allowed me to be flexible thorughout thsi process, while I also being able to advocate for users effectively.

All of this was done in collaboration with the team’s PM and engineers. When working on any designs, I kept a tight feedback loop, leaning on their knowledge of the business and underlying technology, and focusing on how to translate what our product does to users’ overall data analytics goals. The goal was to remove barriers to comprehension, roadbumps and friction. This experience-related work would happen in tandem with performance improvements, with a goal of having changes happen holistically and not feel disjointed.

The above describes the overall arc of this work; in practice, it can be broken down into the following chunks:

Major update to the cloud-based IDE

Initally based on patterns of feedback from users, as well as the business goals of increasing NPS scores and user satisfaction, improvements were made to how information was displayed to users, how they could access common actions and execute dbt commands, and how they could create necessary files and perform git operations within their organization’s projects.

Adding linting functionality

at the time, I provided multiple options to encourage feedback and discussion on the team, and to help plan how we could respond to user feedback after release. This was one feature that came up multiple times, both internally and externally. This work also included working closely with the dbt communit to flesh out supporting this functionality.

Incorporating new dbt Core functionality

The goal of this work was to implement features that had been added to the core functionality of dbt, and think about how to present it for Cloud users. The top releases included supporting dbt mesh and deferring to production.

Improving error messaging

Starting with good usability principles, this work focused on including permanence to viewing errors, improvements in error language and presentation, and automating error prevention and correction. This was all informed by feedback from users through multiple channels, along with collaboration with engineering to understand the feasibility of solutions over time, and learning about the context of the errors that would occur.

Improving git workflows and functionality

This work was broken up into even smaller iterations, and the implementation is still in progress, but the overall goal is to make the different git actions easier for users to understand. This work started with minor improvements based on user feedback, seeing issues after viewing session analytics in Datadog, and observing training & onboarding sessions. Based on that information, I focused on making changes that would “translate” git for users, such that the interface better represented what the consequences of their actions would be, rather than just mirroring git terminology and what is usually shown in VS Code.

This work included improving git error messaging, rethinking the workflow for committing work, submitting PRs, and creating branches, and adding the ability to perform git actions on individual files.

Results and conclusion

The outcome of all of this work? Continued and sustained increases in NPS scores, increased weekly and monthly active users, and an increase in the number of organizations using dbt Cloud for their dbt projects. There was also a decrease in customer support tickets, and praise directly from users for the improvements they were invested in seeing and gave feedback on. Overall, there has been a more positive reception to the IDE experience after the many improvements my team has implemented, and our ability to deliver a steady stream of improvements over time.

Advocating for the importance of multiple data practitioners

Outside of reaching business goals and increased metrics, another impact of these projects is related to the perception of our product users. Throughout this work, I worked to build an understanding of all data practitioners, beyond only engineers, and emphasize the importance of their roles within dbt projects and as dbt users. This has never been a singular effort; other designers, engineers, and PMs within the organization have also advocated for data practitioners, and we’ve all built upon each other’s research and insights to get to where we are now.

Lessons learned

Updating an agency website and workflow