max-ostrozhinskiy-132517-unsplash.jpg

Digital Product Metrics Reporting Strategy

Digital Metrics Reporting Strategy

 Until recently, this same benefits processing organization has only allowed applicants to submit their applications on physical paper forms. In an attempt to modernize, the organization has been digitizing forms over time, so that applicants can transition from mailing applications to submitting online with the click of a button. As the organization continues their digital transformation, our team set out to learn the following:

How might we create a human-centered metrics reporting strategy that allows stakeholders to make informed and timely decisions, prioritize research efforts and form generation and demonstrates the impact and value of the organization’s business investments?


My Role

For this initiative, I lead a team of researchers, interaction designers and business analysts through the design process, beginning with the creation of the research plan and culminating in the implementation of a metrics and data collection pilot.

*Due to confidentiality terms, client names and strategy have been intentionally omitted.

Approach

Design

  • Taking the metrics that emerged in the stakeholder interviews, created a metrics reporting framework to assist in data point identification and prioritization for current forms and features in production, as well as for planning future releases

  • Hosted co-creation session with Product Owners to prioritize and down select the most important features based on level of impact and perceived value to both the user and the business

Discover

  • Hosted visioning workshop to understand the client’s vision for the product

  • Conducted ~30 stakeholder interviews spanning across 7 job types to understand how data was currently being used and to understand what information employees needed to know to make key decisions

  • Synthesized research findings into high level themes and key insights in the form of a research readout

  • Analyzed ~66 user tasks of forms and features currently in production to determine task beginning and end and success and failure

Key Insights

  1. Because there were multiple interpretations of the product vision, decisions were being a made against varying definitions of success, making it increasingly difficult for employees and leadership to see their impact.

  2. Without being able to show the product’s impact and reach, the organization was missing a larger opportunity to capitalize on a sustainable approach to product development and iteration.

  3. Measuring functionality was only one piece of the puzzle. While it was important to capture whether or not the product was working, functionality metrics were not telling the organization anything about the user experience or business value.

  4. When the value is clear, the direction is clear. If the organization utilized data as evidence to show the value behind work prioritization decisions, it may be easier to align parties with conflicting interests. 

Strategic Framework

Screen Shot 2019-06-02 at 11.20.27 PM.png

During our research, we learned that increasing digital filing was unanimously a key indicator of product success. However, there were multiple variables that impacted whether or not filing rates increased or decreased. Some the organization had direct influence over, but just as many, they did not.

Many of the stakeholders we interviewed identified adoption as essential to product success. In this context, we defined adoption as the result of the three following components: User Experience, Awareness and Functionality.

While the product team did not have direct control over what was happening with their business partners, it became clear that monitoring business operations metrics had the potential to provide insight into how actions taken by the product team might affect the business in the following three areas: Strategic Priorities, Assessment of benefits, and inquiries via the Contact Center.

  • Measuring User Experience answers the question: Do people enjoy using it?

  • Measuring Awareness answers the question: Do people know the product exists?

  • Measuring Functionality answers the question: Does it work?

  • Measuring the achievement of Strategic Priorities answers the question: Is the product helping the business reach its organizational objectives while reducing cost?

  • Measuring the Assessment of benefits answers the question: Is the product providing benefits processors with the information the need to make a case decision?

  • Measuring inquiries via the Contact Center answers the question: Is the product decreasing

    call volume and increasing online self-service?

Outcome

To test the customized framework, we prioritized 5 high profile features to test in a pilot. The infrastructure has received funding and resourcing, and the infrastructure is currently being built to support the pilot, slated to begin in July 2019.