For a group of professionals that are so focused on measuring data, quantifying goals, and monitoring key performance indicators, there seems to be a lack of guidance for how we measure our own effectiveness. Part of that is because our function as data professionals is a support function. Our efforts indirectly effect the outcomes of other parts of the company. This leads us to not knowing if our efforts actually had an impact or not because we are a decision support group.
Our Core Mission
I firmly believe that our core mission is to assist others in making better decisions. This means if our users don’t trust our data analysis then they are not going to consider it in their decision making process. If this happens then we have failed in our core mission. Also, if reports take too long to load, users won’t use them and again, we fail. If the data warehouse crashes and data is lost, we also fail in our mission. Looking at this mission statement, we can see that we have two sets of metrics we can monitor: usage and performance.
Usage metrics refer to how people use the reports and dashboards. WIth SQL Server Reporting Services, we can see if people look at them on-screen or if they export them to PDF, Word, or Excel formats. The basic assumption is that if the report is exported it is going to be shared with another person or multiple people. I also like to track which report gets the most views and most exports. All of this data can be gathered from the report server database.
This is possibly the largest category. In this category, I track my ETL performance (how many inserts, updates, duplicate rows, execution time), my data warehouse metrics (number of fact tables, number of dimension tables), report performance (data retrieval time, processing time, and rendering time) as well as my performance (new reports deployed, new dashboards deployed, projects finished).
Reporting These Metrics
Tracking metrics is fine and all, but if you don’t share them with someone, it doesn’t do any good. Remember, the value from data comes from the action one takes after consuming the data, not from the data itself. In order to share those metrics, we need to format them so they are simple to understand. Going back to our core mission, we fail it if users do not understand our results. In order to make reporting this data as simple as possible, I manually enter the value of each metric into a simple spreadsheet. The spreadsheet contains a column for the metric name, 5 columns for a rolling 5 week period, and a column for a spark line showing the changes over those five weeks. For those metrics with goals, I also use conditional formatting to color those cells as they relate to the goal (green for on track, yellow for danger, and red for off track).
This may seem overly simplistic, but I have found that you need to start simple and then scale up as the situation requires. The simplistic reports are oftentimes the best reports.
If you want me to elaborate on gathering any of this data, I would be happy to do so in future posts – but first, you need to leave a comment and tell me that you want to see it.