Bad data viz can have bad consequences

By Samantha Grant

Good data visualization matters. I think about, blog about and train others on ways to improve the way that we share our evaluation findings.

Recently, I had personal evidence of how good data visualization matters. My daughter came home from school upset because she scored "urgent intervention" on a standardized math test. I initially didn't believe her because she's a strong student, even though she would rather have her nose in a book than solve multiplication problems. We went through all of the reasons that her score could have been low-- from a bad test-taking day to the test covering topics that weren't discussed in class.

At parent-teacher conferences the next week, I saw the report on her math score. My stomach dropped, not because my daughter needed help - just the opposite. Her math score was fine, but the data viz was bad. This is exactly how the graph looked:

The triangle at the top shows how she did on this assessment. She scored at the 92nd percentile for this test - far from a need for intervention. The “urgent intervention” category at the bottom refers to the black area on the left, not to the line above the bar.

I explained to my daughter and her teacher that in my job, I work to make sure that my team doesn’t create graphs like this. Because of this poorly made graph, my 10-year-old spent a week thinking that she needed math remediation. As a general principle, we don’t want our data visualizations to cause stress in our target stakeholders.

Let’s take a look at the graph to see where it went wrong and how it can be re-made. Some solutions to very common problems:

1. Look at your graphs in color and in black and white. No doubt the graph above was constructed in color. Our school avoids color printing costs- as is typical for many schools and non-profits. If you think your stakeholders are going to print in black and white, take a look at how it looks in that format.
   
   Cool tip: you can create a color version of your graphs and then create another one for black and white printing.


2. Put the data labels under the corresponding bars. If the urgent intervention label wasn't found directly underneath the student benchmark line, the connection between those two would never be made. Labels are intended to simplify your graph.


3. Make meaningful titles. Right in the title it could have said. “Student is at/above benchmark.” Even a fourth grader (or an exhausted parent after a night of conferences) could make sense of that.


4. Test it out. Chances are if the designers of this graph showed it to someone, they would have noticed the errors. Testing takes time, but it makes up for it in the end. I promise.

Now, let’s take a look at an updated version of the graph. Much better, right?

I'm someone who spends a lot of time working with graphs, and making graphs presentable in black and white is always hard. Color helps our eyes to find what we should be focusing on, but until all of our reports are web-based, working with black and white is going to be a necessary evil.

Is this graph perfect? No, but it gets the right information to the target stakeholder.

Is this graph going to make sense to a fourth grader? Yes. Problem solved.

What data visualization tricks do you have? Have you been affected by a data viz blunder? Share your story here!

-- Samantha Grant, evaluation director

You are welcome to comment on this blog post. We encourage civil discourse, including spirited disagreement. We will delete comments that contain profanity, pornography or hate speech--any remarks that attack or demean people because of their sex, race, ethnic group, etc.--as well as spam.