Skip to main content

Bad data viz can have bad consequences

By Samantha Grant

Good data visualization matters. I think about, blog about and train others on ways to improve the way that we share our evaluation findings.

Recently, I had personal evidence of how good data visualization matters. My daughter came home from school upset because she scored "urgent intervention" on a standardized math test. I initially didn't believe her because she's a strong student, even though she would rather have her nose in a book than solve multiplication problems. We went through all of the reasons that her score could have been low-- from a bad test-taking day to the test covering topics that weren't discussed in class.

At parent-teacher conferences the next week, I saw the report on her math score. My stomach dropped, not because my daughter needed help - just the opposite. Her math score was fine, but the data viz was bad. This is exactly how the graph looked:


The triangle at the top shows how she did on this assessment. She scored at the 92nd percentile for this test - far from a need for intervention. The “urgent intervention” category at the bottom refers to the black area on the left, not to the line above the bar.

I explained to my daughter and her teacher that in my job, I work to make sure that my team doesn’t create graphs like this. Because of this poorly made graph, my 10-year-old spent a week thinking that she needed math remediation. As a general principle, we don’t want our data visualizations to cause stress in our target stakeholders.

Let’s take a look at the graph to see where it went wrong and how it can be re-made. Some solutions to very common problems:

  1. Look at your graphs in color and in black and white. No doubt the graph above was constructed in color. Our school avoids color printing costs- as is typical for many schools and non-profits. If you think your stakeholders are going to print in black and white, take a look at how it looks in that format.

    Cool tip: you can create a color version of your graphs and then create another one for black and white printing.

  2. Put the data labels under the corresponding bars. If the urgent intervention label wasn't found directly underneath the student benchmark line, the connection between those two would never be made. Labels are intended to simplify your graph.

  3. Make meaningful titles. Right in the title it could have said. “Student is at/above benchmark.” Even a fourth grader (or an exhausted parent after a night of conferences) could make sense of that.

  4. Test it out. Chances are if the designers of this graph showed it to someone, they would have noticed the errors. Testing takes time, but it makes up for it in the end. I promise.
Now, let’s take a look at an updated version of the graph. Much better, right?



I'm someone who spends a lot of time working with graphs, and making graphs presentable in black and white is always hard. Color helps our eyes to find what we should be focusing on, but until all of our reports are web-based, working with black and white is going to be a necessary evil.

Is this graph perfect? No, but it gets the right information to the target stakeholder.

Is this graph going to make sense to a fourth grader? Yes. Problem solved.

What data visualization tricks do you have? Have you been affected by a data viz blunder? Share your story here!

-- Samantha Grant, evaluation director

You are welcome to comment on this blog post. We encourage civil discourse, including spirited disagreement. We will delete comments that contain profanity, pornography or hate speech--any remarks that attack or demean people because of their sex, race, ethnic group, etc.--as well as spam.
Print Friendly and PDF

Comments

  1. Nice job! I wish all schools had someone on staff to build reports for the teachers, admin and parents to better understand results.

    ReplyDelete
    Replies
    1. Kristen, that would be so great! Until that time, we need to have more people thinking about data visualization in our organizations and communities.

      Delete
  2. Awesome Sam! I like how you came up with solutions for the teacher (and us) as so many parents/guardians don't offer solutions and therefore the learning experience and engagement for adults is lost as well. The new graph is clear, and like you mentioned easily read by a 4th grader. I'm wondering if the teacher is thinking "gosh, I wonder if Mrs. Grant would like to hold a session on data viz for all of us teachers?"

    ReplyDelete
    Replies
    1. Thanks Anita! I hope the teacher liked the suggestion- but I'm sure she was rolling her eyes on the inside. :) In this case, a professional company created this terrible graph. I'm not expecting our classroom teachers to get into data viz, but I do expect educational testing firms to know their stuff.
      I would gladly hold a session for them!

      Delete
  3. A very minor suggestion: Since white is often associated with purity and black with evil, I'd rather see the shades going from dark to light rather than light to dark.

    ReplyDelete
    Replies
    1. I love that suggestion! There are so many layers to good data viz, and you pointed out one more. You'd be interested in reading Stephanie Evergreen's take on How dataviz can unintentionally perpetuate inequality. Check it out here: http://stephanieevergreen.com/inequality-part-2/

      Delete
    2. But another argument is that as scores increase, intensity of color should increase. That says your original color scheme is better. Color is not an easy topic.

      Delete
    3. All of these details are what make data viz so rich! It also points to the importance of testing it out with your audience. They might have a strong reaction to use of color- even if it's in black and white. Showing it to a couple of people to gauge their reactions can really help you to fine tune your practice.

      Thanks for the comments!

      Delete
  4. Sam--Great post! I have suffered from school test 'data viz' in the past as well! Sometimes I wish they would just put data in a table and leave it at that! Ha!

    ReplyDelete
    Replies
    1. Hi Amy. I definitely agree- often times the best data viz is no data viz at all. We might need to start a support group- parents who understand data viz!

      Delete
  5. I love how you presented your case with such a powerful story. Way to go for good dataviz!

    ReplyDelete
    Replies
    1. Thanks Mongkol. Data and stories paired together are perfection!

      Delete

Post a Comment