The Data has a Problem, Part 2

Behold, my beloved and buried 2014 Philadelphia Eagles.


This season was more of gut-puncher than most. We had breakouts from Jeremy Maclin and Darren Sproles; Fletcher Cox became maybe the best not-name-JJ-Watt-lineman; the utter and total collapse of our secondary; Coach Chip Kelly being the lovable-genius-weirdo we’ve been waiting for. The team had talent, the offense showed flashes, and even when losing the team was utterly memorable.

Were the 2014-15 Eagles good enough? No. But rankings don’t tell us “why” or “how” or “who”. If you took this information as a sign we need to clean house or fire the coach, you’re either on Sports Radio or you’re insane, or both. It’s simply not enough data.

Behold, my hard-won 2014 Keystone Data



Four. Four data points. Pennysylvania spent upwards of $400 million on Keystone testing, not including the work-hours of preparation, training, administration, organization, and evaluation. For four *&!%ing data points. The Pennsylvania Common Core1 has 23 reading standards (in addition to writing standards that are not assessed), that here are reduced to four:

1). Reading  Fiction

2). Analyzing Fiction

3). Reading Non-Fiction

4). Analyzing Non-fiction.

So looking at this data, I can tell that my 100% free-lunch eligible students aren’t that good at reading. Which we could learn by listening to them read or by checking their PSAT scores. We could read Johnathan Kozol’s Savage Inequalities, published in 1991, explaining in terribly painful detail how urban children were falling behind. We could ask them. I love my kids, and if they know they are loved they will be brutally honest about…everything.

Yup. That’s it. Four data points that do not tell us “why” or “how or “who”. If you took this information as a sign you need to fire all the teachers you’re either paid to say things or you’re insane, or both.

But wait, there’s less! I could try to decipher exactly why a student got a question right or wrong by looking at the questions. That would be tedious but worthwhile, especially to share with my students. Nope – the Keystone Test is something of a state secret. In fact, English teachers are discouraged from proctoring the English exam because we might “game” a system that seems primed to be gamed. The odd implication here is that our math teachers, now reduced to “active monitoring”, can’t read.

The other problem with a secret test is we don’t know if it’s a good test. We might have a “Pineapples don’t wear sleeves” moment and never know it2.  Even worse: The scoring of these tests is left to employees with minimal training or qualification. Your child’s Algebraic constructed response could be graded by someone who couldn’t successfully complete the problem.

And sure, the fact that we don’t do it right doesn’t mean we can’t do it right. But why would anyone in power do anything to change this? The Big 3 testing companies are swimming in money. The data is showing failure, which gives actors rights to do terrible things. That the assessments seem neither valid or humane is trivial compared to the other, greener incentives.

It doesn’t have to be this way.

The purpose of assessment, standardized or other, is not to announce that kids in Radnor read better than kids in Olney. Teachers, all teachers, can and do use data to inform instruction. This looks like pre-tests or retests. It’s those of us experimenting with Standards Based Grading or Kahn Academy. If the majority of the class fails a test, we do our best to re-teach, re-test, and figure it out3.

Testing should be part of the learning process, rather than the end of learning. Unless the Keystones are the testing, in which learning literally ends while the school gags on the terrible policy. “Is testing worth it?” requires we measure the value of testing. Look at the four data points culled from a secret test graded by people who are compensated less than a Starbucks barista that tells us exactly what we’ve known for decades.

What exactly is this “worth”?

  1. no, it’s not that terrible 

  2. to be completely fair, the same criticism can and should be applied to classroom tests. Local does not equal quality 

  3. this is a serious problem in math, where the standards are too many and too deep 

Leave a Reply

Your email address will not be published. Required fields are marked *