Think, Know, Prove: KPIs (Continued)

Think, Know, Prove is a regular Saturday feature, where a topic with both mystery and importance is posted for community discussion. The title is a shortened version of the Investigative Mantra: What do we think, what do we know, what can we prove? and everything from wild speculation to resource referencing fact is welcome here.

I know that last week’s post was a little scattered; I hope you don’t expect differently from this week’s given that I’m, again, writing it in a bit of a rush, and all jacked up on cold medicine. (Digression: Maybe this will be the year that I start the longitudinal data keeping about when I get sick–I swear that I’ve been sick in the fifth week for at least the last six semesters in a row, and I don’t know if it’s the increase in students interacting with students and then interacting with me (lots of them seem to be getting sick in weeks three and four (or so they say), or if it’s my kids being back in school and rubbing up against their little Petri dish friends, or if it’s the fact that I pretty much stop exercising, eating well, sleeping reasonably, and the rest. It’s annoying, though. I hate having a cold.)

So, a little more on KPIs (Key Performance Indicators) before we move on to PBF next week. In last week’s post I talked a little about what they are and about a little about what we might learn from the story of how putting some emphasis on them led to both resistance and opportunity in professional baseball. I did not mean, however, to overemphasize the connections between education and baseball, and I don’t want to let the conversation get derailed by the emphasis on sport, in general or baseball in particular or even on Moneyball for that matter–the book and story described therein has plenty of critics, too. We don’t have to come up with KPIs because of Moneyball, nor because of our corporate overlords.

We have to develop them because our new Mayor promised while running for election to develop institutional report cards for all of the city and sister agencies that would show how they were doing in key areas of responsibility (see #9, #34, and #36). He’s promised to post these report cards for all of the agencies on the city web site (a.k.a., the data portal), but allowed the institutions themselves, at least initially, to develop the measures. In other words, we, along with everyone else—CTA, Park District, Fire Department, etc.—have to come up with a way of telling the public (vis a vis the Mayor) what our Won-Loss record is (to bring it back to sports terms).

This has to be done with care, obviously for the same reason that it has to be done with care by hospitals: adding up how many have died and how many have lived likely provides a misleading picture of the hospital’s quality. The challenge for the hospital is that to the general public, that may well be the stat they want to see, owing to their own desires and poor or careless thinking about the topic. So, assuming they don’t have the resources or time to embark on a massive health literacy campaign, they would have to define what a Win looks like through the metrics they choose. In other words, their stats have to tell a story. If they don’t, then the public/Mayor won’t accept them and will likely demand the commonly accepted and utterly misleading measures.

When the public thinks about the fire department, they might want to know how many fires the department puts out and how many people are saved and guess that seeing those numbers will tell them how the fire department is doing. But those numbers are out of the fire department’s control, and if the department considers fire prevention a key part of their job, they might say that their goal is to keep those numbers low while the public is thinking that high numbers equal excellence. The department would say just counting the numbers of fires we put out doesn’t tell you what we do, and so might focus their self measurement on their responsiveness, or fire hydrant readiness, or building inspections or something.

So, as the Realist pointed out in comments under last week’s TKP post, defining what a “win” is (or to take out the sports analogy, an effective interaction of our institution with the members of the public who avail themselves of it), has to be the first step, and I am writing here to urge all of those involved to think big and creatively rather than merely follow the path of least resistance or be tempted to “juke the stats” as they say in The Wire in order to merely keep feeding at the trough.

I have a lot more to say about all of that, but not the time to develop it. A few key ideas though, all influenced by but not entirely explained by Moneyball are these:

  1. Not all stats are KPIs and the most dangerous ones are often the familiar stats, widely accepted as KPIs, but empirically unrelated to aim of the organization.  Statistics had along been a part of baseball, and a long time ago (back in the late 19th century) a handful of statistics became the standard KPIs. But not every statistic or data point is a KPI, even ones that have long been thought to be. For example, the RBI (Runs Batted In) stat was a long valued and highly prized indicator of hitting prowess. Turns out though, that it is a very poor predictor of anything. Why? Because the statistic relies on so many variables out of the batters’ control–that other people were on base, in scoring position, etc., there turned out to be no correlation between a batter’s RBI stats and offensive production. For decades, it seems, baseball people had been operating under the assumption that runs were produced by hits, when it turns out that runs are produced by avoiding outs, and so a much better KPI for being a baseball hitter is the measure of how often the hitter avoids making an out. A team that makes no outs scores an infinite number of runs. Batting average is another important and slightly misleading stat. A batter who hits .333 and never draws a walk gets on base one time for every two outs the batter makes. Another batter who hits .250 and gets say two walks in every  five at bats, ends up avoiding an out in 3 out of every nine at bats. In other words, though their stats are very different, their value to the offense is about the same. Pointing this out doesn’t mean that knocking in a run (earning an RBI) isn’t valuable–it’s still a good thing–it is! It just isn’t very helpful in distinguishing between highly valuable hitters and average hitters. The kicker is that most people think it IS exactly that–a means of evaluating the hitters’ excellence. In other words, we should be on the lookout for statistics and measures that have been traditionally accepted as obvious and important that aren’t (because they are based on false assumptions).
  2. Inputs are key. Another lesson of the story is that some things are very hard to measure. In baseball, that hard thing is defense. It’s hard to determine how a team’s defense contributes to a team’s victory (and so how much value to put on a player’s defensive skills as well as how to measure those skills). The traditional statistic for measuring defense was the “error,” which Bill James pointed out is completely misleading. Players get an error for not making a play that the “official scorer” thought they should. In other words, it’s a moral judgment as much as a performative one, and it’s likely that bad defenders could still  have very few errors and good ones would end up with more. Take two second basemen–one who covers a lot of ground and one who covers very little. The former is likely to come near or even get to many balls that when the other one can’t get near. Some of those might get turned into outs, but some of them might require difficult throws or be more challenging glove work and so look like errors, whereas if the other fielder were playing they would have been simply scored as hits. And if that doesn’t make much sense, then don’t worry about it. The point here is not errors or second basemen. The point is that good stats require good, meaningful inputs, and non-meaningful inputs tell lies.
  3. Models matter. Applying data analysis in a meaningful way requires having some testable hypotheses about what matters and then testing those hypotheses. As those hypotheses are developed and tested, they can be combined in more and more complex ways in order to attempt to build a model or a formulaic understanding of what is happening and why. No model is perfect, nor is it supposed to be.  One way to do that is to come up with statistical models (formulae) and then apply those retroactively to see if they accurately predict the outcomes that occurred in connection with years worth of historical data. If they don’t, then one of three things must be the case–either we haven’t included all of the meaningful variables, or we’ve included a variable that is not meaningful (and so skews the prediction) or we have the values wrong. I’m dreaming of a formula built from national statistics that could accurately predict some outcome or other (course completion, degree completion, successful transfer, whatever, more on that in a minute) that provides a prediction for what percentage of our students (taking into account various meaningful factors (preparation, 1st generation student-hood, percentage part time, incoming skills, and whatever others are suggested as meaningful in the literature) that matched (roughly) the actual numbers of students who achieved those outcomes. If we had a model like that, we could then build a formula, punch in our data and see if we had more, less, or the same as the prediction. If we have more students achieve, consistently, maybe we’re outperforming the average schools. If we have fewer, maybe we’re underperforming. If we’re the same, then we’re at least as good as everyone else. Or even just a model that makes some sort of baseline. A model like that could be transformative to community college research and funding, and it would take a lot more work than looking up how many students were awarded a certificate last semester while telling us a whole lot more. And while there is risk in such a scheme, if we really are about students, and excellence, and social justice, then we have a professional, and I would say moral obligation to avoid the temptation to cherry pick a few stats that make us look good and find some meaningful data to report.

 

Now with all of that said, I am the first to admit that I am in way over my head when it comes to stats and quantitative research. I can just barely calculate a batting average, which I admit to as a point of shame, not pride, and perhaps what I’m dreaming is simply not possible. Fine. Still, I say, as I said last week, this is a spectacular opportunity to have some crucial conversations and explore and explain our professional expertise to the general public, their legislators, and the world. Aim big is all I’m saying; swing for the fences as it were.

For lots more on Moneyball, you can go HERE.

 

3 thoughts on “Think, Know, Prove: KPIs (Continued)

  1. Hi PhiloDave,
    Hope you are feeling better. I know I am after my rant from last week. I think we are on the same page so I will proceed to the next phase of this discussion. As I continue to think about KPI’s, I offer these words of advice to those who will be directly involved in the process. These thought have come to mind since reading your comments and also from past experiences.
    1. We should recognize that in the end whatever KPI’s we develop will need to be instituted District-wide. We will not be able to create, much less measure our uniqe KPI’s for funding. We may be able to do it so we can sleep well at night, but no way will the Chancellor, much less the mayor, permit this. We may be separate buildings, but all the funding goes to District and then District distributes.
    2. With this in mind, I believe we need to think about our students; not just the ones that walk through our doors, but also the ones that attend our sister colleges. What do they all have in common? For one, we need to know how many of them are working and attending school. Out of those that are working, how many are employed full-time versus part-time vs not currently working. Then we need to know how many hours they’ve registered for. I’m just brainstormin’, so if others have better ideas, please share.
    3. Given the first two points, I will reinforce what you have stated: I too “urge all of those involved to think big and creatively rather than merely follow the path of least resistance.” The last thing I want to hear is that we can’t conduct such a survey because the free version of surveymonkey limits our options. (BTW, why don’t we have an institutional agreement with a survey company? Why do we keep using a free version with so many limitations. IMHO, this is ‘ass’-inine. We aught to be ashamed of ourselves. Don’t tell me to use Blackboard; more to follow on that.) The other last thing I want to hear is that we don’t know how to do that with PeopleSoft. BTW, PeopleSoft is nothing more than a database on steroids. From what I know about databases, they are really good at collecting data and allowing the user to query the data. We really need to know what PeopleSoft can do and what it can’t. Something else I don’t want to hear is that we will not be given access to some of these features. Why? Why can’t I query the number of students that have enrolled in English 101 over the past 6 semesters to determine if there is a correlation between attendance drop-off and time of day?
    4. Let’s use the right technology for the right reasons. Why are we using Blackboard for anything other than a supplemental resource for a course? Faculty councils and departments across the district are using it to do things other than teach? Why? There is other technology to serve our purposes. We need to look; and look hard, for the right solutions to address the concerns. A misuse of technology could lead to a miscalculation of statistics. I’m a tech simpleton compared to bright minds that gather data for a living. Get the right people on board and give them the right technology.
    5. Make sure that when KPI’s are established, we are not locked into these measures forever. For one, these KPI’s will be meaningless the first year. Repeat: THE FIRST YEAR KPI’S ARE MEANINGLESS. Not until you have a second year can you begin to make comparisons. [Starlin Castro just got 200-plus hits this year. Is he a good hitter? We need to see how he does next year, and the year after that, and the year after that. Ya’ got me?]
    6. Somepeep tell the mayor that our results are not always immediate and to not hold us to the immediacy of his agenda. If we ain’t done nothin’ for some time, according to his report card, then don’t expect us to do somethin’ that quick. (An object in motion tends to stay in motion and an object at rest, well, it takes time.) I’d like to think our educational purpose is more than just a check box on the mayor’s list of initiatives.
    6A. I don’t care if we can distort the numbers to appear successful in the eyes (and budget) of the mayor during the first year. To paraphrase PhiloDave’s words, it is our moral and educational obligation to look past the narrow-minded mission of the mayor. Per the linked article, “Reinvention aims for nothing less than to make the City Colleges an economic engine of Chicago.” We are better than that. The true goal of educating our students is not to ‘get dem a good job.’ It should be to teach them how a job is nothing more than a stepping stone to something greater; perhaps to be a mayor of a town that will do was is morally correct and ethically sound, like knockin’ on the door of a former mayor and holding him equally accountable for this mess we’re in. (Sorry, I promised not to digress. I feel better.)
    7. Ponder the purpose of education while developing KPI’s. As educators we should always address issues from an academic POV. We should always ask ourselves how any endeavor will benefit our students; not us or our salary, or the mayor and his approval rating. Since we educate our students, we should have their best interests in mind, and dare I say, at heart.

    I too shall encourage more peeps to give their two cents. Chump change adds up in a hurry.
    Peace. Out.

  2. In “The School of Rock” Jack Black’s Dewey Finn tells the kids that Rock ‘n Roll used to be a way to “stick it to the Man” but then the Man took care of that with “a little something called MTV.”

    In other words, “the Man” (DEWEY: “Oh, you don’t know the man?”) always appropriates that which would destroy him.

    Below is a link to “Social Power and the Coming Corporate Revolution: Why Employees and Customers Will Be Calling the Shots.” The article celebrates digitally enabled social power only to contain it, advising that corporations must work with and not against social media. A combination of Generation-Millennial rhetoric and photos of jeans-clad CEOs posing in profile before stark brick walls blurs the distinction between a “consumer identity” and a “laborer identity.” (One photo’s caption reads “Employees don’t want to feel like ‘the Man’ is just telling them to do things.”)

    The article’s stylistic absurdity seeks to inoculate it from the charge that capitalism creates a “false consciousness.” In the article Harvard Business School professor Shoshana Zuboff states “the mass-production economy provided existential security for many, many people,” thereby producing a new human mentality of self-determination that was once reserved for society’s elite. She goes on to say that companies no longer create “value.” Instead, value “rests in the unfulfilled needs and desires of the individual,” so consumers are really in charge of corporations.

    And the individual can be reached via social media.

    I’m not suggesting anything negative about the proliferation of blogs since the Reinvention began. My focus is on the article’s style and rhetoric, and its location of “value” within each individual, “empowered” consumer. This mindset or paradigm shift seems to inform the struggle over KPIs. Some argue that the struggle is between supplying “a credential of economic value” vs. merely “a reputational good” (supposedly colleges currently sell these). Others argue that the struggle is between a reductive emphasis on “economic value” vs. BOTH the economic and the often ineffable yet real value that we as educators give to our students (e.g. quality of life, adaptability/critical thinking skills, etc.).

    PHILODAVE on 9-24-11: And if we don’t take it seriously both in planning and execution, that window will close and we may find ourselves on the outside looking in.

    In other words we academics – like the corporations – must “employ the methodology of data analysis.” The sticking point for me is that empowered consumer/self-determination (“false consciousness”?) thing. Or, as PhiloDave writes in TKP 9-11-11 re: Child Development AAS/AA, “I’d be more persuaded by A if the students surveyed were at the end of the program rather than the beginning.”

    Beyond that, I’ve nothing else to say at the moment, except that a purist’s position won’t work here.

    (DEWEY FINN: “Now raise your goblet of rock!”)

    http://www.forbes.com/forbes/2011/0926/feature-techonomy-social-power-corporate-revolution-kirkpatrick.html

    • I hear and respect your reticence, Avramakis, and I don’t want to suggest that KPIs are our chance to call the tune–it’s clearly being called by someone else, but this is, I think, a little like the Saturday Night Request Line. And I am fairly certain that if we don’t do it, it will be done to us. I suppose there’s always the option of non-participation in the system (but I don’t see how that helps anything.

      I’m reminded of Sartre’s short story, “Dirty Hands” (have I mentioned that one before? Feels like I have; sorry if it’s true). I know that you are not advocating for any purist position, but rather expressing legitimate and justified hesitation (as well as thought provoking material). All good…along with Dewey Finn and goblets of rock. Thanks for the thinkish contribution.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s