Think, Know, Prove is a regular Saturday feature, where a topic with both mystery and importance is posted for community discussion. The title is a shortened version of the Investigative Mantra: What do we think, what do we know, what can we prove? and everything from wild speculation to resource referencing fact is welcome here.
My first plan for this post was to kick off a series of posts on a few of the acronyms that you’re likely to be hearing a lot about soon–KPI and PBF in particular. As I started I started to write that post, I found myself writing more about the State of the College address than about KPIs (Key Performance Indicators) and, so, took that as a sign that I should switch topics.
Personally, I quite enjoyed the State of the College address yesterday. Though I loved Metoyer’s playful, hilarious, visual version, I’ve always enjoyed the formality of the presentations, too, and the convening of the entire college into a single room. I have always hated the rooms, though. I can’t hear a dagnabbing thing in room 103–just a word here and there and a muddy blarn about 60% of the time–and 1115 is too crowded when the whole college is there to be comfortable (If I had a magic wand, I’d get us a proper theater). But I digress.
In my time at the college, they’ve typically been more informative than challenging, usually focusing on our successes (enrollment, achievements, and the like) and our circumstances (most often state funding). At our department meeting, which followed the address, there was a general consensus of approval for Don’s focus on how we treat students, and a few people talked about their experiences of being mistaken for a student and being treated one way and then, upon correcting the impression, being treated completely differently, as if a switch had been flipped.
I’ve written a little about this topic before, and it was gratifying to hear that students rated their registration experience better this year by a lot (though, given what I still perceived to be a still general dissatisfaction along with the fact that we were least highly rated among the seven colleges, we obviously have a ways to go–and speaking of those survey results, I wonder why we don’t get to see them; wouldn’t you like to? I would. Maybe if the people involved in the process had more direct access to the ratings of their work, they’d be able to dig a little deeper to find that “will” required to move the needle.).
It seems that I’ve drifted back into my original topic–Key Perfomance Indicators (KPIs), and so I might as well give you a little on that, too, while I’m here.
What is a KPI? Let’s take a baseball team–call them Chicago BestTeam. What’s the most obvious measure of their quality? Their won-loss record. That’s their primary KPI. The team’s record shows how good they are. But for any fan (or gambler) or member of the industry investors or other company deciding whether to hang their name outside the stadium, that isn’t enough. Each constituency will have different interests (respectively: likelihood to win the championship (fan), likelihood to win the game (gambler), current acquirable resources and capacities/best practices (industry), revenue and net income (investor), and brand/customer base (marketing company)). The Sox (a.k.a., Chicago BestTeam) have an interest in providing each of those constituents an accurate measure or indicator of their status in those areas, and their won-loss record won’t be enough. So they’ll have to take and share other sorts of measurements, which reflect these different aspects of their efforts. Those are KPIs, too.
Those measures have to be easily understood–Bill James, one of the founders of a cultural revolution in an industry dominated by traditionalist, almost magical thinking, talked about numbers and data having the power of language when well constructed and presented, i.e., they can tell the story–but they have to tell the truth, too, if they’re going to be useful.
As we’ve all been saying for a while now, our graduation rates certainly tell “A story,” but it’s not ours, because of their reductionism and the ways that measure plays into so many misconceptions of what a community college is and does. Hospitals went through something similar when their industry was hit with the Metric Movement. Imagine working at Cook County Hospital and being evaluated (and compared to, say, Northwestern) on the basis of what percentage of people who come in actually leave alive. You’d be shouting like your hair was on fire that the comparison is an unfair one given the differences in clientele, resources, mission, and all the rest, and you’d be eager to figure out some sort of way to demonstrate how well you do with what you’ve got.
Somewhere along the way the measurers of hospitals (investors and public health officials, it seems–maybe even consultants!) figured out that a focus on process and protocols, rather than outcomes, might yield a better picture of a hospital’s quality–are incoming patients screened for psychiatric, what percentage of heart attack patients are administered a dose of aspirin, etc. There was an article just this week on the results–hospitals now follow “standard protocols” 97% of the time, up from 82% just 8 years ago. You can see the report from the accrediting commission here.
Baseball teams figured it out, too. At least some of them. When school got out in May, the first book I pulled from my shelf was Moneyball, and not because the movie starring Brad Pitt was coming out in September. I wanted to read the book because it tells the story of the way data analysis revolutionized baseball in the early 2000s owing to the unlikely success of one team and their non-traditional methodology. Faced with a small budget in a high cost industry, which created an annually increasing inability to compete with large market teams, the Oakland A’s searched for ways to find and exploit “market inefficiencies.” Their method? To use (initially) and then develop Key Performance Metrics by which they could (they thought) more accurately evaluate baseball players and so spend their money wisely. Over a five year period, or so, the A’s outperformed all or most of the big market teams at a fraction of the cost, and they did it by having better measures of baseball performance and a better understanding of which “performances” were key.
The amazing part is that Bill James and the crew of statisticians he inspired who came up with the methodology and many of the measures the A’s exploited had first published in the late 70’s, 20+ years before, and James was widely considered a krank and an outsider. Baseball was, even in the face of a ton of compelling evidence that many of its “heresies” were truth and many of its “Truths” were myths, a place where the insiders spoke in one voice–that of tradition. “You have to be a baseball man,” they said, “to know a baseball player and be able to run a team.”
Don’t educators say the same sort of thing all the time? Don’t we claim a sort of magical eye that can spot good teaching when we see it, regardless of what the numbers say? The point Bill James and company made was that the difference between a .250 hitter and a .300 hitter is about one hit every two weeks. That isn’t visible to the eye; rather, the eye is likely being influenced by other things–the biases of the industry/tradition, subjective preferences, etc. So the industry resisted, at least most of it, which allowed the A’s to keep on achieving what the industry, even as they did it, said could not be done.
I have a lot more to say about Moneyball, and KPIs, but I’ve run out of time. So, I’ll have to wrap this one up here, and throw it to you with a series of questions this week: what did you think about the state of the college address? What do you think about KPIs? What should our KPIs be? Have you read Moneyball? And what about Brad Pitt?
What do you think? What do you know? What can you prove?
The Harold Lounge 
Hi PhiloDave,
I was (and still am) very interested in your words. I’m waiting for other faculty members to give their two-cents before I go off on a discourse.
I sure do hate to see a baseball analogy go to waste (like the Cubs and Sox seasons) so I’m givin’ peeps until mid-week to reply. Then, you’ll hear from me.
What say you peeps?
I heard some concerned voices at Don’s address regarding KPI’s and PBF so go on and continue it on The Lounge. Tolle Blege!
Beer Here!
I know that community colleges can change student lives but that the change doesn’t look the same for every student. I think it would be great to measure whether and how students view society, nature, the world as more complex entities yet ones that they now have tools to understand after attending community college, but I don’t know how to prove it beyond self-assessment on the students’ part.
As promised PhiloDave, here’s my official reply:
Regarding the President’s address, I like how he was candid in not knowing much about what to say at a State of the College address and how he went into the archives for some historical perspective. Don’s doin’ his homework and the more he learns about our college and our culture, the more he understands us what we do and why we do it. I am impressed with his desire to learn and be the best president for our institution. You’re doin’ good Don!
Regarding the statistics, I believe the surveys were telling. I’d like to see the same survey for next semester to see if we will improve. For me, this was simply a baseline for the Laackman era at HWC. Next semester will give us results so we can begin to compare.
From Don’s speech, however, numbers do not tell the whole story. I was so, so intrigued how the numbers told a somewhat good story and how it permitted Don to give us a pat on the back. Yet it was the one non-survey, non-empirical, completely subjective, and totally personal story of ONE student that permitted Don to warn us about respecting our students. Mind you, I do not disagree with Don. I believe respect is key to our success. My observation was this: The qualitative feedback far outweighed the quantitative feedback.
Again, I do not disagree with Don. I have surveyed and assessed students; yet it is the conversations I have with them that really have a profound affect on what I do what I do at HWC. I think this is a major point that we should not overlook. The voice is mightier than the pen. The concerns of the individual are mightier than the group. We are all so uniquely different. Surveys can been perverted to reduce all to one generic meaningless group categories and numbers. With all the reinvention that’s goin’ on, we better not ignore student voices, and we best not assume that what goes on a survey will tell the complete story.
If anything, Don made a strong argument for listening directly to students and not listening to survey results.
Regarding KPI, baseball, and education:
First PhiloDave: I. Have. Not. Read. The. Book. Haven’t seen the movie either. I appreciate your analogy and I don’t disagree with your observations. However, from the outside of the stadium looking in, I shall share with you some immediate thoughts that concern me ONLY because corrupt minds in our academic culture have perverted our educational goals. I’m just thinkin’ ahead to what could be a problem if your analogy falls into the wrong hands. With all due respect to your post, here goes:
I would hesitate to endorse the analogy on the grounds that baseball is entertainment and education is not entertainment. I know you are pointing out the correlation between KPI and success. Your analogy gets the point across. However, I believe education should not be compared to entertainment. I know that is not what you were trying to do, so please know that here’s where I go in a different direction. Perhaps I will change my tune after reading the book.
Baseball and education have completely different goals and outcomes.
I am against the current fad to ‘corporatize’ the educational system. As I’ve mentioned before on The Lounge, I am tired of hearing the word ‘stakeholder’ used in academia. I’m also tired of ‘customer satisfaction’ being used on the first two floors of our building. This state of mind perverts the true purpose of education.
It is not our goal to fill the college to capacity to the point of having the equivalent of standing-room only crowds in our classrooms.
It is not our goal to graduate as many students as possible in the few amount of semesters as possible.
It is not our goal to ‘give’ as many A’s as possible.
Unfortunately, our funding sources are moving us in that direction. Why? Because they have a Moneyball mentality and they believe that whatever applies to corporate America or entertainment should also apply to our educational system. We are being reduced to these vulgar low common denominators of efficiency in the name of cost-cutting measures; not to better or transform our educational system, but to make the business of education a priority. I do not like it.
In baseball terminology, students are being reduced to a ticket stub filling a seat. If they buy a beer and a dog, then they are good fans. If they don’t buy a rally monkey, well, then perhaps we need to have the vendor wear a monkey suit in order to get the attention of little Johnny so he gets mom and dad to pop for the souvenir. Buy a monkey, and you are a preferred fan. Oh yeah, and the ballgame serves as a means to populate the seats. The ballgame is secondary, just like the purpose of education becomes secondary.
Unfortunately, The folks with the money are callin’ the shots, not the educators. That’s a sad state of affairs – to think that we’ve lost control over what we do best. Here’s what the moneyballers of education would do if they were baseball owners: Before investing in more brats and beer, they would want to make sure each game was a sellout AND that the vendors did all they could to make a profit, even if it meant wearing clown shoes or loud colored wigs. So long as the food and drinks sold, and a profit was turned, they would care less about the final score; the baseball game would become secondary. For the federal and local funders of schools, education has become secondary to every other money-saving (dare I say money-making?) scheme. Students are tertiary.
I’m not against a change to our current educational system. I’m not against looking at our college, the mission, and the success of our students. The system in general is broken. No argument from me. However, we should be talking about the PURPOSE of eduction first, then talk about how to measure performance. We, the educated are allowing the folks with the funding to tell us how we should educate. It’s the equivalent of letting Joe Baseball tell MLB owners what they should do. Would they put up with it? I think not. So why do we?
I’ll end with this. Although I’ve not read the book, I have read a baseball-related speech that I’d like to offer as a staring point. It was Ryne Sandberg’s Hall of Fame Induction Speech. In the speech he talks about respecting the game of baseball. He talks about playing the game for the right reasons. His focus was on baseball, not his batting average, not on trying to cheat his way to an MVP season with steroids. The nuances of the game mattered. The individual contribution to a team sport mattered. Each play was different. The gold gloves and the MVP followed. He showed respect for the game the same way we need to show respect for education. We need to recognize the individuals, not the surveyed and assessed whole.
If we build it right, and with a purpose, the students will come. Respect will guide our academic mission and Don will not have to remind us of our purpose at HWC. Then, and only then, should we talk about KPI’s; but I highly doubt that would be necessary. It would be secondary, perhaps tertiary.
I’d put less weight on the conversation with the student if it didn’t correspond to what we’ve seen in survey after survey of students, in their reviews of the school on Yelp, in TWO iterations, five years apart of an institution wide, random sample (CCSSE) and so on and so on.
Easy to dismiss one conversation; much harder when it fits a years long, multiple measure body of evidence.
As for the Moneyball, stuff, I have a post coming that picks up on some of the ideas that you start to explore and develops a few that I didn’t get to last week. Before that, though I want to clarify two things that are understandably unclear at this point:
1) What the name means: “Moneyball,” as the name says, is about seeking undervalued commodities. In my day, what I regard as the crucial aspects of run-generation, notably on-base percentage, were seriously undervalued, so “moneyball” consisted in finding batters with those skills.” Obviously from that definition, running our school through utilizing a philosophy of Moneyball is not the point. We do not seek out undervalued commodities; if anything, we ARE the undervalued commodity. If anything, this is one of the instructive points–players who understood that they were under valued on the basis of validated data analysis could make a case for themselves that they couldn’t make using the conventionally accepted and statistically suspect (or downright misleading) data points; and
2) Because much of the industry thought they knew how things worked and believed that they were ALREADY quite knowledgeable about determining quality playing from non-quality playing, based on measures which were a) very familiar and b) uncorrelated to their goal (winning games), most of the industry valued evidence that it shouldn’t have, discounted evidence that it should have been valuing, and/or generally denied that the methods employed could teach them anything about their aim because “baseball was different.”
Let’s take the hospital example. The purpose of a hospital is to heal, right? Not to make a profit–I’m sure every health worker would agree. Yet, hospitals do not exist in a vacuum, and so some (most) have to be concerned with their costs and charges. The methodology of data analysis that is employed in the creation and measurement of KPIs is not exclusively “Capitalistic” nor merely profit oriented (though it can be employed to serve those ends). It’s essentially scientific and empirical with the main goal being to isolate (as much as possible) and pay attention to the actions and behaviors and conditions that contribute the most to our goal.
You’re right to say that we need to focus on what our end is first (see the Mission and Vision), and I think, as Don said, one of the great things about HWC, and CCC in general, is the broadly shared sense of mission. I actually think we’ve got that part down pretty good. Where I think we’re less solid is in having a solid, shared understanding of how that end is reached and what behaviors, actions, and conditions contribute the most to it.
The question is do we know how our profession WORKS? If so, then it should be easy to find a few key behaviors or activities to monitor whose improvement leads to improved outcomes. Hospitals started using checklists before surgeries that included writing on the area to be operated on, which has greatly reduced the number of times that the wrong knee or wrist gets cut open. Sure, there is a contribution to the bottom line (and a small cost, too), but by identifying that key performance and measuring how often it is carried out, the practioners of healing do their job better AND have the means of showing it to patients, to funders, to lawmakers, and the rest. But it only matters if it matters
In other words, to go to your second to last point, the folks who are doing the funding/demanding KPIs are not (yet) telling us how to educate nor even how to measure our performance. They are merely asking if we know what we’re doing and whether we’re paying attention to our own practice. And if we can’t say what it is that we do that matters in educating students and/or have no idea how often we’re doing that thing, then I think they’re justified in wondering if we really know what the hell we’re doing. It is at THAT point that they are most likely to tell us how to educate, and what to measure; that is why this project is so important. This is OUR chance to say what we do and how we do it–this is our educational moment.
And if we don’t take it seriously both in planning and execution, that window will close and we may find ourselves on the outside looking in.
Understood PhiloDave,
The book is on the list to read (I don’t believe any movie ever does any book justice, so I’ll watch the movie with that precaution in mind).
I do put less weight on conversations when they do not correspond to surveys. So I was left wondering why, if our satisfaction rating was good, was the focus not there in Don’s speech? I agree that respect should be a priority. However, since it is not as easily quantifiable, we should wonder what the survey misses. I wonder if Don would have learned as much about us if he had simply administered a survey instead of conducting one-on-one interviews? My point is that information gathered from surveys does not tell the whole story and we should be mindful of this as we move forward and continue to survey and assess students. People are more than the statistics they represent.
“…we ARE the undervalued commodity.” Agreed. That needs to be rectified. The attack on CPS teachers (by our new mayor) is uncalled for. I’ll leave it that for now before I really go tangent.
“The methodology of data analysis that is employed in the creation and measurement of KPIs is not exclusively “Capitalistic” nor merely profit oriented (though it can be employed to serve those ends).” This is what we really have to be aware of as we move forward. For every dollar the funder gives, they want to know what they are getting in return. In a corporate world, yes, that is acceptable. However, we are not a corporation. We are not an assembly line. Our students are not washers and dryers that will give you x amount of wash and dry cycles before needing to be replaced. In my eyes, the funders of education are functioning as capitalists and that paradigm does not bode well for the purpose of eduction.
“…as Don said, one of the great things about HWC, and CCC in general, is the broadly shared sense of mission.” To paraphrase Sir Ken Robinson, we are an organic institution. Yet the funders/pseudo capitalists fail to see this as an inherit quality of our educational purpose. These funders want results and they want them now. That’s not how it works. (BTW, with the lack of job opportunities, is it fair to judge our performance on whether our students are employeed? Better to judge the institutions from which these lawmakers and politicos came to see if they ever took courses in ethics or moral awareness.)
I’m all in when it comes to knowing how our profession works. I’m all in when it comes to learning how to be a better educator. Unfortunately the ideals of our profession are trumped by the ideals of the capitalists. They want their returns on their investments and they want it now. That’s not how education works. It takes time to foster critical thinking (hell, some students only start to get it 3/4 of the way in to the semester and then we have to send them away), but the funders want answers now. They expect profitable miracles so they can report back to their stakeholders. That’s not how education works. (Wasn’t it Socrates that said we will only begin to learn at the age of 40?)
“In other words, to go to your second to last point, the folks who are doing the funding/demanding KPIs are not (yet) telling us how to educate nor even how to measure our performance.” Not yet is right. However they are the ones with the money and if they do not agree with what we say, they can modify our proposals. I look to the future and based on past practices and reputations, the needs of the funders appear to outweigh the needs of the students.
I do not want to be a voice of doom. I live in the real world. I understand your points. I agree with you PhiloDave. I’m in when it comes to telling the funders how we should be measured. I will cast all reputations aside and look at this as an opportunity to educate those who are asking for these KPI’s.
I want to be wrong in my forecasting. I want to do what is right. Let’s be that Oakland Team in the face of all the Steinbrenner-minded folks. I’d like to try-out for the second base position and wear No. 23.
Thanks for reading and your thoughtful feedback.