Focus on Graduation Rates

It seems you can find out just about anything you want to know about schools in New York — at least when it comes to searching for data.

The state provides information collected from the school districts on everything from attendance to suspensions to dropouts to graduation to test scores. You can find out the demographic breakdown of the student body and how many students come from families poor enough to qualify for free or reduced school lunches. Now, with the federal No Child Left Behind requirements, the state goes beyond reporting aggregate test scores to give the pass rates for subgroups, including low-income and minority students.

In 2005-06, the state produced a 36-page report card and an 8-page information report for each of 700 districts in the state. That’s not to mention all the other data available on school districts, including financial data and information about the special education program. In the 12 years since I started working with this information, it’s become increasingly detailed and complex. I can’t think of any other field (health, human services, juvenile justice) where so much data is publicly available.

Yet all the numbers often don’t tell us what we want. Or they don’t give the full picture.

Take graduation rates as an example. They’re obviously a key statistic. If you could only ask for one statistic about a school district, you might choose the graduation rate, considering it shows how well the district is doing in moving students across that all-important finish line.

Several years ago, New York began reporting what are called “cohort” graduation rates. Cohort rates are something that some other states are just starting to get their arms around. They measure success from the baseline of the 9th grade year, instead of just counting how many 12th graders graduate in a given year. This is important because school systems tend to lose many students in their 9th and 10th grade years. If you wait until 12th grade to count, you might paint an overly rosy picture.

North Carolina is one state that recently began reporting the cohort rate. For 2005-06, it was 68.1%. That was quite a contrast to the high 90s people had gotten used to seeing for the graduation rate, but it made more sense next to related statistics showing North Carolina was among the states with the most 16- to 19-year-olds lacking diplomas and not in school. The cohort rate was undoubtedly a more accurate portrayal of what’s happening.

New York’s cohort rate in 2005-06 was 66.7%. Our problem is that we can’t go back more than one year in the past to look at our trend. That’s because the state changed the way it defines the cohort between 2003-04 and 2004-05. It turns out that while the state was reporting graduation rates as cohort rates, it was taking out of the calculation students who left school during the first three years of high school. Only those who were still around in the fall of their fourth year were included. It kind of defeated the whole purpose of using a cohort rate, if you ask me.

So now we have a data series that looks like the chart below. Every school district in the chart experienced a marked decline in graduation rates between 2003-04 and 2004-05, but we know that’s an artifact of how the rate was calculated. It looks like graduation rates fell, but they really didn’t. We know that because in 2004-05 the state reported the numbers both ways. Each of these five districts had an essentially flat or, in one case, increasing graduation rate if you use the old way of counting.

Our story’s moral is that you have to be very clear what you’re looking at when you’re looking at educational data. All is not what it seems. States are getting more sophisticated and I think for the most part more honest in how they report the numbers. This may take another leap forward if more states move toward reporting progress using a “growth model.” This tracks students’ progress from grade to grade, rather than simply comparing this year’s fourth-graders to last year’s fourth-graders. That’s the simplest of explanations – it will undoubtedly get much more complicated when New York actually puts it to use. And then all us consumers of educational data will have to sharpen our pencils and wipe clean our glasses to look carefully at what we’re given.