Monthly Archives: July 2017

Why we need comparison groups in PBF research

Tennessee began implementing performance-based funding in 2011 as part of its statewide effort to improve college completions. The figure below uses IPEDS completion data plotting the average number of bachelor’s degrees produced by each of the state’s 9 universities over time.

One could evaluate the policy by comparing outcomes in the pre-treatment years against those in the post-treatment years. Doing so would result in a statistically significant “impact,” where 320 more students graduated after the policy was in effect.

Pre Post Difference
Tennessee 1940 2260 320

This Interrupted Times Series approach was recently used in a report concluding that 380 more students (Table 24) graduated because of the policy. My IPEDS estimate and the one produced by the evaluation firm use different data, but are in the same ballpark.

Anyway, simply showing a slope changed after a certain point in time is not strong enough evidence to make causal claims. In very limited conditions can one make causal claims with this approach. But this is uncommon, making interrupted time series a pretty uncommon technique to see in policy research.

A better approach would add a comparison group (a la Comparative Interrupted Time Series or its extensions). If one were to do that, then they would compare trends in Tennessee to other universities in the U.S. The graph below does that just for illustrative purposes:

By adding a comparison group, we can see that the gains experienced in Tennessee were pretty much on par with trends occurring elsewhere in the U.S.:

Pre Post Difference
Tennessee 1940 2260 320
Other states 1612 1850 238
Difference 328 411 83

The difference-in-differences estimate, which is a more accurate measure of the policy’s impact, is 83. And if we run all this through a regression model, we can see if 83 is a significant difference between these groups.

It is not:

Using this more appropriate design would likely yield smaller impacts than those reported in the recent evaluation. And these small impacts likely wouldn’t be distinguishable from zero.

I wanted to share this brief back of the envelope illustration for two reasons. First, I am working on a project examining Tennessee’s PBF policy and the only “impact” we are seeing is in the community college sector (more certificates). We are not finding the same results in associate’s degree or bachelor’s degree production. Second, it gives me an opportunity to explain why research design matters in policy analysis. I don’t pretend to be a methodologist or economist; I am an applied education researcher trying my best to keep up with social science standards. Hopefully this quick post illustrates why that’s important.

June 30 FAFSA report

Below is a summary of high school FAFSA filing up to June 30 for the current and prior filing cycles.

June 30, 2016: 1,949,067
June 30, 2017: 2,128,524

This is a 9% boost in filing, or 179,457 more filers than last year!

You can download the raw data here or below.

We haven’t yet taken a close look at which high schools have shown the most growth, and I won’t pretend to know what these schools did to boost completions, but below is a quick look at the Top 20 in terms of largest raw number increase in FAFSAs. Way to go, Northside High School in Houston, TX, which saw the biggest jump in completions – going from 25 to 457!

State School Name City June 30 completions (16-17) June 30 completions (17-18) Change Percent Change
1 TX NORTHSIDE HIGH SCHOOL HOUSTON 25 457 432 1728%
2 PA PENN FOSTER HS SCRANTON 911 1213 302 33%
3 FL CYPRESS CREEK HIGH ORLANDO 295 499 204 69%
4 IL LINCOLN-WAY EAST HIGH SCHOOL FRANKFORT 327 529 202 62%
5 FL TIMBER CREEK HIGH ORLANDO 396 572 176 44%
6 TX ALLEN H S ALLEN 674 842 168 25%
7 NC ROLESVILLE HIGH ROLESVILLE 95 258 163 172%
8 FL WILLIAM R BOONE HIGH ORLANDO 304 467 163 54%
9 CA WARREN HIGH DOWNEY 556 710 154 28%
10 CA RANCHO VERDE HIGH MORENO VALLEY 465 614 149 32%
11 IL JONES COLLEGE PREP HIGH SCHOOL CHICAGO 226 374 148 65%
12 FL OLYMPIA HIGH ORLANDO 318 460 142 45%
13 FL FREEDOM HIGH ORLANDO 374 515 141 38%
14 NY NEW UTRECHT HIGH SCHOOL BROOKLYN 372 511 139 37%
15 NY BRENTWOOD HIGH SCHOOL BRENTWOOD 491 630 139 28%
16 TX THE WOODLANDS H S THE WOODLANDS 419 555 136 32%
17 PA PHILADELPHIA PERFORMING ARTS CS PHILADELPHIA 7 142 135 1929%
18 TX LOS FRESNOS H S LOS FRESNOS 341 476 135 40%
19 UT COPPER HILLS HIGH WEST JORDAN 256 390 134 52%
20 CA VALENCIA HIGH VALENCIA 318 449 131 41%

And in Wisconsin, here’s a list of the Top 20 schools in terms of raw growth in completions:

State School Name City June 30 completions (16-17) June 30 completions (17-18) Change Percent Change
1 WI KING INTERNATIONAL MILWAUKEE 201 278 77 38%
2 WI BADGER HIGH LAKE GENEVA 148 217 69 47%
3 WI OCONOMOWOC HIGH OCONOMOWOC 194 262 68 35%
4 WI SUN PRAIRIE HIGH SUN PRAIRIE 238 306 68 29%
5 WI CASE HIGH RACINE 160 227 67 42%
6 WI EAST HIGH APPLETON 170 235 65 38%
7 WI REAGAN COLLEGE PREPARATORY HIGH MILWAUKEE 206 269 63 31%
8 WI RIVERSIDE HIGH MILWAUKEE 183 243 60 33%
9 WI MIDDLETON HIGH MIDDLETON 239 296 57 24%
10 WI DE PERE HIGH DE PERE 180 235 55 31%
11 WI BAY PORT HIGH GREEN BAY 232 281 49 21%
12 WI FRANKLIN HIGH FRANKLIN 229 277 48 21%
13 WI NORTH HIGH WAUKESHA 119 166 47 39%
14 WI HAMILTON HIGH MILWAUKEE 128 174 46 36%
15 WI EAST HIGH MADISON 173 218 45 26%
16 WI CENTRAL HIGH SALEM 134 178 44 33%
17 WI CENTRAL HIGH LA CROSSE 128 171 43 34%
18 WI MUSKEGO HIGH MUSKEGO 220 263 43 20%
19 WI WAUNAKEE HIGH WAUNAKEE 154 193 39 25%
20 WI WEST HIGH WAUKESHA 149 188 39 26%

We will continue to analyze this data and plan to merge with other data sources to gain a better understanding of the variation that exists in filing rates.

We want to be sure to make this data available along the way, so please feel free to download and use the following high school and state-level data comparing the two cycles: FAFSA completions to June 30.xlsx

I wish I had time and resources to make this data more user-friendly and to share more widely. But until then, hopefully this good old fashioned Excel file is of use!