How We Analyzed the Performance of Bureau of Indian Education Schools

Until now, most parents whose children attended schools funded by the federal Bureau of Indian Education couldn’t accurately compare the performance of their school to others.

But a comprehensive analysis by Stanford University’s Educational Opportunity Project, The Arizona Republic and ProPublica has filled that gap, providing families, educators and tribal education departments with detailed school performance information, often for the first time.

The BIE, a federal agency that funds more than 180 schools and dormitories for Native American students across the country, for years failed to comply with a major federal education law requiring school systems to set uniform standards for student learning and to publish detailed information on school performance.

And while the lack of school report cards has made comparing local options difficult, those looking to evaluate BIE schools nationally faced an additional challenge: BIE schools operate in 23 states and over the years have administered as many as two dozen different standardized assessments, depending on the states in which they were located. Scores from the various tests cannot be directly compared.

The analysis, which took more than a year to complete, offers an unprecedented glimpse into how BIE schools have performed for the past decade.It found that students in BIE-funded schools learned at a faster rate, but scored more than two grade levels below the national average on standardized assessments.

The Data We Used

Stanford researchers from the Educational Opportunity Project have been reporting nationally comparable school performance metrics since 2019. Their analysis uses the U.S. Department of Education’s EDFacts database, which reports schools’ student scores on standardized tests. Since states use different standardized tests, the scores in EDFacts are not directly comparable. However, a subset of students across the country all take the National Assessment of Education Progress exam — often called “The Nation’s Report Card.” The Stanford researchers use NAEP scores to standardize the state-specific scores in the EDFacts database, resulting in comprehensive and nationally comparable measures of school performance. Read more details about their methods here.

Before this collaboration, the researchers were unable to produce nationally comparable scores for BIE schools because they didn’t know which tests students at each school took. The Republic and ProPublica compiled this information through calls with states, schools and tribal education departments, which allowed the Stanford researchers to include BIE schools in their analysis.

The analysis included standardized test scores from the 2008-09 to the 2016-17 school year, covering a total of about 193,000 individual test scores from BIE students in grades 3 through 8. The analysis included 141 schools. It excluded high schools because at different schools students take standardized assessments in different grades. It also did not include scores from the school years ending in 2013, 2014, 2015 and 2018 because the BIE did not submit scores to the U.S. Department of Education in time for inclusion in its EDFacts database. Data from the 2018-19 school year was not available at the time of the analysis, and most students did not take standardized exams in 2020 because of the pandemic. Students living in BIE-funded dormitories, who typically attend nearby public school districts, were not counted as BIE students. Growth rates could only be calculated for 92 schools because of gaps or issues in the underlying test data.

Performance Metrics

The adjusted, nationally comparable test scores were used to calculate three metrics: average achievement, grade slope and cohort slope.

Average achievement is the overall performance on standardized tests by students attending the school. This attempts to measure what students know at a given point in time.

The grade slope, also called the learning rate or growth rate, tracks how much an individual class improves on the tests each year. For example, this could reflect how much a third grade class’s scores improve the next year, in fourth grade, compared to the national average. This measures how quickly students learn, and is considered a marker of school quality.

The cohort slope tracks the school’s overall change in its achievement scores per year. For instance, a school with a negative cohort slope could indicate that each year, students in the third grade fall further behind the national average, compared to the school’s third graders the year before.

Experts said of the three measures, the grade slope most directly reflects the actual performance of the schools. They said BIE students’ relatively low achievement scores, paired with above-average growth rates, suggest many students are academically behind schedule before they reach the third grade, when standardized testing begins. That creates an achievement gap that even the highest-performing schools would struggle to help students close. Experts suggested the gap could be due to

the lack of widespread access to quality preschool programs or out-of-school learning opportunities in many tribal communities.

Comparing BIE Schools’ Performance

To determine the BIE schools’ collective performance on each metric, relative to all schools nationally, the news organizations averaged the schools’ Stanford-standardized scores, weighted by the number of students at each school. This produced separate weighted averages for each of the three metrics. The Stanford-standardized metrics are created in such a way that they capture how each school differs from the nation as a whole.

In addition to comparing BIE schools to all schools in the country, we also compared the BIE schools’ performance to Native American students in other schools within the boundaries of the public school district where the BIE school is located, defined by the Department of Education. The gap in overall achievement between the two groups shrunk from BIE schools being more than two grade levels behind to 0.3 grade levels behind, while the BIE schools’ learning rate advantage increased slightly.