REPORT: Internal Review of High School Test Participation

Key Highlights

  • The internal review process was labor intensive and the credibility of the review is validated by the fact that the district independently identified many of the same students that the Tennessee Department of Education (TDOE) inquired about.
  • After four months of research, the district found no indication of systemic “gamesmanship” where students were being moved out of classes to avoid end-of-course (EOC) exams in order to inflate school performance data, which was the primary allegation made against the district.
  • The district did identify a relatively small number of cases where students were removed from EOC courses and placed in credit recovery without having the opportunity to first attempt the second semester of the course in a traditional classroom.  Most of these students did not take the EOC exam. However, these students accounted for only 0.1% of EOC exams administered during that particular school year.
  • In response to the internal review findings, the district has revised and strengthened its course retake policy to give more specific guidance on when credit recovery should be used to help struggling students.

Review Process

  • Interim Director of Schools Chris Henson asked Chief Operating Officer Fred Carr to investigate the allegations being made against the district related to the use of credit recovery and EOC testing.
  • The TDOE separately asked for information on more than 500 students in regard to EOC exam participation.
  • The Chief Operating Officer is responsible for testing and accountability through the Department of Research, Assessment and Evaluation (RAE). The Chief Academic Officer is responsible for principal supervision.
    • This structure is set up deliberately to separate the two and keep accountability independent from instructional leadership. One does not answer to the other.
  • RAE worked for four months carefully analyzing student data from multiple sources to answer TDOE inquiries and internal questions about student class assignments and EOC participation.
  • This was a time-intensive process that required many hundreds of staff  hours. The result is an “Internal Review” report, as well as a spreadsheet and multiple pieces of documentation delivered to TDOE.
  • In addition, RAE did a system-wide analysis to review school practices related to credit recovery and participation that have resulted in changes to policy and practice.

SUMMARY OF ANALYSIS AND FINDINGS

 Credit Recovery

  • There is no system-wide course code to identify credits earned in credit recovery, making it impossible to accurately identify the number of students taking credit recovery.
  • Furthermore, there is little state guidance on use of credit recovery programs and great variation from district to district.
  • In Metro Schools, there are inconsistencies in the way credit recovery is applied and and used.
  • In several cases, it appears district policies and procedures were not followed, allowing 42 students to earn course credit through credit recovery without initially completing the second semester of the course in a traditional classroom.
    • 42 students represent 0.1% of the 35,561 EOC exams administered in 2014-15.

End-of-Course Exams

  • While there is one outlier among the district’s 24 high schools, deep and detailed analysis of EOC participation rates found no evidence of systemic gamesmanship or intentional manipulation of EOC data.
  • With the exception of one school, all high schools had EOC errors in numbers small enough to be attributable to human error or complicated student mobility.  At the outlier high school, the issue is being addressed with administration.
  • The state calculates participation rate in a simple manner that counts the number of attempted tests against the number of answer documents returned. This counts all students who are enrolled in an EOC class on that day, but not all students who were enrolled in that EOC class at some point during the year.
  • To account for this difference, RAE used several different methods to calculate participation rate. One included following a cohort of students through four years of high school and analyzing their EOC participation independent of the state’s calculation.
    • This method helps account for students moving in and out of classes and taking tests at different times.
    • It also takes into account legitimate exclusions, such as absent students, students in Advanced Placement or International Baccalaureate classes, certain English learner students and others.
  • In both calculations, the state’s official calculation and the district’s more thorough calculation, participation rates were very high.
    • State Calculation
      • 98% overall in 2014-15
      • At or above 97% in each subject for each of the past three years
      • All zoned high schools exceed 95% annually
    • District Cohort Calculation (including legitimate exclusions)
      • Algebra I – 98.4%
      • Algebra II – 96.4%
      • English I – 98%
      • English II – 96.8%
      • English III – 93.1%
      • Biology I – 99.3%
    • In the case of English III, some of these students are scheduled to take the EOC this spring.  Others missed the test last spring and were not required to take it over the summer because the state did not offer EOCs last summer.
    • Given this exceptional circumstance, and in order to get a more typical view of Algebra II and English III participation at these schools, RAE applied the district calculation to the 2013-14 cohort of 11th graders.
      • In this analysis, Hunters Lane High School did have an unusually high number of students (21) with no EOC record in Algebra II. All other schools had 7 or fewer.
      • Hunters Lane also had an unusually high number of students (27) with no EOC record in English III. All other schools had 3 or fewer.
      • The majority of these Hunters Lane students were moved out of the course during the second semester and into credit recovery.

ACT Participation

  • Calculating ACT participation rate can also be done in multiple ways. To answer media inquiries, RAE was asked to calculate the rate as a percentage of 11th graders, which is typically the year students take the test.
  • But in fact, TDOE has indicated that official participation rates, soon to be used in district accountability, will likely be based upon high school completers, not tied to a specific grade level. TDOE even encourages the use of vouchers to allow 12th graders to take the ACT if they miss it in the 11th grade.
  • Obviously, this means vastly different rates can be produced depending on the method used. The district believes it should be calculated against high school completers, as the ACT is seen as a culminating activity for students.
  • Using this method, the ACT participation rate for the class of 2014 was 94%. That number dropped to 87.4% in 2015 due to a snow storm that closed schools on the ACT test day. The make-up day was during spring break. This caused many students to miss the test entirely.
  • In addition, a great deal of data quality issues exist in calculating ACT participation. ACT will often generate multiple records for one student or put students in the district file who are not enrolled in Metro Schools.
  • Because of these issues, RAE is hesitant to generate more detailed participation data until TDOE finalizes the rules surrounding accountability for participation.

Tennessee Department of Education Inquiries

  • Having TDOE provide a list of specific student names and cases for review was extremely helpful. Some cases could be easily explained by absences, mobility, discipline or other circumstances. Others could not.
  • After reviewing cases and sending responses back to TDOE, Department officials still had questions about 86 students. Of these 86 cases, only four schools had more than 5 students on the list:
    • Hunters Lane – 30
    • Maplewood – 14
    • Glencliff – 7
    • Pearl-Cohn – 7
  • Another TDOE inquiry centered on a relatively high percentage of 9th graders without English I EOC answer documents (20.2% in 2013-14 and 17.8% in 2014-15).
  • Analysis of student course history shows that many of these students have been classified as 9th graders for more than one year because they have not earned the required number or type of credits to be on track for four-year graduation.
    • In many other districts, students are not listed as “9th grade repeaters” regardless of credits earned. That is not the case in Metro Schools.
  • So even though these students may be classified as 9th graders, most would have taken the English I EOC in a previous year and many were enrolled in English II or III.
  • In addition, many of those 9th graders were English learners who are taking language development courses before English I.
  • Taking these into consideration, course enrollment analysis shows that less than 3% of 9th graders during these years were not enrolled in an English class.

RECOMMENDATIONS AND NEXT STEPS

  • Metro Schools takes test participation and academic intervention very seriously. While we do not feel there have been widespread, systemic issues in either area, it is clear that more monitoring needs to occur and new policies and procedures implemented.

Credit Recovery

  • Credit recovery can provide good reinforcement of core academic concepts, and we stand by our use of it to help students earn credits in classes they previously failed.
  • However, the TDOE’s current four year calculation for graduation rate puts increased pressure on students and teachers to earn credits, particularly in subjects that require four years of classes like math and English.
  • We recommend the state adopt a five year calculation for graduation rate to give students more flexibility in retaking classes and earning credits.
  • The new student information system through Infinite Campus will allow for course codes that specifically identify credit recovery. This will make reporting and oversight more robust.
  • The course retake policy has been revised and strengthened to give greater guidance for principals, counselors and teachers on the best educational practices for when to use credit recovery to help struggling students.

End-of-Course Exams

  • RAE will produce an EOC course enrollment file at the time of testing to help verify which students need to take EOC exams. This will be post-exam verification much more thorough than it is currently.
  • RAE will also generate a new report to monitor EOC exam participation based on the more strict calculation method to share with district leadership.
  • Based on the arduous data analysis work done by RAE staff to answer TDOE inquiries, a new report will be built in the Data Warehouse that allows for automatic generation of the same data analysis. This report will be available to school counselors and district staff for easy monitoring of student credits earned and EOC requirements.

ACT Participation

  • ACT participation is now part of district accountability to TDOE, though districts do not yet know how TDOE will calculate the rate. Once these business rules are finalized, RAE will refine its own reporting.
  • RAE will continue working with school staff to identify students eligible to test.
  • RAE will continue attempts to address ACT data quality concerns prior to calculating participation rate.
  • We recommend that TDOE allow 12th graders to test on the one day each year set aside for statewide ACT testing.

TNReady and TCAP Tests

  • RAE recommends using the same methods to monitor EOC participation that are detailed above and applying them also to TCAP and TNReady exams in grades 3-8.

Downloads

Mayor announces $6 million capital plan for student technology

Mayor Karl Dean will file legislation with the Metro Council for $6 million in capital funding to purchase student technology needed for PARCC testing.

From the Mayor’s Office:

Mayor Announces Investment of City Capital Funds to Purchase Technology for Metro Schools

New Computers Would be Purchased With $6 Million Capital Plan

NASHVILLE, Tenn. – Mayor Karl Dean today announced that legislation has been filed with the Metro Council to use city capital funds to purchase $6 million of technology for Metro Schools, which would fund computers to prepare students for Common Core State Standards and taking new state-required standardized tests.

“Education has been and remains my top priority, and as a supporter of the more rigorous Common Core State Standards and the testing that accompanies them, I am dedicated to ensuring that our students have the tools they need to succeed,” Mayor Dean said. “Using city capital funds will also allow Metro Schools’ reserve fund to remain at its current healthy level, which is critical in the event that an unforeseen financial emergency should occur. I appreciate the collaborative effort that Metro Schools has made to reach this agreement and help ensure that we are providing our students with the resources they need while still exercising fiscal restraint.”

After discussions between the city and Metro Schools, both agreed a $6 million capital plan would allow Metro Schools to purchase computers and computer carts for students to prepare for Common Core and take the new tests. Metro Schools plans to use existing resources in its operating budget to cover teacher training. The Metro School Board initially voted to use $14.8 million from reserve funds to pay for new technology, teacher training and other items. This new capital plan allows Metro Schools to meet its testing needs without tapping into reserve funds.

Common Core State Standards (CCSS) are more rigorous educational standards to ensure students graduate high school prepared for college or career. In many states that are implementing Common Core, students in grades 3 to 11 take new state-mandated tests, called Partnership for Assessment of Readiness for College and Careers (PARCC). In Tennessee, students will take PARCC tests for the first time next school year.

“On behalf of Metro Schools, I want to thank Mayor Dean for proposing this agreement to our technology funding request,” said Jesse Register, director of Metro Schools. “Now, we should be able to fully implement our technology plan by the time school starts next year with this solution. We are fortunate to have a Mayor who is willing to use city resources to support the needs of our students.”

Today’s substitute legislation amends a $15 million capital spending plan for heavy equipment that was filed a week ago by adding the $6 million technology purchase.

A letter to parents: So you’re a Focus School. What does that really mean, anyway?

UPDATE: The Tennessee Department of Education has awarded Focus School grants to Amqui, Carter Lawrence, and Ruby Major Elementary Schools, as well as Pearl-Cohn Entertainment Magnet High School. These grants are worth anywhere from $100,000-300,000 and will be used to help close achievement gaps at these schools.

To learn more about these grants, visit the DOE website.


Dear Parents,

When the Tennessee Department of Education released the list of “Focus Schools” with a few MNPS schools on it, we heard from parents right away. There was some confusion and more than a little concern. Focus is a new label with a new definition, and it’s not immediately clear what it means. Some assumed it was a replacement for the old label of a High Priority or “Failing” school.

That’s not the case. The state defines Focus Schools like this:

“The 10 percent of schools in the state with the largest achievement gaps between groups of students, such as racial and ethnic groups, students from economically disadvantaged backgrounds, students with disabilities and English-language learners.”

The achievement gap is a key concern of Tennessee’s new accountability system for schools and districts. MNPS wants to see all groups of students achieving at high levels, and we are making progress across the district.

Here’s what it can mean to be a Focus School:

  1. The school has high achieving students.
    The state’s own guidelines say, “Schools on the Focus list are not necessarily there because of low achievement. In fact, many showed excellent growth last year.” If your school saw big gains in all students – including those at the very top of the honor rolls – that’s a wonderful thing. But, it also means the gap between the top students and everyone else didn’t get any smaller. We need to increase all students’ achievement and close the achievement gap at the same time.
  2. The school is diverse.
    Our schools have many students with different backgrounds, different home lives, and different abilities, and they’re all held to the same standard. Some students have a great balance of support systems at home and at school to help. Some don’t. By giving schools this label, the state is asking that we “focus” part of our attention on these schools to increase achievement for all students of all abilities. That’s just what we’ll do.
  3. The school is eligible for additional financial help to close the gap.
    Again, from the state guidelines:

“Focus Schools will be eligible to apply for grants aimed at dramatically closing the achievement gap. Schools not awarded a competitive grant will be provided state resources to close their achievement gaps.”

The school labels under the new system are much more accurate and provide a fuller picture of a school than the old No Child Left Behind labels. The new labels place the focus on increasing academic performance for our schools’ highest achievers as well as on those who need the most help, so everyone achieves more.

Sincerely,

Your Metro School

ACT Now! Our plan to raise ACT scores across the district

Every high school student in Tennessee is required totake the ACT. Tennessee is one of just a handful of states with this requirement and while that is not always good for our public image – it perhaps puts statewide test scores below other states where only the best and brightest take the test – it is very good for our students. Some students who might not otherwise take the test are required to, giving them an additional college credential or even a HOPE scholarship.

We want all our high school students to score a bare minimum 21 on the ACT. Of course, we prepare them to reach much higher than that, but we set the minimum at 21 because that is when scholarships and financial aid to state schools are within reach. With a 21, many students can afford college.

This year’s ACT results for our district show we still have a long way to go, but we’re making progress.  In fact, we’ve advanced our scores at a faster rate than the rest of the state.

But it’s still not good enough. We need better results in a shorter amount of time. We need results like the ones at Hillwood High School.

In the short time since Hillwood principal Dr. Steve Chauncy started his ACT Now program, the percentage of his students scoring 21 or higher has risen nearly 10%. When you consider statewide gains are measured in tenths of a percent, that is a huge jump.

How did Hillwood do it? And how will Dr. Chauncy and the Hillwood team spread similar results district-wide?

  1. Practice, practice, practice
    Before students take the actual ACT in their junior year, they sit for up to six practice tests – one as freshmen, the PLAN test, two practice assessments as sophomores, and two as juniors. Those tests are as real as they come, often actual ACT exams from previous years given through a program called Learning Express. By the time they sit for the real thing, they know exactly what to expect.
  2. Targeted instruction
    Rather than teaching students how to take a test, data from these practice tests is used for targeted instruction in the classroom. Areas of need are identified for individual students, and lessons can be are tailored to those needs. That starts right away. Freshmen come into Hillwood carrying data from the EXPLORE test taken in middle school. That data is used to start building ACT college skills in freshmen seminar classes.
  3. Critical Thinking Classes
    All Hillwood sophomores take a Critical Thinking and Imaginative Writing course where they learn how to apply their knowledge to problem solving and real world scenarios. That helps work through tough questions like those on the ACT and the even tougher questions students will later address in college and in their careers.
  4. Adaptable Solutions for All Schools
    This program has been shared across the district to all zoned high schools. But not every high school is the same, so it is can be easily molded to fit a particular school’s needs, resources, and abilities. As schools find improvements, they will be incorporated into the ACT Now program so the district is in a cycle of continuous improvement.

Development for this program is far from over. We are going even deeper into the data, looking at individual questions on practice tests to identify the academic support students need and we are providing afterschool tutoring for juniors with support from HCA.

The program began at Hillwood in the fall of 2010 and is spreading so all our high schools see rapid growth and all students are college ready and eligible for scholarship dollars.

It’s Official: Metro Nashville Public Schools rates “Intermediate” status

District meets seven of nine benchmarks; student achievement grows

Metro Nashville Public Schools showed growth in achievement among all subgroups of students last year, placing the district in intermediate status – the second highest accountability category. The State Department of Education earlier today released district-level status for all Tennessee districts, including for Metro Nashville Public Schools.

Under this new accountability framework, the top-performing districts are “Exemplary” while the bottom performing districts are in two “In Need of Improvement” categories; the remaining districts are in an intermediate category. Tennessee’s new accountability system replaces No Child Left Behind’s Annual Yearly Progress measures. Rather than expecting all districts to meet the same benchmarks year after year, the new system acknowledges that districts are starting from different places and rewards those that show the most growth. Under the new system, approximately 43% of districts were categorized as “In Need of Improvement” or “In Need of Subgroup Improvement.”

“These results show that thousands more of our students are performing at a higher level,” said Director of Schools Dr. Jesse Register. “Tennessee standards are among the highest in the country and this new accountability system is real, it is holding districts to standards that are difficult but attainable.

“The growth we have seen this year is the result of a lot of hard work, of making changes to instructional practice, providing more professional development and meeting our students’ diverse needs. We want to accelerate that growth at all levels and close achievement gaps.”

The accountability system, adopted after Tennessee secured a waiver from part of NCLB earlier this year, looks to districts to increase achievement levels for all students and reduce achievement gaps that exist between certain groups. Metro Schools serves approximately one-third of the state’s English Learner students, as well approximately 12,000 Students with Disabilities. The district also serves more than 56,000 economically disadvantaged students, which is about 71% of total enrollment for last year.

Metro Schools’ students made significant academic progress in the 2011-12 school year and the district met the majority of Tennessee’s academic achievement targets. In 2010, Tennessee adopted new academic performance standards that are among the most demanding in the nation. Metro Schools have shown steady improvement against these higher standards.

Grade 3-8 TCAP Tests
% proficient/advanced
  2010 2011 2012 
Reading/Language Arts 33.9% 38.9% 42.1% 
Math 25.9% 32.2% 38.4%
Science 36.2% 38.8%  44.5%
High School End of Course Exams
% proficient/advanced
  2010 2011  2012 
Algebra I 28.7% 37.6% 41.8%
English II 47.4% 46.6%  48.7% 

In addition, the district made impressive improvement at every grade level on the TCAP writing assessment with nine out of ten middle and high school students scoring competent or better.

Writing Assessment
Competent or Higher
  2010 2011  2012 
Grade 5  77% 77% 81% 
Grade 8 86% 86% 90%
Grade 11 90% 88%  91%

With this new system, Metro Schools met seven of nine benchmarks. The district showed improvement but narrowly missed the set benchmark for third grade reading/language arts. The district also missed in the graduation rate, the one calculation that has a one-year lag. The state has changed its calculation of graduation rate from a 5-year calculation to a 4-year calculation. Many students served by Metro Schools, such as English Learners or Students with Disabilities, require five years to graduate and are not included in the district’s overall graduation rate. Last year, Metro Schools’ graduation rate was 76.2%, down from 82.9% under the 5-year calculation. The 2012 graduation rate is not available.

Under Tennessee’s new accountability framework, the top-performing districts are exemplary while the bottom performing districts are in two in need of improvement categories; the remaining districts are in an intermediate category.

“I appreciate the support of our Board of Education as we transform our district. We knew test scores would be the last thing to change and we are pleased to see this growth,” said Register. “We must continue to improve in all categories. Reducing the achievement gap among student groups is our most significant challenge.”

Individual student reports are in transit to the schools where students are assigned for the 2012-13 school year. Each school will send the reports home to families. The State of Tennessee has not yet released school-level data.