- The internal review process was labor intensive and the credibility of the review is validated by the fact that the district independently identified many of the same students that the Tennessee Department of Education (TDOE) inquired about.
- After four months of research, the district found no indication of systemic “gamesmanship” where students were being moved out of classes to avoid end-of-course (EOC) exams in order to inflate school performance data, which was the primary allegation made against the district.
- The district did identify a relatively small number of cases where students were removed from EOC courses and placed in credit recovery without having the opportunity to first attempt the second semester of the course in a traditional classroom. Most of these students did not take the EOC exam. However, these students accounted for only 0.1% of EOC exams administered during that particular school year.
- In response to the internal review findings, the district has revised and strengthened its course retake policy to give more specific guidance on when credit recovery should be used to help struggling students.
- Interim Director of Schools Chris Henson asked Chief Operating Officer Fred Carr to investigate the allegations being made against the district related to the use of credit recovery and EOC testing.
- The TDOE separately asked for information on more than 500 students in regard to EOC exam participation.
- The Chief Operating Officer is responsible for testing and accountability through the Department of Research, Assessment and Evaluation (RAE). The Chief Academic Officer is responsible for principal supervision.
- This structure is set up deliberately to separate the two and keep accountability independent from instructional leadership. One does not answer to the other.
- RAE worked for four months carefully analyzing student data from multiple sources to answer TDOE inquiries and internal questions about student class assignments and EOC participation.
- This was a time-intensive process that required many hundreds of staff hours. The result is an “Internal Review” report, as well as a spreadsheet and multiple pieces of documentation delivered to TDOE.
- In addition, RAE did a system-wide analysis to review school practices related to credit recovery and participation that have resulted in changes to policy and practice.
SUMMARY OF ANALYSIS AND FINDINGS
- There is no system-wide course code to identify credits earned in credit recovery, making it impossible to accurately identify the number of students taking credit recovery.
- Furthermore, there is little state guidance on use of credit recovery programs and great variation from district to district.
- In Metro Schools, there are inconsistencies in the way credit recovery is applied and and used.
- In several cases, it appears district policies and procedures were not followed, allowing 42 students to earn course credit through credit recovery without initially completing the second semester of the course in a traditional classroom.
- 42 students represent 0.1% of the 35,561 EOC exams administered in 2014-15.
- While there is one outlier among the district’s 24 high schools, deep and detailed analysis of EOC participation rates found no evidence of systemic gamesmanship or intentional manipulation of EOC data.
- With the exception of one school, all high schools had EOC errors in numbers small enough to be attributable to human error or complicated student mobility. At the outlier high school, the issue is being addressed with administration.
- The state calculates participation rate in a simple manner that counts the number of attempted tests against the number of answer documents returned. This counts all students who are enrolled in an EOC class on that day, but not all students who were enrolled in that EOC class at some point during the year.
- To account for this difference, RAE used several different methods to calculate participation rate. One included following a cohort of students through four years of high school and analyzing their EOC participation independent of the state’s calculation.
- This method helps account for students moving in and out of classes and taking tests at different times.
- It also takes into account legitimate exclusions, such as absent students, students in Advanced Placement or International Baccalaureate classes, certain English learner students and others.
- In both calculations, the state’s official calculation and the district’s more thorough calculation, participation rates were very high.
- State Calculation
- 98% overall in 2014-15
- At or above 97% in each subject for each of the past three years
- All zoned high schools exceed 95% annually
- District Cohort Calculation (including legitimate exclusions)
- Algebra I – 98.4%
- Algebra II – 96.4%
- English I – 98%
- English II – 96.8%
- English III – 93.1%
- Biology I – 99.3%
- In the case of English III, some of these students are scheduled to take the EOC this spring. Others missed the test last spring and were not required to take it over the summer because the state did not offer EOCs last summer.
- Given this exceptional circumstance, and in order to get a more typical view of Algebra II and English III participation at these schools, RAE applied the district calculation to the 2013-14 cohort of 11th graders.
- State Calculation
- Calculating ACT participation rate can also be done in multiple ways. To answer media inquiries, RAE was asked to calculate the rate as a percentage of 11th graders, which is typically the year students take the test.
- But in fact, TDOE has indicated that official participation rates, soon to be used in district accountability, will likely be based upon high school completers, not tied to a specific grade level. TDOE even encourages the use of vouchers to allow 12th graders to take the ACT if they miss it in the 11th grade.
- Obviously, this means vastly different rates can be produced depending on the method used. The district believes it should be calculated against high school completers, as the ACT is seen as a culminating activity for students.
- Using this method, the ACT participation rate for the class of 2014 was 94%. That number dropped to 87.4% in 2015 due to a snow storm that closed schools on the ACT test day. The make-up day was during spring break. This caused many students to miss the test entirely.
- In addition, a great deal of data quality issues exist in calculating ACT participation. ACT will often generate multiple records for one student or put students in the district file who are not enrolled in Metro Schools.
- Because of these issues, RAE is hesitant to generate more detailed participation data until TDOE finalizes the rules surrounding accountability for participation.
Tennessee Department of Education Inquiries
- Having TDOE provide a list of specific student names and cases for review was extremely helpful. Some cases could be easily explained by absences, mobility, discipline or other circumstances. Others could not.
- After reviewing cases and sending responses back to TDOE, Department officials still had questions about 86 students. Of these 86 cases, only four schools had more than 5 students on the list:
- Hunters Lane – 30
- Maplewood – 14
- Glencliff – 7
- Pearl-Cohn – 7
- Another TDOE inquiry centered on a relatively high percentage of 9th graders without English I EOC answer documents (20.2% in 2013-14 and 17.8% in 2014-15).
- Analysis of student course history shows that many of these students have been classified as 9th graders for more than one year because they have not earned the required number or type of credits to be on track for four-year graduation.
- In many other districts, students are not listed as “9th grade repeaters” regardless of credits earned. That is not the case in Metro Schools.
- So even though these students may be classified as 9th graders, most would have taken the English I EOC in a previous year and many were enrolled in English II or III.
- In addition, many of those 9th graders were English learners who are taking language development courses before English I.
- Taking these into consideration, course enrollment analysis shows that less than 3% of 9th graders during these years were not enrolled in an English class.
RECOMMENDATIONS AND NEXT STEPS
- Metro Schools takes test participation and academic intervention very seriously. While we do not feel there have been widespread, systemic issues in either area, it is clear that more monitoring needs to occur and new policies and procedures implemented.
- Credit recovery can provide good reinforcement of core academic concepts, and we stand by our use of it to help students earn credits in classes they previously failed.
- However, the TDOE’s current four year calculation for graduation rate puts increased pressure on students and teachers to earn credits, particularly in subjects that require four years of classes like math and English.
- We recommend the state adopt a five year calculation for graduation rate to give students more flexibility in retaking classes and earning credits.
- The new student information system through Infinite Campus will allow for course codes that specifically identify credit recovery. This will make reporting and oversight more robust.
- The course retake policy has been revised and strengthened to give greater guidance for principals, counselors and teachers on the best educational practices for when to use credit recovery to help struggling students.
- RAE will produce an EOC course enrollment file at the time of testing to help verify which students need to take EOC exams. This will be post-exam verification much more thorough than it is currently.
- RAE will also generate a new report to monitor EOC exam participation based on the more strict calculation method to share with district leadership.
- Based on the arduous data analysis work done by RAE staff to answer TDOE inquiries, a new report will be built in the Data Warehouse that allows for automatic generation of the same data analysis. This report will be available to school counselors and district staff for easy monitoring of student credits earned and EOC requirements.
- ACT participation is now part of district accountability to TDOE, though districts do not yet know how TDOE will calculate the rate. Once these business rules are finalized, RAE will refine its own reporting.
- RAE will continue working with school staff to identify students eligible to test.
- RAE will continue attempts to address ACT data quality concerns prior to calculating participation rate.
- We recommend that TDOE allow 12th graders to test on the one day each year set aside for statewide ACT testing.
TNReady and TCAP Tests
- RAE recommends using the same methods to monitor EOC participation that are detailed above and applying them also to TCAP and TNReady exams in grades 3-8.