REPORT: Internal Review of High School Test Participation

Key Highlights

  • The internal review process was labor intensive and the credibility of the review is validated by the fact that the district independently identified many of the same students that the Tennessee Department of Education (TDOE) inquired about.
  • After four months of research, the district found no indication of systemic “gamesmanship” where students were being moved out of classes to avoid end-of-course (EOC) exams in order to inflate school performance data, which was the primary allegation made against the district.
  • The district did identify a relatively small number of cases where students were removed from EOC courses and placed in credit recovery without having the opportunity to first attempt the second semester of the course in a traditional classroom.  Most of these students did not take the EOC exam. However, these students accounted for only 0.1% of EOC exams administered during that particular school year.
  • In response to the internal review findings, the district has revised and strengthened its course retake policy to give more specific guidance on when credit recovery should be used to help struggling students.

Review Process

  • Interim Director of Schools Chris Henson asked Chief Operating Officer Fred Carr to investigate the allegations being made against the district related to the use of credit recovery and EOC testing.
  • The TDOE separately asked for information on more than 500 students in regard to EOC exam participation.
  • The Chief Operating Officer is responsible for testing and accountability through the Department of Research, Assessment and Evaluation (RAE). The Chief Academic Officer is responsible for principal supervision.
    • This structure is set up deliberately to separate the two and keep accountability independent from instructional leadership. One does not answer to the other.
  • RAE worked for four months carefully analyzing student data from multiple sources to answer TDOE inquiries and internal questions about student class assignments and EOC participation.
  • This was a time-intensive process that required many hundreds of staff  hours. The result is an “Internal Review” report, as well as a spreadsheet and multiple pieces of documentation delivered to TDOE.
  • In addition, RAE did a system-wide analysis to review school practices related to credit recovery and participation that have resulted in changes to policy and practice.

SUMMARY OF ANALYSIS AND FINDINGS

 Credit Recovery

  • There is no system-wide course code to identify credits earned in credit recovery, making it impossible to accurately identify the number of students taking credit recovery.
  • Furthermore, there is little state guidance on use of credit recovery programs and great variation from district to district.
  • In Metro Schools, there are inconsistencies in the way credit recovery is applied and and used.
  • In several cases, it appears district policies and procedures were not followed, allowing 42 students to earn course credit through credit recovery without initially completing the second semester of the course in a traditional classroom.
    • 42 students represent 0.1% of the 35,561 EOC exams administered in 2014-15.

End-of-Course Exams

  • While there is one outlier among the district’s 24 high schools, deep and detailed analysis of EOC participation rates found no evidence of systemic gamesmanship or intentional manipulation of EOC data.
  • With the exception of one school, all high schools had EOC errors in numbers small enough to be attributable to human error or complicated student mobility.  At the outlier high school, the issue is being addressed with administration.
  • The state calculates participation rate in a simple manner that counts the number of attempted tests against the number of answer documents returned. This counts all students who are enrolled in an EOC class on that day, but not all students who were enrolled in that EOC class at some point during the year.
  • To account for this difference, RAE used several different methods to calculate participation rate. One included following a cohort of students through four years of high school and analyzing their EOC participation independent of the state’s calculation.
    • This method helps account for students moving in and out of classes and taking tests at different times.
    • It also takes into account legitimate exclusions, such as absent students, students in Advanced Placement or International Baccalaureate classes, certain English learner students and others.
  • In both calculations, the state’s official calculation and the district’s more thorough calculation, participation rates were very high.
    • State Calculation
      • 98% overall in 2014-15
      • At or above 97% in each subject for each of the past three years
      • All zoned high schools exceed 95% annually
    • District Cohort Calculation (including legitimate exclusions)
      • Algebra I – 98.4%
      • Algebra II – 96.4%
      • English I – 98%
      • English II – 96.8%
      • English III – 93.1%
      • Biology I – 99.3%
    • In the case of English III, some of these students are scheduled to take the EOC this spring.  Others missed the test last spring and were not required to take it over the summer because the state did not offer EOCs last summer.
    • Given this exceptional circumstance, and in order to get a more typical view of Algebra II and English III participation at these schools, RAE applied the district calculation to the 2013-14 cohort of 11th graders.
      • In this analysis, Hunters Lane High School did have an unusually high number of students (21) with no EOC record in Algebra II. All other schools had 7 or fewer.
      • Hunters Lane also had an unusually high number of students (27) with no EOC record in English III. All other schools had 3 or fewer.
      • The majority of these Hunters Lane students were moved out of the course during the second semester and into credit recovery.

ACT Participation

  • Calculating ACT participation rate can also be done in multiple ways. To answer media inquiries, RAE was asked to calculate the rate as a percentage of 11th graders, which is typically the year students take the test.
  • But in fact, TDOE has indicated that official participation rates, soon to be used in district accountability, will likely be based upon high school completers, not tied to a specific grade level. TDOE even encourages the use of vouchers to allow 12th graders to take the ACT if they miss it in the 11th grade.
  • Obviously, this means vastly different rates can be produced depending on the method used. The district believes it should be calculated against high school completers, as the ACT is seen as a culminating activity for students.
  • Using this method, the ACT participation rate for the class of 2014 was 94%. That number dropped to 87.4% in 2015 due to a snow storm that closed schools on the ACT test day. The make-up day was during spring break. This caused many students to miss the test entirely.
  • In addition, a great deal of data quality issues exist in calculating ACT participation. ACT will often generate multiple records for one student or put students in the district file who are not enrolled in Metro Schools.
  • Because of these issues, RAE is hesitant to generate more detailed participation data until TDOE finalizes the rules surrounding accountability for participation.

Tennessee Department of Education Inquiries

  • Having TDOE provide a list of specific student names and cases for review was extremely helpful. Some cases could be easily explained by absences, mobility, discipline or other circumstances. Others could not.
  • After reviewing cases and sending responses back to TDOE, Department officials still had questions about 86 students. Of these 86 cases, only four schools had more than 5 students on the list:
    • Hunters Lane – 30
    • Maplewood – 14
    • Glencliff – 7
    • Pearl-Cohn – 7
  • Another TDOE inquiry centered on a relatively high percentage of 9th graders without English I EOC answer documents (20.2% in 2013-14 and 17.8% in 2014-15).
  • Analysis of student course history shows that many of these students have been classified as 9th graders for more than one year because they have not earned the required number or type of credits to be on track for four-year graduation.
    • In many other districts, students are not listed as “9th grade repeaters” regardless of credits earned. That is not the case in Metro Schools.
  • So even though these students may be classified as 9th graders, most would have taken the English I EOC in a previous year and many were enrolled in English II or III.
  • In addition, many of those 9th graders were English learners who are taking language development courses before English I.
  • Taking these into consideration, course enrollment analysis shows that less than 3% of 9th graders during these years were not enrolled in an English class.

RECOMMENDATIONS AND NEXT STEPS

  • Metro Schools takes test participation and academic intervention very seriously. While we do not feel there have been widespread, systemic issues in either area, it is clear that more monitoring needs to occur and new policies and procedures implemented.

Credit Recovery

  • Credit recovery can provide good reinforcement of core academic concepts, and we stand by our use of it to help students earn credits in classes they previously failed.
  • However, the TDOE’s current four year calculation for graduation rate puts increased pressure on students and teachers to earn credits, particularly in subjects that require four years of classes like math and English.
  • We recommend the state adopt a five year calculation for graduation rate to give students more flexibility in retaking classes and earning credits.
  • The new student information system through Infinite Campus will allow for course codes that specifically identify credit recovery. This will make reporting and oversight more robust.
  • The course retake policy has been revised and strengthened to give greater guidance for principals, counselors and teachers on the best educational practices for when to use credit recovery to help struggling students.

End-of-Course Exams

  • RAE will produce an EOC course enrollment file at the time of testing to help verify which students need to take EOC exams. This will be post-exam verification much more thorough than it is currently.
  • RAE will also generate a new report to monitor EOC exam participation based on the more strict calculation method to share with district leadership.
  • Based on the arduous data analysis work done by RAE staff to answer TDOE inquiries, a new report will be built in the Data Warehouse that allows for automatic generation of the same data analysis. This report will be available to school counselors and district staff for easy monitoring of student credits earned and EOC requirements.

ACT Participation

  • ACT participation is now part of district accountability to TDOE, though districts do not yet know how TDOE will calculate the rate. Once these business rules are finalized, RAE will refine its own reporting.
  • RAE will continue working with school staff to identify students eligible to test.
  • RAE will continue attempts to address ACT data quality concerns prior to calculating participation rate.
  • We recommend that TDOE allow 12th graders to test on the one day each year set aside for statewide ACT testing.

TNReady and TCAP Tests

  • RAE recommends using the same methods to monitor EOC participation that are detailed above and applying them also to TCAP and TNReady exams in grades 3-8.

Downloads

We need your input! Take the director of schools survey now

As you have probably heard, a renewed Director of Schools search is underway. The Board has worked hard to make this an inclusive process by engaging and leveraging leadership from the Mayor’s Office, education leaders, parents, faith leaders, civic leaders, private sector partners, the nonprofit community and others. A unified effort is essential as we work to put the city’s collective best foot forward to recruit and hire a terrific leader who can take our public schools boldly forward.

As a first step, the Board joined hands with the Mayor and the Nashville Public Education Foundation to convene a Search Advisory Committee to provide initial recommendations on the search process. Their job is not to identify candidates per se, but rather provide advice and counsel about what kind of leader we need, where we might find people with the strongest track record of success, and how we can best compete for top-tier candidates. With these recommendations as a starting point, the Board will commence a robust national recruitment effort.

As part of their work, the Search Advisory Committee has reviewed in depth the community feedback we received during the initial search effort earlier this year. For those of you who participated in the focus groups and community survey last summer, we are deeply appreciative as that has provided a strong foundation. The Committee has expressed an interest, however, in seeking some additional, more specific feedback from our educators and parents in particular.

We are hopeful you would take a few minutes to provide some additional insights. Anything you share in the survey below is strictly confidential. It is being hosted, collected and analyzed by the Nashville Public Education Foundation, an independent, non-profit community organization. You will not be asked for any identifying information (i.e. name, school, etc.).

Click here to complete the follow-up survey: www.nashvillepef.org/director-survey

Facts on district revenue and per-pupil spending

In past articles, we’ve shown you how much money we get and where it all goes.

Our community deserves an honest dialogue – without ideology and special interest talking points – about school funding that includes a fair assessment of needs, expenses, and revenue.

So let’s talk for a moment about where money comes from and under what circumstances we get more of it.

Where does revenue come from?

About a third of our operating budget comes from the State of Tennessee. They give us a certain amount of money per student and according to what kind of student. When more students enroll, our revenue from the state goes up, to the tune of about $3,200 per student.

The other two-thirds comes from the Metro Government, paid from county taxes and allocated by the Mayor’s Office and the Metro Council. The amount of money we get from the Metro Government is fixed. It is not tied to a per-student formula the way state money is.

This year our enrollment is up by nearly 1,800 students. About a third of our revenue (from the state) reflected that increase. The other two-thirds of our revenue (from the Metro Government) remained flat.

[NOTE: We received a $26 million increase in our operating budget from the 2012-13 school year to the 2013-14 school year. Of this $26 million, $12 million came from our own fund balance, $10 million came from the state and $4 million came from Metro Government.]

How does that money move around the district?

It’s a phrase people use a lot: per-pupil spending. Metro Schools spends around $9,100 per pupil in grades K-12. When a student moves from one school to another, the money theoretically follows him.

If the student leaves his neighborhood school and moves into a magnet school, he is moving from one district-operated school to another. The money attached to him stays in the same pot of district money and our per-pupil spending amount stays the same.

When a student leaves his neighborhood school – or a magnet school – for a charter school, the $9,100 moves to the charter management organization that runs the charter.

When a student leaves a district school for a private school, the money does not follow him to the private school. The district loses the state revenue associated with that student, but the private school does not get it. The funding from Metro Government is unchanged. The private school gets private money in the form of tuition.

The expenses of the neighborhood school don’t increase with every additional student and don’t decline with every student departure. The school is spending the same amount to keep its lights on and its hallways clean. The school still has teachers to pay and technology to buy and maintain.

Innovative practices that are designed to quickly improve achievement could face tough times, like extended learning time, model classrooms and certain kinds of school-level professional development.

What’s the solution?

Our strategic plan calls for more equitable use of resources, placing money where it’s needed most for fair service of students. We are already moving toward a model that puts more money directly in the hands of principals, allowing them to spend money in ways that will most directly benefit the unique needs of their students. That will help everyone decide best allocation of existing resources and ensure that money is spent in the most effective ways.

But what about revenue?

The answer will come from a community discussion of school finances. We must have an honest and open dialogue about revenue and expenses. The Board of Education began that conversation this fall, well in advance of the usual budget cycle.

This issue has wide-reaching effects, beyond just budgeting and buying textbooks. It can affect a school’s character and existence.

Let’s look at it at the individual school level. When a school is constantly threatened with drastic budget cuts or even closure, it has a more difficult time attracting students, which leads to further budget reductions.

It is a cycle that puts even more of a burden on teachers and principals to properly educate the students they already have.

School-based budgeting gives power to the principals

“Knowledge about the needs of students is greatest closest to the student… School leaders require the ability to make decisions based on their knowledge, expertise and professional discretion.”

In a world of principal autonomy and school-based decision making, what is left for the central organization of a modern school district? Where does it fit in, and what role does it have to play?

Here in Nashville, the role of central office is changing dramatically. The top-down management structure is disappearing. In its place is a knowledge and support organization designed to provide central services, study and share innovative practices, develop leaders and keep schools accountable. In fact, this change is already reflected in the district budget and in a pilot program working in 15 schools right now for school-based budgeting.

What is school-based budgeting, and what does that look like?

At these 15 schools, principals have direct control over $6,300 per student (on average), meaning they can spend that money as they see fit. That number is expected to increase over time. The rest of the money goes to central services like transportation, food services, human capital, textbooks, building services and more.

SEE the school’s individual budgets.

The idea is to bring powerful decision-making power right into the schools, where the most knowledge about individual students lives. Next year, this program is expected to expand to 50-60 schools and could go district-wide by 2015-16.

During that time the whole concept could go even further, putting 100% of per-pupil funding on school level budgets. That would greatly expand the level of flexibility and discretion given to each principal and ensure funding is distributed equitably based on individual student need. In that scenario, school leaders would “buy” central services from the district, and there would be certain non-negotiable services like the Board of Education.

This is a culture change, moving central office to a system of specialized support for schools and giving more decision-making power to principals.

Updated: Where does the money go?

UPDATE: Our comprehensive and award-winning budget book is now available for your review. At 228 pages, is contains all the financial facts you could want for fiscal year 2013-14.

DOWNLOAD the FY2013-14 Budget Book
(PDF – 33.8MB)

Contained within the budget book is a greatly expanded version of our original post, the 2012-13 School Year Expenditure Analysis. It gives an in-depth (50 page) look at all of the money spent from the Metro Schools operating budget and grants.

For the sake of convenience, we’ve broken it out into a separate document.

DOWNLOAD the SY2012-13 Budget Analysis
(PDF – 2.1MB)


PREVIOUSLY:

Let’s look at the facts of public education funding in Nashville.

Here is a breakdown of the 2012-13 operating budget:

  • $714 million – 2012-13 District Operating Budget
  • $441 million – Amount sent directly to district-run schools
  • $28.2 million  – Amount sent to charter schools
  • $163.3 – million – Amount spent on school-specific operations and instructional support services like transportation, maintenance, custodial and grounds, IT
  • $41.4 million – Amount spent on Board of Education, Human Capital, Finance and Purchasing, instructional departments and other central support functions
  • $39.2 million – Amount spent on retiree benefits, pension and debt service

When those are added together, they give a great deal more context:

  • Including charter schools, nearly 66% of resources serve schools directly.
  • Including school specific services like transportation and maintenance, more than 88% of MNPS expenditures flow to schools to provide and support instruction.

In chart form:

Directly to District-run Schools $441,107,943 61.7% All school-based staff such as teachers, principals, counselors, librarians, bookkeepers, etc.
Centrally Managed, School-based Services $163,348,961 22.9% Transportation, maintenance, custodial, utilities, textbooks, and itinerant social workers, psychologists, and other
Central Services $41,458,693 5.8% Board, Superintendent, Finance, Purchasing, HR, Leadership & Learning Supervision, other centralized services (e.g., most of IT/data warehouse)
Obligations $39,252,805 5.5% Pension, retiree insurance, debt service
Charter Schools $28,240,250 4.0% Pass-through to charter schools
Other $1,032,606 0.1% Pass-through expenses
GRAND TOTAL $714,441,258 100%

How implementing Common Core State Standards has changed my classroom

by Cicely Woodard, 8th grade math & Honors Algebra I teacher at Rose Park Middle Magnet

Since implementing Common Core State Standards in my middle school mathematics classroom, my role as teacher has changed.  No longer am I the ultimate source of knowledge in the classroom.  Instead I am a facilitator of student-centered, student-led learning.  No longer am I helping students to choose the right answer.  Now I am helping them to develop deep understanding of mathematics that has meaning to their lives.  My expectations are high, and everyday my students are rising to meet those expectations.

The role of my students has changed as well.  In the past, my students were doing math activities just to master concepts.  Now they engage in high level math tasks that require them to think critically about mathematical ideas.  Before my students listened to me explain math concepts and gave some explanations to me.  Now they are participating in deep discussions with their peers in which they must formulate their own understandings, justify their thinking, and critique the reasoning of others.  At one time, my students focused on choosing the right answer on a multiple choice test.  Now they concentrate on learning math that is relevant to their everyday lives and communicating their understanding of that math.

My students are becoming effective communicators and courageous problem solvers who are not afraid to tackle rigorous math tasks.  They have a sincere desire to be successful.  When I see them, I see young people who are one step closer to being ready for life after high school.  They truly are becoming prepared for college and the workforce.  I attribute all of these positive changes to the implementation of Common Core State Standards in my classroom.

Learn more about Common Core State Standards & PARCC Testing:

Opinion Piece: Banning all virtual schools isn’t the answer

What practical effects would come from revoking the Tennessee Virtual Schools Act? For one thing, it might actually decrease accountability for online education.

That’s the conclusion of Dr. Kecia Ray, head of Learning Technology in Metro Schools. She gave her thoughts about all the talk of banning virtual schools this week in The Tennessean.

Public virtual schools have the same accountability standards as brick-and-mortar schools. Under the current system, underperforming virtual schools are at risk of state takeover or even closure. Revoking the Tennessee Virtual Schools Act would actually allow schools to circumvent accountability by operating as virtual programs, not schools, with student achievement results sent to the student’s zoned school. This is not the transparency most desire.

Dr. Ray also gives a possible alternative to this drastic measure at the end of the piece.

To read her full opinion column, visit Tennessean.com.