Progress towards NAPLAN reform

The campaign to review and ultimately replace the NAPLAN assessment has achieved further gains in recent weeks, with the Education Council releasing a communique stating that state and territory education ministers had agreed that the 2018 NAPLAN test results will not be published on the MySchool website, with any further decisions to be taken at the council’s December meeting.

This announcement comes after the validity of the results, and their comparability not only to previous years but also within the 2018 cohort, was challenged in an independent analysis conducted by Dr Les Perelman, formerly of the Massachusetts Institute of Technology (MIT), and Dr Walter Haney, of Boston College.

A key concern was the lack of reliable methods for comparing the 80 per cent of the test completed by pen and paper (P&P), and the 20 per cent conducted as part of the NAPLAN online trial.

The report stated that “there has been no publication of studies that are reported to equate the two test modes and to demonstrate that the marks on the two test modes are comparable”.

The report further states that “the strategy employed by Australian Curriculum, Assessment and Reporting Authority (ACARA) for cross-mode design invalidates comparisons of the 2018 NAPLAN, both the CAT and P&P versions, with prior years’ NAPLAN tests” and that “online writing tasks are inherently incomparable with P&P tests”.

On the issue of comparisons, the report outlines reasons why results from different modes of assessment are often incomparable, the first being because handwritten tests require more attention to be paid to deciphering handwriting, reducing markers’ attention for common errors in grammar, punctuation, spelling and even logic.

Further, students with substantial word processing experience have been shown to receive higher marks in other computer-based writing assessments. This experience appears to also have been borne out in the 2018 NAPLAN test, with reports that year 9 students completing the computer-based test receiving notably higher average results that those who completed the pen and paper test.

Head of ACARA Rob Randall acknowledged in the media that this difference was likely the result of the test mode, and of students’ familiarity with computer-based writing allowing them better opportunities to review and edit their work that are not similarly available to students doing a paper test.

The Perelman/Haney report concludes that “the design and execution of the 2018 NAPLAN make is so flawed that its results are of very limited use to students, teachers, parents and schools”, and that “in sum, the 2018 NAPLAN results should be discarded”.

Federation welcomes the Ministerial Council’s decision to withhold the 2018 NAPLAN data from comparative publication, which would have had the potential to significantly disadvantage schools and students through the false comparison of otherwise incomparable data sets.

Federation has expressed many concerns about NAPLAN since its inception under former education minister Julia Gillard, and in the past 12 months there has been a particular focus in the campaign to review and replace NAPLAN with a more valid assessment of student abilities that reflects the broad curriculum available to students in schools.

In mid-2017, Federation campaigned against the introduction of computer-based marking, in which extended written tests were to be marked by software looking primarily for grammar, syntax, punctuation and spelling, but was unable to evaluate writing for meaning or suitability in response to writing prompts.

This campaign included a hotline, inviting members to report their issues with the technical trials of NAPLAN online being conducted by Pearson, the global education testing company, as well as a publication and speaking tour by Dr Perelman outlining the impossibility of using software to mark writing in a way that fostered improvements in students writing for meaning.

That campaign resulted in the majority of states and territories rejecting computer-based marking, even though they proceeded with NAPLAN online trials.

Federation will continue to pursue improvements to national testing, both as an issue of teacher workload, and of quality assessment and reporting of the curriculum in schools.