Algorithmic Systems in Educational Assessment: Potential and Problems
By Kim Ochs*
Source: Akshay Chauhan (Unsplash)
Disclaimer: The inclusion of apps and services featured in this article does not reflect any product endorsement. Links to commercial websites are not affiliate links.
Algorithmic systems are changing educational practices and processes. This includes educational assessment of academic performance or student behaviour, and related functions such as exam proctoring and drop-out warning systems, which assist with the monitoring of students who are at risk of falling behind academically.
In the area of assessment or grading, TurnItIn, for example, provides tools for instructors to streamline manual grading. Instructors can save their feedback comments or phrases made on one student’s paper and then reuse them for multiple assignments, or leave voice comments instead of text comments. Other features make it easy to navigate multiple student submissions. The company is perhaps best known for its plagiarism detection software, which can check the originality of a student’s submitted work in a matter of minutes. The TurnitIn algorithm checks the strings of words found in the student’s paper against its repository and assigns the paper an originality score, devoid of human judgement. These functions aim to improve efficiency and free up the time of educators and administrators to focus on other work, such as class preparation. A newer offering is to coach students as they write to improve citations and check for similarity as they type. Some critics argue, however, that efficiency is being prioritised over learning and teaching.
Other algorithmic systems are being used to capture soft skills, which might even extend beyond the classroom. ClassDojo can be used by teachers to award points for behaviour in school, or by parents to recognise finishing homework, helping a sibling, or helping around the house. With the approach, they are also aiming to support building community.
Drop-out warning systems use machine learning algorithms to pre-emptively identify students who are at risk of quitting school before the successful completion of their examinations. A potential benefit is that the system could catch patterns humans could miss, or identify issues sooner, so the student could get assistance earlier. A potential harm, however, is if the system misses signals that educators would have caught or causes them to doubt their own judgment. For example, an experienced teacher might flag an issue, but decide not to report it since it was not caught by the system. Instead of trusting her own judgment, she might defer entirely to the algorithm. With these systems, it is also important to look at the underlying predictive modelling behind the algorithm. As some scholars have pointed out, as an example, “the inherent class imbalance could pose difficulty in building accurate predictive modeling for a dropout early warning system.” In other words, the algorithmic system could potentially introduce or reinforce an existing bias (which might be part of the system design itself).
A recent example that highlights the challenges of applying algorithmic system in educational assessment is the case of Ofqual in England. Following the mass-migration to home schooling in England during the pandemic, new approaches to assessment were taken in 2020 after school leaving examinations (General Certificate of Secondary Education (GCSE) and A-Levels) were cancelled. Ofqual, the regulator of examinations, texts and qualifications, produced a grade standardisation algorithm to predict students’ grades, with the intention of moderating teacher-predicated grades and combatting grade inflation. For each student and for every subject, teachers provided an estimated grade and a ranking of the student, which were put through an algorithm, which also factored into the calculation the school’s performance over the three prior years. Teacher-assessed grades were not taken into consideration, with 82% of “A-level” grades and 97% of GCSE grades were assigned by the algorithm. When the results were announced in August 2020, 36% of the A-Level grades were one grade lower than teachers’ assessments and 3% were down two grades.
As it was explained in a BBC article, the algorithm based on past school performance was problematic: “a bright student from an underperforming school was likely to have their results downgraded through no fault of their own. Likewise, a school which was in the process of rapid improvement would not have seen this progress reflected in results.”
As this case illustrates, a general problem issue with the use of algorithmic systems in education is a lack of transparency, or unknown, algorithmic decision-making process, and subsequent lack of accountability. If the parent or student wants to contest a decision, what is the process? Who is ultimately responsible for the decision – the teacher? the school? the vendor of the algorithmic system?
Following a massive public uproar, a decision was made that school leavers could use the teacher-assessed grades rather than the grade predicted by the algorithm, making it possible for students to re-apply to universities. leaving top-tier universities with a capacity problem.
In 2021, following another year of disrupted schooling, exams were cancelled once again. This year, however, no algorithm will be used in England. Teachers assessed grades will be used, and schools will be able to gather and use additional evidence such as mock exams, coursework, or essays to factor into teachers’ grading decisions. This example shows the importance of using algorithmic systems responsibly and equitably.
The tale about entrance exams in England might be a cautionary one, but it is not putting off investors. The Korean company Riiid, which developed test-prep apps for companies such as Kaplan, is developing an app to help Colombian students prepare for their national college entrance exam. As of April 2021, the company had raised US$70.2 million in venture capital. According to HolonIQ, US$16.1 billion of venture capital was invested in education technology globally in 2020, up from only US$500 million in 2010 (and increase of 32 times). The growing investment in the edtech sector suggests algorithmic systems and related technologies will continue to play a big role in education.
The Center for Democracy and Technology, a think-tank based in Washington, DC, offers a nine-step checklist on using algorithmic systems responsibly and equitably in education:
1. Consider the effectiveness and potential impacts of the system.
2. Govern and document appropriate uses of algorithmic systems.
3. Engage stakeholders early and throughout use.
4. Implement data governance.
5. Examine input data for bias.
6. Keep humans in the loop.
7. Conduct regular audits.
8. Create protocols for accountability and redress.
9. Ensure legal compliance.
Another recommended strategy is to pay attention to algorithmic hygiene, as described in a framework proposed by Brookings, to mitigate algorithmic bias. This means updating policies around equity and non-discrimination to apply to digital practices and outlining self-regulatory best practices.
As new companies emerge, and more innovation and investment go towards education technology, the debate about the potential benefits and harms of using algorithmic systems in education will continue. So too must the discussion around their responsible and equitable use. Algorithms are not only parts of digital technologies, but they are also social products.
*Kim Ochs has been active in the field of educational technology for more than a decade, spanning work in higher education, research, and start-ups, working with international organisations, NGOs, private companies, and edtech investors. Kim holds a doctorate in educational studies from the University of Oxford.