I reviewed the literature published by the Department for Education (DfE) on using Artificial Intelligence (AI) in education. This work on the shortcomings of using AI in education is drawn from the literature. It includes the research and guidance of the DfE, and opinions from teachers, SEN staff, parents, and pupils. The limitations identified are drawn from the literature as a whole, so they may be issues brought up by staff, students or parents, or other stakeholders who were consulted, but it may also be the findings of the DfE.

I identified and organised the consequences of using AI in education into categories by topic, so that policy work can address these issues.

Definitions

AI is a machine learning system that is used to make decisions within a given scope. It operates as a component of a system, as a set of programming techniques integrated with existing systems, or as a standalone model that humans can interface with through applications.

LLMs (Large Language Models) are standalone models that use the patterns identified in datasets to statistically manipulate natural language, as required by human users. LLMs are typically accessed through a web interface. Versions of LLMs can be developed, which are tailored to the needs and data of a specific person or organisation. LLMs are usually freely accessible, but larger organisations purchase Enterprise plans which cater to the needs of their institutions and provide additional controls over how data is saved and used.

Data privacy and handling, and intellectual property rights

AI training and data handling is opaque. Student data may be collected and used for training models but model providers do not provide clarity on how this data is collected, processed and used. Sensitive data, especially special category data (such as information about Special Educational Needs and Disabilities or SEND), could be shared improperly or stored indefinitely. Whilst sensitive data and special category data is of especial concern, children and staff's data being held and used poses a range of concerns. One concern is the infringement of the intellectual property rights of teaching staff, who are using LLMs to create lesson plans, worksheets and other resources. The regulatory frameworks which LLM-providers use change regularly, which further limits clarity for users on how their information will be stored and used. The work by staff, such as lesson plans, may well be used to inform future iterations of the model, and if staff are pushed to using LLMs to support their work, they have limited control over their intellectual property.

Parents have expressed concerns that future employers could access the data from the AI developer companies and this may negatively impact the employment opportunities of the children. It is not clear how this may occur and whether it would occur at all but this raises the more important question of how far the impacts of data privacy and IP concerns go.

Further research questions

• How will the data from students, teachers and staff be used to train the model?

• How accessible this data will be and to whom?

• What intellectual property rights will be protected and compromised/infringed?

• What adverse impacts could using AI in education have?

Solutions

Companies need to be transparent on their user data sharing protocols, use of training data, collection, storage, use, sharing and whether it will be used for wider non-education purposes or not.

If individual companies have different data policies, schools and government bodies will have to select the best one of them. This takes up more time. Each company will have different financing criteria for usage of the models, and so schools may select the company that they can afford and have the digital infrastructure for. Considering the digital divide between schools in the North and South of England, and across localities, this is problematic because it exacerbates inequalities.

Materials protected by copyright can only be used to train AI if permission is granted by the copyright holder or a statutory exception applies. Pupils and students own the intellectual property (IP) rights to original content they create. Schools must not allow or cause students' original work to be used to train GenAI models unless appropriate consent or exemption applies.

Cognitive development and skills

Concerns about overreliance on AI were prevalent among parents and pupils. Cognitive offloading hinders the development of cognitive skill. Using AI is shown to reduce critical thinking abilities amongst people of all ages, but it impacts young people especially severely because they have had less of a chance to practice. Skill loss occurs too due to a lack of practice.

Using AI will reduce and damage the ability of pupils to develop key social and technical skills, such as empathy, creativity, writing, reading, critical thinking, reasoning, self-expression and playfulness. Outsourcing cognitive tasks like essay writing formulating written answers compromises the knowledge and skill development by encouraging them to passively consume information.

Over-reliance on AI for tasks deemed "mundane" (like marking or writing reports) could hinder the development of teacher skills and professional judgement, especially for early career teachers.

Epistemic erosion

LLMs can produce inaccurate or unreliable content. This is often referred to as "hallucination," where the AI provides fake facts or made-up quotes that sound plausible but are incorrect. As the LLM combines inaccurate and accurate information from datasets, inaccurate information is produced and pollutes our resources, knowledge and resource infrastructures.

Reduced human interaction and embodied learning

Teachers feel that feedback is a social exercise as well as an academic exercise and removing this task from the teachers damages the relationship between the teacher and student. The deep subject knowledge, professional judgement, knowledge of students' specific barriers to learning, and the ability to deliver feedback effectively, of teachers cannot be replaced.

Further research

• Why do teachers and teaching staff matter? What role do parents play in a child's education? Why can AI not replace them?

• What is the probable consequence of replacing people with AI or other technology? (This study would have to be conducted without testing the hypothesis on children, because otherwise children would experience those consequences and it would be unethical.)

• What does a vision for human infrastructure in the education domain look like? How can we achieve this vision?

• What are the skills of a teacher and what is the necessary knowledge for teachers to gain and practice? Much of this is covered in the understanding and work on pedagogy and teacher training, but it warrants further examination with a view to contextualising it again, and fortifying the argument for human teachers and human-human relationships being the ones students (and staff) develop and value. AI should be examined explicitly as a replacement for people, in education, to identify the areas which are for people alone, and the areas of overlap.

• The things which humans and AI can both do are an interesting area to figure out where natural intelligence and skill have to be developed further, such that Artificial Intelligence is not the limit of our intelligence.

Attachment issues (human-AI)

Chatbots respond in a way that is intended to resemble humans and this can create attachments between students and the chatbot. Chatbots also result in students being less likely to ask for help and more reliant on AI than on humans, such as teachers and designated support staff. This creates a strong likelihood that safeguarding opportunities will be missed. Children using it may suffer from its inability to be genuinely sensitive, caring and empathetic.

Integrity of achievements and work

Academic malpractice is one of the most prominent concerns raised by educators. There is widespread concern that GenAI facilitates academic misconduct (plagiarism and cheating) in assignments and assessments. Instances of academic malpractice due to AI were anecdotally reported by educators.

The validity of existing assessment methods may be undermined by generative AI tools that can generate text.

AI detection tools are not yet trusted because they can easily be misused and circumvented. Detection tools could also produce 'false positives' which means students may be penalised for using AI, unfairly.

Teachers are wary of using GenAI for marking due to accuracy and performance concerns, and the high stakes involved for students' futures. There is concern that students may not accept results of AI-marked assessments. Marking exams using AI was identified as an unacceptable use case where AI error could negatively impact educational outcomes.

Digital and AI literacy

In DfE surveys, 76% of teachers and leaders reported not being confident in advising pupils about the appropriate use of AI tools. This is a major barrier to adopting AI in education, for teachers. Many teachers expressed a desire for increased support, such as training and guidance for how to use AI in education.

The digital divide and educational attainment gap will be exacerbated if pupils do not have the access to the necessary devices, stable internet or softwares. Different schools have different qualities of digital infrastructure available to them, and differing abilities to use it based on the levels of digital literacy amongst teachers and students. If the AI systems are to be rented by the schools, then the access levels for schools would depend on the paywalls and school budgets.

Displacement of human resources

Participants worried about job losses caused by the displacement of teachers by AI, particularly if AI is used to compensate for teacher shortages, leading to a lower standard of teaching.

Manipulation/bias in teaching

AI systems show bias because the AI model is trained on data sets that are not necessarily representative of the populations the AI is designed to serve. LLMs trained on internet data could encode and perpetuate biases, outdated attitudes, or unbalanced political perspectives. This narrows, skews and reduces the scope of knowledge that can be provided and cultivated in conjunction with these tools.

Concerns were raised that particular viewpoints or biases, including those within the curriculum, could become more entrenched in AI if government agendas determine the content used to optimise the tool.

The majority of participants wanted reassurance that AI outputs would be monitored by a human to ensure accuracy.

Trust in large technology companies was extremely limited, in the surveys conducted by DfE. Parents were concerned that BigTech companies are primarily profit-motivated and might exercise undue power within the education system. The possibility this has to cause harm to curriculum content is detailed in this section, Manipulation/bias in teaching. Technology companies lack transparency and accountability so government regulations are preferable to the technology companies self-regulating.

Further Directions

Future work should discuss the solutions to the limitations and problems identified and specify the work of each stakeholder in creating and implementing the chosen solution. The structural and technical difficulties in the implementation of these solutions can and should be addressed through specific resources. At the moment, such a system of resources does not exist. The suggestions for guidance for teachers and resources are generic and unspecific, which means they are not responsive to specific issues. The structure of resources and support provided by the DfE is not yet developed to be this concrete, specific and tailored, but this development would greatly improve the coherency of DfE systems, and improve educational outcomes in the short-, medium- and long-term.