What Was Once Science Fiction Is Now Reality: Orwellian Uses of Safety Tech in K-12 Schools

Prior to releasing our survey research of students, parents, and teachers this past September, our team spent months working with an independent research firm to brainstorm questions that would thoughtfully reveal how schools are currently implementing various uses – including more extreme uses – of educational data and technology (edtech) aimed at keeping students safe. Anecdotally, we had heard of school districts utilizing predictive analytics, remote proctoring, facial recognition, law enforcement data sharing, weapon detection systems, and student location tracking – which all carry serious, documented risks. But we wanted to know: How common are these edtech tools, really?

To cover all our bases on potential “extreme” use cases, we originally included armed drones that could surveil school grounds for safety reasons and even respond to safety threats; however, we quickly decided to cut it since it seemed far too outlandish. Fast forward to August when our surveys were in the field, that assumption was proven wrong.

Philadelphia, a large urban school district, announced that it would be rolling out district-owned drones “to patrol violence-prone areas without the need for police on the ground.” In some cases, the drones would be piloted by students, but drone footage would be monitored (presumably) by a school safety official. This seemingly dystopian use of technology paired with the results of our survey research made one thing very clear: Invasive school safety tools are actively being implemented by school districts across the country, even ones we previously thought “too outlandish.”

Experimental, Potentially Harmful Safety Tools Are Being Used Regardless of Student, Parent Concerns

Uses of school safety technology largely driven by artificial intelligence (AI) are expanding in schools to respond to mass shootings, the youth mental health crisis, and other ever-present safety threats to staff and students. Even though our survey research shows there is not yet widespread adoption of some of the more invasive tools, schools still report sizable use of such tools to prevent safety issues, respond to safety issues, or involve law enforcement.

What is more alarming is that students and parents report high levels of concern about most of these more extreme use cases, but schools are still deploying them regardless. For example, 36 percent of teachers reported that student data is being analyzed to predict who would be more likely to commit a crime, act of violence, or act of self-harm, even though 69 percent of students and parents reported being extremely or somewhat concerned about that particular use. This shows a deep disconnect between schools, parents, and students in their priorities when it comes to edtech procurement decisions.

Teacher Q: Listed below are potential ways that data or technology could be used in schools. Which of the following is your school or school district doing today? Student & Parent Q: Listed below are potential ways that data or technology could be used in schools. How concerned would you say you are with each if they were used at your school?
	Teachers	Students	Parents
Prevent safety issues
Student data are being analyzed to predict which individual students would be more likely to commit a crime, commit an act of violence, commit an act of self-harm, etc.	36%	69%	69%
Monitoring what students post publicly on their personal social media accounts	37%	71%	68%
Using cameras with facial recognition technology to check who should be allowed to enter a school building or someone who should not be there	33%	55%	58%
Respond to safety issues
Tracking students’ physical location through their phones, school-provided devices like laptops, or digital “hall passes” when they leave the classroom	36%	74%	71%
Gunshot detection system on school property	27%	45%	55%
Using cameras that use artificial intelligence to notice unusual or irregular physical movements, which could identify an emergency or critical event at the school	31%	58%	60%
Involve law enforcement
Student data such as grades, attendance, and discipline information are being shared with law enforcement	38%	65%	66%

Table showing the high rates of concern among students and parents about specific uses of school safety technology (e.g. a gunshot detection system), and the sizeable rates of their use in schools as reported by teachers.

Why Should Education Leaders And Policymakers Be Concerned About These Uses?

As previous CDT research has affirmed, the use of technology in the name of student safety presents significant risks such as:

• Lack of efficacy and accuracy: Many tools used to prevent or respond to safety issues lack evidence that they actually live up to their stated intent – to keep students safe. They can be subject to technical limitations, difficult to audit, and produce false positives, which could lead to students experiencing excessive, unsubstantiated disciplinary action or interaction with law enforcement. Additionally, some safety tools are not designed specifically for the school context, making them potentially unequipped to handle the nuances and highly sensitive nature of student data.
• Chilling effects: Having various invasive safety technology tools as a regular part of a student’s learning environment can actually cause students to feel less safe in the classroom. Excessive monitoring and surveillance can chill speech, associations, movement, and access to vital resources, posing serious risks to students’ privacy, free expression, and ability to learn. 
• Disproportionate impact: Safety tools driven largely by AI, like student activity monitoring and predictive analytics, are proven to cause disproportionate negative privacy and equity harms to protected classes of students on the basis of race, sex, and disability status. Not to mention, algorithmic risk assessment and facial recognition tools are often trained on biased data that lack social nuance factors. This could cause students of color to be overly identified in especially high-stakes safety situations, thus subjecting them to increased scrutiny and unwarranted encounters with law enforcement.
• Cost and resource management: Schools often lack resources, expertise, and personnel to effectively monitor and measure the impact of school safety tools, leading them to overly rely on the data gathered and decisions made by a machine. Again, this is particularly concerning given the lack of evidence that these tools are actually effective.
• Governance mechanisms: Often school safety technology is procured out of fear of imminent threats to students and staff, without clear policies and procedures on how these systems and student data will be used and governed. There is also no uniform practice of parent, student, and broader community engagement in the procurement process of these tools, raising concerns about transparency and centering concrete student needs.
• Cybersecurity: Related to governance issues, schools having more tools to collect sensitive student data introduces more risk for cyberattacks and data breaches, especially with many schools lacking sufficient IT/privacy personnel.

Recommendations

With increasing safety threats to schools, expanded use of these tools is likely. However, to responsibly utilize high-stakes school safety technology to best serve students’ safety and wellbeing needs, schools must approach procurement and implementation with students’ privacy, civil rights, and civil liberties at the top of minds. Looking ahead, schools should:

• Expand their “safety” definition: On top of the increasing devastating safety threats like school shootings, students face countless other safety and wellbeing challenges that, if not acknowledged prior to procurement of new technology, will only be exacerbated. For example, students of color have faced negative impacts of overbroad policing, surveillance, and discipline in schools. School officials must take into account not only imminent safety issues, but also systematized safety concerns, especially facing historically marginalized and under-resourced students.
• Assess capacity and feasibility: A misconception is that acquiring new safety technology may offset the workload of existing school safety staff; however, that often is not the case given the time and care needed to ensure these tools are being used responsibly. Responding to and preventing safety threats during school hours requires expertise and proper personnel, which school districts can already lack given the many other functions they are required to perform. Schools also face increased pressure to monitor and respond to threats that happen outside of school hours, such as students’ social media posts. Before adding a new tool onto their already existing school safety infrastructure, school officials should ask questions like:
  • Given our existing resources and staffing, can we ensure (a) that this new tool would effectively and positively impact students’ safety; (b) that we have the capacity and capability to use it effectively; and (c) that we can mitigate any risks of discrimination or other harms that it presents?
  • Would acquiring this new piece of technology detract from our mission of providing kids a quality education?
• Practice data minimization: Particularly when dealing with minors, the best data practices are collecting only strictly necessary information; limiting how the data can be used, who can access it; how long it is retained; and determining whether specific features of the acquired tool can be disabled.
• Be transparent about data management: Vendors of school safety technology can often lack transparency around how data is stored, how long it is retained, and the security measures they have in place. It is vital to acquire this information and communicate that information clearly to students and parents. Parents and their children have a right to know and understand how their data is being used by the school and these third-party companies.
• Probe vendors about effectiveness and auditing during procurement: School officials must ask vendors to provide evidence demonstrating their product’s effectiveness, particularly in a school setting. In the same vein, vendors should also be able to provide information about auditing their product for disproportionate impact on protected classes of students. If vendors cannot answer some of these basic questions about their product, schools should rethink purchasing:
  • What evidence can you provide that the tool is effective at fulfilling the purposes for which we are purchasing it?
  • How is safety and impact being measured, if at all?
  • Do you have data on how your product performs across varying student demographics? If so, may we have access to it?
• Bring all necessary school staff to the decision-making table: Procuring new safety technology is a complex process that requires a breadth of knowledge and expertise to ensure that tools can be used responsibly and for the benefit of students. On top of traditional school administrators in charge of procurement and contracts, schools should ensure that their chief information officers (CIOs), chief privacy officers (CPOs), and civil rights coordinators are included in the conversation to bring their unique perspectives and recommendations.
• Engage parents, students, and other community members in the procurement process: Before making a decision to purchase from a vendor, it is imperative for school officials to understand the thoughts, concerns, and needs of who the technology is ultimately serving – students. Schools should ask questions like:
  • Do students and parents perceive these tools as modes of keeping them (or their child) safe? If not, how would this impact their learning experiences? 
  • Do parents want to be informed about our vision for this procurement decision? If so, what details would they like to know?
• Create clear governance policies and procedures prior to procurement: Before engaging in purchasing conversations, it is imperative that schools have a plan in place for what data they can securely collect, how it will be stored, how long it will be stored, deletion procedures, who has access to it, and the process for responding to safety threats that may be flagged by these systems. Humans must be involved in the monitoring, analyzing, and decision-making of the data these systems generate since, left on their own, safety tools may make erroneous mistakes or decisions that significantly impact students’ wellbeing and educational outcomes – thus creating a mechanism for accountability. This includes determining when designated school personnel should alter or override the determination/decision made by a safety tool. Humans, particularly those trained to monitor safety systems in real time, may have knowledge or context clues that a safety tool might not be able to account for. Additionally, due to the potential impact that these tools have, schools must also have an accessible redress process if students and their families feel a decision was wrong or unfair.

Conclusion

Though the idea of being at the cutting edge is alluring, CDT research has shown that schools are already failing to properly implement seemingly more straightforward edtech tools, like content blocking and filtering (which has been around since the early 2000s), in ways that protect students’ privacy, equity, and civil rights. Piling on these more complex, invasive modes of school safety technology without adhering to the recommendations made above can only serve to further exacerbate these harms, especially on protected classes of students who are already disproportionately harmed by edtech tools. 

Innovation, privacy, and equity considerations can and should go together when deciding to adopt new school safety technology tools, especially for more extreme, high-stakes safety uses. Classrooms are a space for children to learn and grow, without having to think of personal safety, threats to their wellbeing, and invasive data practices. The promises of technological advancement can be realized alongside thoughtful procurement by school administrators, which includes elevating the voices of students, parents, and teachers.