Data Privacy and Ethics in AI School Counseling: A Guide for US High Schools and Districts
By Dev·Published ·Updated
Why AI Counselling Tools Create a New Class of Data Obligation
AI counselling tools do something no previous generation of educational technology has done at scale, for instance, like they process deeply personal student information, academic records, financial circumstances, family background, and college aspirations and use that data to generate guidance that directly shapes a student's postsecondary trajectory.
That combination of data intimacy and operational scale creates obligations that extend well beyond a standard software procurement checklist. The regulatory environment is evolving to align with these changes. In 2026 alone, 134 bills related to AI in education have been introduced across 31 states, with student data privacy as the single dominant legislative concern (MultiState Associates, 2026). Districts that treat AI tool adoption as a technology decision rather than a governance decision are already lagging.
This guide organises the challenge across three dimensions that every district leadership team, legal, instructional, and operational, must resolve together before a single student record flows into any AI counselling platform. Each dimension builds an architectural structure with legal compliance that sets the floor; ethical principles define responsible operation above that floor, and governance structures make both sustainable over time.
Dimension 1: The Legal Data Privacy Framework
The legal data privacy framework for AI school counseling operates across two tiers: federal law (FERPA and COPPA) sets the baseline, while California, New York, and Texas have each enacted amplifications that exceed federal minimums. Districts in all three states must address state-specific DPA requirements before deploying any AI counseling platform.
AI counselling platforms sit at the intersection of three overlapping legal regimes, including federal statutes (FERPA and COPPA), state amplification laws that exceed federal minimums, and an accelerating wave of new AI-specific legislation at the state level. Districts must understand all three layers before contracting with any vendor.
FERPA: The Federal Baseline
The Family Educational Rights and Privacy Act (20 U.S.C., 1232g) governs how schools collect, maintain, and disclose student educational records, and it applies without exception to every third-party AI platform a district uses to process, store, or transmit those records. According to FERPA, an AI counselling platform that takes in student GPA, test scores, course history, financial aid eligibility, and demographic data is considered a school official. Before any student data can be shared, the platform must sign a Data Processing Agreement (DPA) (U.S. Department of Education, 2025).
Educational institutions currently face an average of more than 4,300 cyberattack attempts per organisation per week, and breaches affecting student data have impacted over 1.8 million students in the United States since 2020 (Concentric AI, 2025). FERPA compliance is not a legal formality – it is the district's primary legal shield for student data in a high-risk digital environment. The U.S. Department of Education's 2025 FERPA guidance update raises the accountability bar further by clarifying that schools and state agencies must proactively demonstrate compliance and submit documentation (PikMyKid, 2025).
The three core obligations that flow from the school official designation under FERPA are the following:
Legitimate educational interest: The AI tool can only access student data for the specific educational purposes described in the executed DPA, not for broader product development, AI model training, or platform analytics. Any data use not named in the DPA is a violation.
Redisclosure prohibition: Vendors cannot share student data with any third party, including sub-processors, analytics partners, or affiliated platforms, without the explicit district authorisation set out in the DPA. Sub-processor relationships must be named and disclosed at contracting.
Inspection and amendment rights: Students and parents retain the right to inspect and request corrections to educational records, including those processed or generated by AI systems. Districts must be able to demonstrate, on request, how a platform surfaces and corrects student data.
Practical question for district legal counsel: Has every AI vendor used in counseling executed a DPA that explicitly designates them as a school official, restricts data use to stated educational purposes, names all sub-processors, and has been updated to reflect the 2025 Department of Education guidance? If any vendor cannot produce this documentation on request, that absence is your compliance answer.
COPPA: Students Under 13
The Children's Online Privacy Protection Act applies to any online service that collects personal data from children under 13. Districts deploying AI counselling tools with 8th or 9th graders, a common scenario for college-readiness programming, must ensure that the platform is COPPA-compliant.
Schools may provide parental consent on behalf of families under the school official exception, but this exception applies only to tools used strictly for educational purposes with no commercial data use. The Federal Trade Commission's January 2025 amendments to the COPPA Rule substantially tightened the standard: vendors can no longer assume parental consent for advertising or data sharing and must explicitly document every data decision (Federal Trade Commission, 2025). Legacy COPPA compliance documentation predating January 2025 is insufficient. Districts must require vendors serving younger secondary students to demonstrate compliance with the 2025 amended rule, not the prior version.
State-Level Amplifications
California – From SOPIPA to CALPIPA (AB 1159)
California's Student Online Personal Information Protection Act (SOPIPA), enacted in 2014 and now known as KOPIPA, was the nation’s first law to directly regulate educational technology companies. It prohibited ed-tech operators from selling student data, building commercial profiles, and using that data for targeted advertising. At the time of enactment, it was truly groundbreaking. The problem, as privacy researchers explained in detail in 2026, is structural:
SOPIPA was designed for the 2014 technology world and applies only to goods mainly made for K–12 students.
Captain Compliance (2026) says that this leaves big coverage gaps for AI platforms that are sold to schools but not only to them.
California AB 1159 – the California Learner Personal Information Protection Act (CALPIPA) – was introduced to close these gaps. The bill passed the California Assembly in January 2026 and advanced through the Assembly Judiciary Committee in March 2026. Its key provisions that are directly relevant to AI counselling platforms include a clear ban on EdTech operators using student data to train AI models; expanded coverage to any online service the operator knows is used for school purposes; extended protections for college students; and a private right of action with a 45-day notice and cure provision (California Assembly Committee on Privacy and Consumer Protection, 2026).
For California districts, the compliance question is no longer only, “Is your vendor SOPIPA-compliant?" It is, “Does your vendor's DPA explicitly prohibit using student data to train AI models, and does that prohibition survive the vendor's standard terms of service?” "AB 1159 will make the answer legally mandatory. Responsible districts should be requiring it contractually now.
New York – Education Law 2-d
New York's Education Law 2-d says that tech companies have to sign an extra document called the Parents' Bill of Rights and follow stricter security rules than the federal FERPA minimums. These rules include encrypting data while it's being sent and stored, keeping track of who has access to it, and giving parents more time to report breaches than the federal rule (New York State Education Department, n.d.). Families in New York City asked the NYC Panel for Educational Policy in May 2026 to stop all AI deployments in schools until a governance framework was finalised. They said that the city's preliminary March 2026 guidance did not provide clear parental opt-out rights or enforceable safeguards (Pursuit, 2026). Districts that work in New York should expect that AI vendor agreements will be looked over by an informed, organised group.
Texas – TEA Student Data Privacy Framework
Texas districts must follow the Texas Education Agency Student Data Privacy Framework for requirements about data use agreements, vendor security assessments, and parent notification standards that are stricter than the federal FERPA minimums. Idaho's SB 1227, enacted in 2026, provides a similar model, requiring clear data privacy protections for AI tools used in schools, and shows a wider trend of states creating AI-specific student privacy rules that go beyond the federal baseline (MultiState Associates, 2026).
Dimension 2: Ethical Principles for AI in School Counseling
Legal compliance defines the minimum threshold for responsible AI deployment. Ethical governance defines what responsible operation looks like above that threshold in the day-to-day practice of counselling, in vendor selection, in family communication, and in the institutional culture that a district builds around AI-assisted services. These four principles are not aspirational values statements. They are operational design requirements with research backing and, increasingly, legislative grounding.
Ethical deployment of AI in school counseling requires four non-negotiable operating principles. Equity and human oversight carry the heaviest evidentiary weight: peer-reviewed research documents demographic bias in AI college recommendations, and multiple states are now codifying human review as a legal requirement.
Principle 1: Transparency, Tell Students and Families What Is Happening
When AI generates a college list, monitors FAFSA completion, flags academic risk, or responds to a parent query after hours, the student and family should know this is happening and should understand, in plain language, what data the system used to reach its output.
Transparency is not only ethically required. The 2026 state legislative wave includes multiple bills focused explicitly on parental consent to data collection and disclosure of AI involvement in educational decisions (MultiState Associates, 2026). Districts that proactively communicate AI tool use through updated family handbooks, revised annual FERPA notices, and counselor-led orientation sessions will be compliant with these requirements before they are enacted, rather than scrambling to meet them afterward. The evidence also supports transparency as a trust-building mechanism: schools that communicate clearly about AI tool use in counselling report stronger family engagement, not weaker (SchoolAI, 2026).
Principle 2: Human Oversight – AI as Decision Support, Not Decision Maker
The distinction between decision support and decision-making is not semantic. It is the difference between accountable guidance and unaccountable guidance and between a counsellor who can explain a recommendation and an institution that cannot.
AI recommendations must be subject to a counsellor’s review before they become institutional guidance. An AI-generated college list, financial aid pathway, or academic intervention flag that reaches a student without counsellor review is not an efficiency gain instead it is a transfer of professional accountability from a licensed educator to an algorithm. Oklahoma and Maryland have both introduced 2026 legislation explicitly prohibiting AI from making high-stakes educational decisions about students without human review (MultiState Associates, 2026). The direction of regulatory travel is unambiguous.
In practice, this means districts should evaluate AI counselling platforms not only for the quality of their outputs but also for the design of their human-in-the-loop architecture:
How does the platform surface AI outputs to counsellors before they reach students?
What override mechanisms exist?
Are those mechanisms documented in the DPA and acceptable-use policy?
A platform that makes it difficult for counsellors to review or override AI outputs is, by design, a decision-making tool, regardless of how its vendor categorises it.
Principle 3: Equity– Interrogate the Training Data
AI systems trained on historical admissions and academic data do not simply learn patterns, but they encode the distributional properties of the data they were trained on, including historical patterns of inequity. The peer-reviewed evidence on this point is now substantial.
Research conducted at the University of Maryland documented that AI counselling tools, including large language models used in advising contexts, recommended colleges with lower selectivity ratings and lower projected graduate salary outcomes to Black students when presented with academically identical profiles to those of students from other demographic groups (University of Maryland, 2024). A 2024 study published in AERA Open confirmed measurable calibration bias across racialised groups in college student-success prediction algorithms, finding that models relying on historical data systematically capture and perpetuate societal inequities (Gándara et al., 2024). A parallel study in the World Journal of Advanced Research and Reviews found that the 2023 ban on race-conscious admissions has compounded the problem, as models attempting to remove race as a variable without bias-mitigation techniques increase outcome arbitrariness disproportionately for students from under-represented groups (Singh et al., 2025).
These findings carry a direct implication for district procurement. Vendor claims of "unbiased AI" are not self-certifying. Before signing any AI counselling contract, districts should require vendors to provide documented answers to the following questions:
How was your training dataset assembled, and what is the demographic distribution of the students represented in it?
Have you conducted independent bias audits covering race, gender, income level, English language learner status, and first-generation college-going status? What did those audits find, and what remediation was applied?
What ongoing post-deployment bias monitoring do you conduct, and how are findings reported to district clients?
The absence of clear, documented answers to any of these questions is itself a procurement signal, and it is one that districts have both an ethical and a legal responsibility to act on.
Principle 4: Data Minimisation – Challenge Every Data Field Requested
AI platforms have a structural commercial incentive to collect more data than is necessary for the stated counselling function, as broader data collection increases model generalisability and future commercial value. Districts are under no obligation to accommodate that incentive” and under FERPA's legitimate educational interest standard, they have a legal basis to refuse it.
For every data field a vendor requests access to, districts should apply a single test:
Is this specific data point genuinely necessary for the counselling function that this platform is contracted to deliver?
If the answer is unclear or not demonstrable, the district should decline to provide the data. This applies equally to data collected passively through platform usage, like click patterns, session length, and interaction history, as to data imported from student information systems. Data minimisation is both an ethical principle and, under FERPA, a legal one. Collecting student data that exceeds the scope of the stated educational purpose is a FERPA violation, not merely a policy preference (U.S. Department of Education, 2025).
Dimension 3: Governance Structures Before Deployment
Governance is not what happens after a tool is deployed. It is what makes deployment responsible in the first place. The five-element governance framework below should be completed, if not initiated, before any student data flows to an AI counselling platform. Each element has a named owner, because accountability requires ownership.
Five governance elements must be completed, not initiated, before any student data flows to an AI counseling platform.
Vermont's HB 650, introduced in 2026, is the leading indicator of where the national governance floor is heading: annual vendor certification, public product registries, and state agency review of privacy compliance (MultiState Associates, 2026). Illinois SB 3735, also introduced in 2026, would grant families the right to opt out of school technology and AI grading decisions and restrict how companies use student data for AI training without explicit consent. Districts that build annual reviews into their governance structures now will be compliant with these requirements as they propagate nationally, rather than retrofitting governance onto deployed systems.
What Responsible AI Deployment Looks Like in Practice
The districts that are navigating AI counselling deployment well share four operational characteristics that cut across all three dimensions of this framework.
They complete governance before procurement. The DPA, acceptable-use policy, and vendor security assessment are not post-deployment paperwork, they are procurement gates. No contract is signed until all five governance elements are complete.
They treat equity as a technical contractual requirement. Bias audits, demographic representation disclosures, and post-deployment monitoring are written into vendor contracts as deliverables, not left as voluntary commitments that vendors can modify through terms-of-service updates.
They communicate proactively with families. Annual FERPA notices are updated to name AI tools specifically. Counsellors are trained to explain, in plain language, what AI does and does not do in the counselling process and what a student’s or family’s options are if they have concerns.
They conduct reviews annually, not just at the time of renewal. The regulatory environment for AI in education is moving at a legislative pace. A DPA adequate for 2024 state law may not satisfy 2026 requirements. Annual review is not a governance best practice. In Vermont, it is now a legal requirement, and this requirement is spreading.
"Responsible AI deployment in school counseling is not primarily a technology question. It is a governance question. The technology is ready. The governance frameworks are the work." CollegeFind.ai Editorial Team; June 2026
Frequently Asked Questions
a) Does FERPA apply to AI counselling tools used by US high schools?
Yes. Any AI counselling platform that processes, stores, or transmits student educational records, including GPA, test scores, course history, and financial aid data, is subject to FERPA (20 U.S.C. 1232g). The vendor must be designated a school official under a signed data processing agreement before any student data is shared. The U.S. Department of Education's 2025 guidance update expanded compliance documentation requirements for all third-party technology providers (U.S. Department of Education, 2025).
b) What is SOPIPA, and does it apply to AI tools in California schools?
SOPIPA (now KOPIPA in current legislative form) prohibits ed-tech operators from selling student data or using it for commercial profiling. California's AB 1159 / CALPIPA, advanced through the Assembly in 2026, extends these protections to explicitly prohibit the use of student data to train AI models and creates a private right of action for violations (California Assembly Committee on Privacy and Consumer Protection, 2026). California district DPAs should reflect this requirement contractually now, ahead of enactment.
c) Can AI tools make final college counselling decisions for students?
No. Oklahoma and Maryland have both introduced 2026 legislation explicitly prohibiting AI from making high-stakes decisions about students without human review (MultiState Associates, 2026). All AI-generated recommendations must be reviewed by a qualified counsellor before they become institutional guidance. A platform that lacks counsellor review of AI outputs before student delivery is, by design, a decision-making tool, not a decision-support tool.
d) What governance steps must a district complete before deploying an AI counselling tool?
Five steps must be completed if not initiated and before any student data is shared with a vendor:
signed DPA designating the vendor as a school official under FERPA;
written AI acceptable-use policy for counseling with human-review thresholds;
staff training on AI data use and output override procedures;
and an annual review cycle for vendor agreements (Vermont HB 650, 2026, now requires this by law).
e) What is the COPPA school official exception, and when does it apply?
The school official exception allows schools to consent on behalf of parents for online services used with children under 13, but only for tools used strictly for educational purposes with no commercial data use. The FTC's January 2025 amendments require vendors to explicitly document every data decision; assumption of consent is no longer permitted (Federal Trade Commission, 2025). Districts deploying AI counselling tools with 8th or 9th graders must verify COPPA 2025 compliance, not legacy pre-amendment compliance.
f) How does algorithmic bias affect AI college counselling recommendations?
Peer-reviewed research found that AI counselling tools recommended colleges with lower selectivity and lower projected salary outcomes to Black students who had academically identical profiles to those of other demographic groups (University of Maryland, 2024). AI systems trained on historical data encode historical inequities. Districts must require vendors to disclose bias audit results and demographic representation in training data before procurement and to provide ongoing post-deployment monitoring as a contractual deliverable (Gándara et al., 2024).
Data Privacy and Ethics in AI School Counseling: A Guide…
