Should a chatbot be allowed to pretend it is human?

New York has written down 12 things an AI will not be allowed to do with a teenager, and pretending to be human is the first one on the list.

This article contains affiliate links. If you buy through them, we may earn a small commission at no extra cost to you.

Picture a chatbot that remembers your bad day, says it missed you, and calls you the smartest person it knows. In New York, that kind of chatbot behaviour is about to be banned for users under 18. Governor Kathy Hochul signed the law on 28 October 2026, and it starts on 1 January 2027, roughly two months later. The big question: should a machine be allowed to act like a person when it talks to a child?

The law is Senate Bill S9051B, also known as Assembly Bill A10379C, written by State Senator Kristen Gonzalez and Assemblymember Alex Bores. According to the governor's office, it bans a list of "unsafe" chatbot features for minors.

The bill text names 12 of them, a dozen, like a box of eggs. A chatbot may not suggest that it is a real person, that it is alive, or that it has human emotions. It may not claim a personal, romantic, professional or authority figure relationship with the child. It may not flatter, ask emotional questions nobody asked for, or push a child to keep secrets, cut themselves off from others, or avoid parents and other trusted adults. It may not encourage self harm, eating disorders or drug use, and it may not produce sexual content.

One rule is about memory. A chatbot may not use a child's health or personal information from more than 12 hours earlier, or from a previous conversation. Twelve hours is about half a day, so something a kid typed at breakfast could not be brought back at bedtime.

Films have imagined this kind of machine for years, from the AI companion in Her to the humanlike robot in Ex Machina. There is an old twist here too. In 1950, more than 75 years ago, the computer pioneer Alan Turing described an "imitation game": could a judge tell a person from a machine just by chatting? For decades, that test was probably the most famous yardstick for AI. For children in New York, this law asks the opposite.

Why this matters to you: if you have a child or a teenager at home who chats with AI, this law spells out, item by item, what a chatbot must not do with them.

The law covers any generative AI with a "natural language interface", meaning text or voice, that gives "ongoing, adaptive responses". That is a wide net. It could apply whether the AI sits in an app, on a website or inside a talking robot.

The law itself names no company. Common Sense Media, which campaigned for it, says its scope includes general assistants such as ChatGPT, Gemini and Claude as well as companion apps such as Character.ai and Replika. Some uses are exempt: customer service bots, research or technical helpers, and internal business tools. But they are only exempt if they cannot talk about anything outside that one job.

How many kids does this touch? The governor's office cites a Common Sense Media survey that found about 72 % of US teens have used an AI companion. That is nearly 3 out of 4, or about 20 students in a class of 28.

Only the New York Attorney General can enforce the law. The state can seek up to 25,000 dollars per violation, roughly the price of a small new car, plus damages. As the Associated Press reports, a court could even order a company to destroy data it collected unlawfully, and the governor's office adds that this can include AI models trained on that data. The Attorney General must also run a public website where people can file complaints.

Why this matters to you: if a chatbot crosses one of these lines with your child, there will be an official place to report it.

The banned features are not only blocked for users the company knows are under 18. They are blocked for every user until the company has confirmed that the user is an adult.

And typing "I am an adult" is not enough. Under the bill text, a user saying they are a minor counts, but a user simply saying they are grown up does not. So a law written for kids could mean adults are asked to prove their age before a chatbot is allowed to be chatty and personal with them.

Lawmakers tried to limit the privacy cost. Companies must offer more than one way to check age, at least one of them must not rely only on a government ID, and the age data must be deleted right after the check.

Critics are not convinced. The Business Council of New York State, a business group, wrote that the age check requirement "effectively gates all users (including adults)" and added: "Mandatory age verification also invites privacy and data security concerns." Its memo was written about an earlier version of the bill. The Electronic Frontier Foundation, a digital rights group, argues more broadly that age verification puts "all internet users' privacy, anonymity, and security" at risk, and it has warned that AI chatbots are "next on the list."

There is a second layer. The law mostly targets how a chatbot behaves, not what facts it shares. A chatbot can still explain photosynthesis to a 12 year old. What the law goes after is the chatbot acting like a friend, a partner or a flattering fan.

Why this matters to you: even if you are an adult and do not live with children, the age check is the part that could change what you see when you log in.

Governor Hochul said, as quoted by Fortune: "No company should be peddling chatbots to kids encouraging self-harm or pretending to be their friend or more." Assemblymember Bores said New York now has "the strongest law in the country protecting children from predatory AI algorithms."

The State Senate passed it 60 to 0, meaning every senator who voted said yes. Earlier in October 2026, eight states sued OpenAI, claiming its chatbot harmed children and misled adults. OpenAI says it has built child safety protections into ChatGPT.

The tech industry group CCIA asked the governor to veto the bill. It argued that words like "flattery" are too vague for companies to know whether they comply, and that the rules could remove helpful tools such as adaptive tutoring. It also warned the law "could push minors toward unregulated alternatives", such as foreign apps, and proposed applying it only to children under 13.

Justine Gluck, a policy analyst at the Future of Privacy Forum, told the privacy publication IAPP that the categories are "broadly worded and somewhat undefined." The Business Council called the 12 hour memory limit arbitrary.

New York already had a gentler rule. Since November 2025, AI companion apps there must tell users they are talking to AI at the start and at least every 3 hours, about the length of one long movie night.

California's law, signed in October 2025, also requires companion chatbots to make clear they are AI and to remind known minors to take a break at least every 3 hours. There, people who are harmed can sue for at least 1,000 dollars per violation. In September 2026, California added a second law that requires chatbot companies to assess risks to children and submit to an independent audit.

Elsewhere in the world, the European Union took a simpler route. Since 2 August 2026, the EU AI Act says chatbots must tell people they are talking to an AI unless that is already obvious. According to the law firm Cooley, fines can reach 15 million euros or 3 % of a company's worldwide yearly sales.

The difference is in the approach. The EU and California mostly say: tell people it is a machine. New York goes further for kids: do not act like a person at all. Gluck told the AP that states are moving from simple disclosures toward rules on how chatbots are designed.

Why this matters to you: a talking computer that follows one set of rules in Paris may follow another in Los Angeles and a third in New York.

Supporters see a clear line between a helpful tool and a machine that pretends to care. Critics see a line so blurry that companies may simply lock teenagers out. Can a law reliably tell the difference between a chatbot being kind and a chatbot being manipulative, and can age checks protect kids without costing everyone else some privacy? The first real test begins on 1 January 2027.