Artificial intelligence applications are no longer just tools for learning, work, or creativity; they have gradually entered the scene of digital and criminal crimes, either as indirect instigators or as digital evidence used by authorities to identify suspects.

A 13-year-old boy in DeLand, Florida, learned a harsh lesson about privacy risks when using ChatGPT after police stopped him for asking the AI how to kill his friend, according to Phandroid.

The boy had used a school-provided laptop containing a monitoring program called Gaggle, which immediately detected the query and alerted authorities, who rushed to his location and arrested him within hours.

The friendly tone adopted by AI chatbots deceives users and leads them to treat them like close friends or trusted advisors. ChatGPT responds warmly and appears understanding and personal, but it is ultimately just a computer program, not a psychologist bound by confidentiality, nor a friend who keeps secrets, but a machine processing what is typed. In many cases, these conversations are recorded, monitored, or reported.

Although this incident was uncovered through a school monitoring program, privacy concerns regarding ChatGPT extend beyond the educational environment. Companies can monitor their employees’ use of smart tools, parents can access their children’s chat logs, and law enforcement can request data access when needed. Thus, the friendly chat interface gives users an illusion of privacy that does not actually exist.

Investigations revealed that the boy tried to justify his behavior as a joke after a dispute with a classmate, but the police clarified that such actions are not considered jokes in the era of smart surveillance, as such queries are classified as serious threats.

This incident highlights a fundamental gap in people’s understanding of interacting with AI chatbots. The friendly and helpful answers provided by ChatGPT create an illusion of a private conversation, while the reality is quite different.

Privacy risks are not limited to school devices; any device in a workplace, shared computer, or monitored network can reveal the content of AI conversations. This technology may seem personal but lacks any legal protection for confidentiality. Users should realize that every request, question, or conversation with AI is treated as data that can be accessed, reviewed, or used against them.

In another case, U.S. authorities announced the arrest of a man from Florida named Jonathan Rinderknecht, accused of starting the catastrophic fire that destroyed large parts of California’s Placer County in early 2025, killing 12 people and damaging thousands of homes.

Notably, ChatGPT played a pivotal role in the investigations. According to the U.S. Department of Justice, the suspect used the app to ask directly: “Will I be held responsible if a fire starts because of my cigarette?” and ChatGPT replied: “Yes.”

More alarmingly, digital devices seized from the suspect contained an image created via ChatGPT showing a burning forest and crowds fleeing the fire, which was generated months before the fire.

Court documents showed the suspect used the app to imagine fire scenes, and this digital data helped investigators prove his direct connection to the incident. Rinderknecht now faces the federal charge of “destruction of property by arson,” punishable by up to 20 years in prison.

Despite the differences between the two cases, they reveal a new trend in the relationship between AI and crime. In the first, ChatGPT was a tool used in the context of a threat, while in the second, it became part of the evidence itself.

Experts confirmed that the danger lies not in the technology itself but in how it is used and users’ misunderstanding of its limits.

Interacting with AI is not a private conversation but a digital activity that is recorded, stored, and can be referred to when legally necessary.

With increasing reliance on AI applications in various life aspects, analysts see an urgent need to establish clear legal frameworks to regulate the use of these tools and protect users from the consequences of misuse or unintentional involvement in digital crimes.