San Francisco-based AI chatbot maker Replika which runs a freemium “virtual friendship” service based on customizable digital avatars.

Personalised responses are powered by artificial intelligence Italian Authority for privacy protection ordered to stop processing data of local users.

The guarantor said it was concerned that Replika’s chatbot technology posed a risk to minors.

The company lacked a proper legal basis for processing children’s data under EU data protection rules.

Chatbots

The regulator is also concerned about the risks AI chatbots could pose to emotionally vulnerable people. It also accuses Luka Inc, the developer behind the Replika app, of failing to meet regional legal requirements to clearly communicate how it uses people’s data.

The order to stop the processing of Italian data is effective immediately.

In a press release announcing its crackdown, the watchdog said:

Interfaces

“The AI-powered chatbot, which generates a ‘virtual friend’ using text and video interfaces, will not currently be able to process the personal data of Italian users.

Italian company Garante imposed a temporary data processing restriction on the US-based company that developed and operates the app; the restriction will take effect immediately.”

“Recent media reports, along with tests carried out by the SA [supervisory body] on ‘Replika’.

App poses

Shown that the app poses real risks to children – primarily the fact that they are being delivered answers that are totally disproportionate to their age “, he added.

Replika was the first API partner for OpenAI’s large language model text generation technology.

GPT-3 – although its service does not run on a carbon copy of GPT-3 (nor is it the same technology as OpenAI’s busy ChatGPT).

Rather, the startup claims to have “tuned” GPT-3 using a network machine learning model trained on dialogue to refine the generative technology for its particular use case: conversational (and it claims “empathetic”) AI companions.

But concerns have been raised before about the risks technology can pose to children.

From concerns that children will be exposed to inappropriate content, to more general concerns that they could become addicted to the interactions or just be encouraged to spend a lot of money.

Customizing their avatars or gaining access to other paid content. However, the Italian watchdog appears to be the first regulator to take formal action on child safety.

Stores

Garante’s order states that several user reviews of the app report sexually inappropriate content.

It also notes that although the app is listed as 17+ in Apple’s iOS and Google Android app stores, the developer’s terms of service only prohibit use by those under 13.

And while those under 18 must get permission from a parent or guardian.

The watchdog points out that the app doesn’t attempt to verify the age of users or block minors who provide information about their age — which is why it believes Replika is failing to protect children.

“In reality, there is no age verification mechanism: no mechanism to verify children, no blocking of the application if the user declares that he is a minor.

When creating an account, the platform only asks for the user’s name, email account and gender,” it states.

“And the ‘answers’ provided by a chatbot are often clearly at odds with the increased safeguards children and vulnerable people are entitled to.

Several reviews on the two major App Stores contain comments from users who flag sexually inappropriate content.

“Replica” violates the EU Data Protection Regulation: it does not meet transparency requirements and processes personal data illegally.

Performance

As the performance of a contract cannot be invoked as a legal basis, even by implication, given that children cannot enter into a valid contract under Italian law.”

Garante added, ordering its US developer to stop processing data relating to Italian users – giving it 20 days to communicate measures taken to comply with the order.

Failure to comply with the order is punishable by a fine of up to 20 million euros or 4% of the total worldwide annual turnover, it further states.

Replica was contacted to request a response to the Guarantor order.

The EU’s General Data Protection Regulation (GDPR) places a strong emphasis on protecting children’s information and privacy.

For example, it suggests that services likely to have minors as users should consider incorporating child-friendly design and be proactive in risks.

Assessment to ensure that it detects potential security and other rights issues.

Watchdogs

Watchdogs in the region have shown a willingness to pay attention to offenses in the area.

Last fall, for example, Instagram was fined nearly $440 million for violating children’s privacy.

Consumer protection authorities in Europe have also raised concerns about the safety of children on TikTok – although an investigation into how TikTok handles children’s data is still ongoing in Ireland.

Safety

Italy’s data protection watchdog has proven particularly sensitive to child safety concerns in recent years. Two years ago it used emergency intervention to order TikTok to block users it could not verify the age of in response to the death of a child who was reported to have been involved risky platform challenges. This resulted in over half a million accounts being purged.

GDPR

However, despite some enforcement of the GDPR (and consumer protection laws) on child safety issues, campaign groups have argued that children are still not being properly protected – and are continuing to push for stricter laws.

So the restrictions will probably only get tighter.

In the UK, an age-friendly design law aimed at protecting minors from safety and privacy risks came into force in autumn 2021.

Although the French data protection authority has also published a set of recommendations on how to ensure the protection of children’s digital rights.

The UK is also working to pass an online safety bill aimed at keeping children safe, responding to public concerns about what children are exposed to online.
The regulator is also concerned about the risks AI chatbots could pose to emotionally vulnerable people. It also accuses Luka, the developer behind the Replika app, of failing to meet regional legal requirements to clearly communicate how it uses people’s data.

The order to stop the processing of Italian data is effective immediately. In a press release announcing its crackdown.

The watchdog said: “An AI-powered chatbot that creates a ‘virtual friend’ using text and video.

In recent months, EU lawmakers have also agreed to completely ban the processing of minors’ data for ad targeting in two major updates to the bloc’s digital rules.

The Digital Services Act and the Digital Markets Act, which are due to come into force later this year. .

Sources: Techcrunch | yro.slashdot