Online platforms would face new child-safety reporting and fee obligations under the proposal, with designated very large social media and video-sharing services required to submit audited compliance plans.
As well as introducing age limits for social media and video-sharing platforms, the proposal also imposes wider safety-by-design obligations on platforms, games, app stores and AI chatbots.
What AI does it apply to?
The proposal is intended to ‘complement’ the AI Act as regards child-specific safety requirements for AI companions and general conversational chatbots, which are defined as follows:
- “AI companion” means an AI system, including a general-purpose AI system, that provides sustained, personalised interaction or companionship which simulates or facilitates a social, emotional or interpersonal relationship with a user.
- “General conversational chatbot” means a general-purpose AI system with general conversational functionalities for direct interaction with users that is capable of providing assistance across multiple domains and tasks; this definition excludes AI systems whose conversational functionality is limited to a specialised service, task or pre-defined set of functions, including specialised customer-service, business operation, technical-support, transactional, educational, information-retrieval, industrial or manufacturing AI applications.
Based on the above, it seems that narrow vertical models and bots such as speech to text and text to speech operating in the customer service space will be out of scope. The focus is more generally capable chatbots.
Default child-safe settings
Unlike social media, AI chatbots are not generally subject to a graduated access model. Under the proposal where a chatbot is available to under-18s, Article 8 extends a general safety-by-design obligation to it. The service must provide an age-appropriate experience, with child-safe settings on by default unless the provider can establish that the user is an adult.
According to one of its FAQs on the KIDS Act, the Commission confirms that:
AI companions and chatbots accessible to minors may not use designs that simulate human relationships in ways likely to create emotional dependency.
By default, they may not carry a child’s earlier conversations into later ones. They must be tested for risks to children before launch and monitored for harm afterwards. Under-13s can use them only through parental control tools. Built into a platform or game, a chatbot may not switch on automatically, may not be pushed at children, and must be easy to turn off".
Design obligations
So, systems must avoid design features or behaviour that simulate interpersonal relationships likely to create emotional dependency and must not by default carry information from a child's earlier interactions into later conversations, except where needed to protect the child or implement protective settings. The Commission also plans a voluntary child-safety code of conduct for AI.
As well as this, a range of provisions of the proposal apply to chatbots like:
- Article 4 - anti-circumvention
- Article 8 - general obligation on safety by design
- Article 14 - specific obligation on safety by design
- Article 18 - agency of minors
- Article 19 - child-friendly reporting and support tools
- Article 20 - tools for guardians, and
- Article 21 - complaints.
Supervision and enforcement
The AI Office would directly supervise the AI chatbots most widely used in the EU. National market surveillance authorities remain responsible for other AI systems covered by the Act. Investigations are to conclude within 90 days.
Fines can reach up to 6% of global annual turnover for intentional or negligent breaches. Providers under Commission supervision would pay a top-up to the DSA supervisory fee, capped at 0.03% of worldwide annual net income.
Expected enactment
Generally speaking, a new EU Act can take a few years to go from this stage to application. The huge focus not only in the EU but more broadly on minors’ use of social media and sophisticated technology, and the political momentum behind it, could see this proposal move a lot faster. This proposal stipulates that its relevant measures would apply to chatbots six months from the Act entering into force.
For more information, contact a member of our Artificial Intelligence team.
EU AI Digest
Our weekly EU AI Digest is written for in-house legal, compliance, risk and senior business teams who are dealing with AI-related issues in their organisations.
The content of this article is provided for information purposes only and does not constitute legal or other advice.
