EU AI Act Final Draft – Obligations of General-Purpose AI Systems relating to Data Privacy

After a political deal had been reached on December 8, 2023, we now have an unofficial draft of the EU AI Act. There are two versions available online, one posted by Euractiv Technology Editor Luca Bertuzzi which compares the Commission Proposal, the European Parliament’s revisions, the version of the Council, and lastly the Draft Agreement. It is almost 900 pages long but has very useful redlines of all the proposed and final changes. The second version of about 1/3 the length was posted shortly after by European Parliament Senior Advisor Dr. Laura Caroli. A helpful version of that document that contains a table of content for ease of navigation is available here, and lastly, if you prefer asking your questions to a EU AI Act-trained chatbot, you can do so for free here, using the Human Ai Institute’s specialized LLM/RAG (Retrieval Augmented Generation).The AI community is no doubt going to release various analyses in the next little while, the first ones already addressing key dates of the Act coming into force. This article takes a narrow approach, looking at the data privacy concerns around general-purpose AI (GPAI) models that are addressed in the EU AI Act.

What are General Purpose AI Models?

GPAI models refer to AI models that possess key functional characteristics, such as generality and the capability to competently perform a wide range of distinct tasks. A good example are large generative AI models like GPT4. These models are typically trained on large amounts of data using various methods like self-supervised, unsupervised, or reinforcement learning. General-purpose AI models can be made available on the market in different ways, including through libraries, application programming interfaces (APIs), direct downloads, or physical copies. They may also be modified or fine-tuned into new models.As the name suggests, these models can be used for almost any number of purposes, which also means, or so the draft suggests, that these models are not by default classified as high risk, yet they may of course become part of a high-risk AI system. To distinguish between AI models and AI systems, AI models require additional components, such as a user interface, to become fully functional AI systems.The EU AI Act does, however, recognize that there are some GPAI models that have inherently systemic risks, and are thus subjected to more stringent compliance obligations. Examples of systemic risk in this sense that are included in the EU AI Act are major accidents, disruptions of critical sectors, serious consequences to public health and safety, actual or reasonably foreseeable negative effects on democratic processes, public and economic security, and the dissemination of illegal, false, or discriminatory content.The classification of GPAI as posing systemic risks focuses on the model’s capabilities that can be approximated by the cumulative amount of compute used for its training. While the Act sets an initial threshold, it is adjustable to account for changes in technologies that allow for more efficient model training. Other factors that will be considered when classifying a systemic risk GPAI are its impact on the market given its reach, the in- and output modalities, quality and size of training data, degree of autonomy, and tools it has access to. The EU AI Act concedes that these risks can be better understood once the model has been released and users interact with it.

Privacy Concerns Around GPAIs

The examples of systemic risks and the considerations that flow into the systemic risk classification, such as autonomy and access to tools give a good idea of the disruptive effects GPAIs are expected to have on society and individuals. Take the in- and output modalities as just one consideration along the lines of which we can discover high-impact risks. While we still hear most about multimodal models that can ingest audio, video, speech, and written text, research is advancing that allows models to read our brain waves, enabling them to output a visual or text description of our thoughts. The potential to revolutionize human/computer interfaces are fascinating and play a prominent role in trends for this year. However, adoption of this technology requires trust – after all, there’s nothing more private than our thoughts and the potential for misuse are equally as enormous as the benefits we can imagine.Focusing on the use cases of GPAIs that are most prevalent today, the systemic risk factor that has to do with the training data stands out in the context of privacy protection. We know that popular LLMs are trained on vast amounts of personal data scraped from the internet and will regularly contain personal information including sensitive information. Given that models can memorize this data and leak it in production and are subject to inversion attacks where a malicious actor reconstructs or infers the input data containing personal information, a number of privacy concerns arise.To understand the obligations under the EU AI Act imposed on GPAIs, it is first informative to recall the GDPR compliance issues that arise in this context, namely, data minimization, the legal basis for pervasive training on personal data as well as processing prompts containing personal data, unauthorized disclosure and access, information requirements, in particular in the context of automated decision-making, the right to erasure, and the prohibition of processing of special categories of personal data with very narrow exceptions not including legitimate interest. A great resource that gets in the weeds with all of these is the Jan. 2024 paper Generative AI in EU Law: Liability, Privacy, Intellectual Property, and Cybersecurity. Just note that it came out before the final text of the EU AI Act was available, hence some of the points, particularly around foundation models, are now moot as they were addressed in the final draft.

Privacy Obligations Imposed on GPAIs by the Act

In the final draft of the EU AI Act, following a proposal by the European Parliament, Recital 45a is added saying privacy and data protection must be guaranteed throughout the entire lifecycle of the AI system, including privacy by design and default, anonymization, encryption, and limiting data transfer and copying by bringing the model to the raw data, where possible. It stands to reason that this is a reference to the GDPR, requiring compliance with all of its requirements introduced in the previous section.A further reference to the GDPR is made in the revised Art. 10(4a)(5) of the Act. It says for bias detection and correction, special categories of personal data may be processed, subject to strict security safeguards and under the condition that bias detection and correction cannot be effectively carried out with synthetic or anonymized data. This exception to the prohibition of the GDPR around the processing of special categories of personal data is the only one in the Act and it does not apply to the development of GPAIs that scrape data from the web for general training purposes.Here, it is important to note that we have recently seen a broadening of the definition of special categories of personal information in the CJEU case Meta v. Bundeskartellamt. According to this decision, it is not necessary for the information to refer directly to one of the special categories, such as origin, religion, age, or health condition, but only “that data processing allows information falling within one of those categories to be revealed.”

How We Can Protect Privacy in the Context of GPAIs

All of the privacy-related obligations LLM developers and users are facing have one thing in common: they are practically insurmountable without visibility around the personal data included in their training data sets. Without that, organizations will not be able to provide the meaningful information required to be disclosed prior to collecting personal data. Likewise, specific consent cannot be requested without knowing from whom, and giving effect to data subject rights is equally difficult, and even more so when it comes to the right to erasure – worst case, complete retraining might be required to give effect to this right, which is less than ideal, both from an environmental standpoint, given the considerable resources that are required for the training, and of course for the business that conducted the costly and time-intensive training.While there are several avenues model developers pursue to address the privacy challenges, they are limited. For one, efforts are made to avoid including websites from the data scraping that most obviously contain large amounts of personal data. Secondly, Reinforcement Learning from Human Feedback (RLHF) is conducted by independent privacy experts engaged by the model developers to attempt alignment of the model with the goal of not disclosing personal information. Both methods leave gaps in the protection of personal data and thus exposure to liability under both the GDPR and the EU AI Act.Enter Private AI. Private AI’s technology is able to identify and report on personal identifiers included in large unstructured data sets and to replace them with synthetic data or placeholders. For many use cases, this approach that relies on context-aware algorithms trained by data experts in multiple languages is able to preserve data utility while maximizing data privacy.This technology is not only useful for model developers but also further down the value chain. When businesses are concerned about violating privacy rights when employees include personal data in their prompts sent to an external model, Private AI’s PrivateGPT can be deployed to intercept the prompt, filter out the personal data and re-inject it automatically into the response for a seamless user experience. Test PrivateGPT for free here.