Generative AI and GDPR Part 2: Privacy considerations for implementing GenAI use cases into organizations

This is the second of a two-part article reflecting on EU data protection considerations for the implementation of GenAI use cases in organisations. Part one is available here.

The rapid ascent of generative AI poses fresh challenges for data protection. With these AI systems handling and generating data, including personal information, there's a growing necessity for companies utilizing such systems to handle this data in accordance with GDPR guidelines. 

This article will delve into some of the most crucial GDPR requirements applicable to the use of generative AI (“GenAI”).

Rights of Data Subjects

The GDPR provides for various data subject rights that may become relevant in connection with AI. The risk of “hallucination”, i.e., inaccurate information generated by AI, creates problems if personal data is included. These hallucinations occur because the output data is based on statistical correlations that are not fully transparent due to the black box effect, therefore the process of data evaluation and decision-making by AI is not fully comprehensible to humans due to its complexity and cannot be easily changed. If personal data is included as part of such hallucinations, the question is whether the data subject has the right to have the data corrected and/or deleted. Art. 16 and 17 GDPR are relevant sections to allow data subjects to do this.,

Due to the black box effect, the identification and deletion of specific data sets from an AI model is extremely complex both technically and logistically, especially if the data has already been integrated into the model and can no longer be clearly identified. This raises the question of whether these difficulties justify the deletion of the entire AI model, including the personal data in question, as rectification is currently almost impossible. Such an approach was chosen by a US-Court ruling in 2022, whereby the WW International was ordered to delete not only illegally collected data of subjects but also algorithms that it trained. Though, this solution appears to have favoured personal rights it creates issues for the technological progress, freedom of speech and seems to also be disproportionate under primary EU law (i.e. the EU Charter of Fundamental Human Rights, namely Art. 16 the “freedom to conduct a business” and the principle of proportionality under Art. 52 Para. 1 sentence 2). The EU Charter of Fundamental Human Rights is mandatory and must be taken into account when interpreting secondary EU law like the GDPR.

An alternative to the deletion of AI models could be to emphasise transparency to users and build trust while safeguarding technological progress for future compliance requirements. Disclosure of data sources and processing methods could help users understand how their data is handled and make more informed decisions. Clear communication (e.g. in privacy statements or user notices) could foster a closer relationship between AI provider and AI users.

Automated Decision-Making

ADM is playing an increasingly important role in the age of AI. ADM refers to the process of making decisions based on data and algorithms, without human intervention. Art. 22 GDPR states that data subjects may not be subjected to a decision based solely on automated processing of personal data if that decision produces legal effects concerning the data subject or similarly significantly affects the data subject. It is currently being discussed by the European Court of Justice (request for a preliminary ruling from the Administrative Court of Wiesbaden (Germany) lodged on 15 October 2021 – OQ v Land Hessen, Case C‑634/21) whether the mere preparation of decisions by AI already qualifies as falling under Art. 22 GDPR (with respect to granting loans), although this decision will ultimately be executed by a human being. There is a tendency to construe this broadly (which the Advocate General´s Opinion in said case confirmed where a controller draws strongly on that recommendation for its own decision), and it is expected that courts will apply Art. 22 GDPR to preparatory decision making going forwards. Automation bias may be one reason for this. However, looking at the wording of Art. 22 GDPR, it should be taken into account that it is only necessary for a human to make the final decision, in order to avoid the risk of making people purely into objects of automated decisions. Automated preparations of human decisions are generally not prohibited.

Not only are the legal bases difficult to establish in cases of Art. 22 Para. 2 GDPR, but meeting the procedural requirements established by Art. 22 Para. 3 GDPR will be even more difficult for AI users. In these cases, AI users must take appropriate measures to safeguard the rights and freedoms, as well as the legitimate interests of the data subject, which includes the right to obtain the intervention of a person on behalf of the controller, to express their point of view and to contest the decision. The intervention of a person constitutes the most important element. Any review must be carried out by a person who is authorised and empowered to change the decision. This person should thoroughly review all relevant data, including any additional information provided by the data subject. It will require high efforts to establish this in a meaningful manner, as required under GDPR.

Data Security

Ensuring data security is a key aspect of integrating AI in accordance with data protection law. Technical and organisational measures such as access controls, encryption, data backups and restores etc., play a crucial role in minimising any risks and upholding data protection principles. It is important to consider the novel risks resulting from AI, such as model inversion attacks, in which an attacker attempts to extract information about the training data or the internal model of a machine learning system by sending targeted queries to the model and analysing the output. AI users must monitor guidance published by regulators, in order to stay ahead of the curve, e.g., by ENISA or the German Federal Office for Information Security, which recently published a paper on opportunities and risks for industry and authorities with respect to Large Language Models (see here).

Ultimately, most of these risks will have to be dealt with by the AI provider as the “owner of the system”, who can address security issues effectively (it is also the “cheapest cost avoider” from a law and economics’ perspective). AI users need to make deliberate choices among the growing array of AI providers as an integral part of managing risks. Depending on individual data protection requirements and technical prerequisites, companies should select AI providers that offer suitable security precautions and data protection mechanisms. Eventually, given the rapid growth of the market, “risk minimisation through supplier selection” should be a sensible opportunity in the future to mitigate security risks. Meaning that while the advancement of GenAI is currently in its early stages, further growth with more players entering the market offering potentially different solutions is expected (for further background see the OECD´s initial policy considerations for GenAI here).

Non-Discrimination and fairness (Accountability)

In cases where AI models are used to make inferences about individual people, the utmost care must be taken to ensure that the results are statistically accurate, unbiased, and non-discriminatory. An illustrative example of this problem is image generation tools that repeatedly present images of a 'CEO' as a white male, revealing gender and ethnic biases. Similarly, the use of AI systems in the medical field has shown that biases towards certain medical conditions can occur among minorities, leading to incorrect diagnoses. To address this issue, internal processes and training for responsible staff are critical. This staff needs to develop an understanding of the impact of statistical inaccuracies and consider appropriate measures to mitigate such effects. This includes continuous reflection on the results of AI models to identify data imbalances that potentially cause bias, as well as to avoid historical or “ingrained” discrimination.

While clear best practices are currently lacking, it is critical that industries develop future guidance to address these challenges. Continued collaboration between technology companies, privacy experts and lawmakers will help promote the development and application of ethical guidelines in the field of AI and privacy.

Governance

The governance of AI systems requires a clear allocation of responsibilities within an organisation. The question of who is responsible for AI governance takes on particular importance in view of the close connection with data protection.

In this context, it is sensible to integrate the data protection officer (“DPO”) as a key player in the governance structure with respect to the privacy implications of GenAI. The expertise of the DPO is important, as the DPO could cover both the data protection and any related ethical implications of the implementation of GenAI.

As part of this, a comprehensive view of AI governance is needed. The complexity of AI requires an interdisciplinary team that considers not only legal but also technical and ethical perspectives. Finally, data protection impact assessments must be taken seriously in terms of effective risk mitigation for data subjects. It can be assumed that privacy regulators take DPIAs very seriously as they serve to review and assess how and whether AI users and or the GenAI providers have mitigated the privacy risks.

Outlook

The points discussed above illustrate the importance of data protection compliance while using GenAI on the EU market. AI users must ensure that they take the necessary steps to implement the appropriate safeguards to ensure GDPR compliance. This does require an understanding of how AI works and its limitations. This should be embedded in staff through internal training and processes. It is advisable to establish accountability processes and form a governance team in the first instance. Employees need to be informed that processing of personal data through GenAI is a sensitive compliance topic and certain processes must be completed to process personal data with GenAI within organizations.

*Thanks to Bird & Bird trainees Lennard Winrich and Dylan Boßmann Everitt for their contributions to this article.