Privacy Attacks against Data and AI Models (Part 3)

Kathrin Gardhouse
Apr 10, 2024
Share this post
Sharing to FacebookSharing to LinkedInSharing to XSharing to Email

Privacy Attacks against Data and AI Models

Part 1 of this series covers the benefits of AI models in healthcare and the data sources available for training such models. Part 2 addresses privacy compliance challenges as well as challenges surrounding privacy-preserving techniques employed during AI development. This last part of our series gets a bit more technical when discussing attacks on AI Models with the goal of obtaining illicit access to the data, as well as re-identification attacks launched against the data used by AI developers to train the model.

For a recent resource discussing further attacks against AI models and data, with a broader focus on privacy and security, see Security and Privacy Challenges of Large Language Models: A Survey.

It is fairly obvious why an attacker would be interested in gaining access to an AI model or its training data if the model is trained on health data. These data are, for the most part, not publicly available as we saw in Part 1. (We discuss the value of health data here.) But note that the attackers’ motivation persists even when the model is trained on data scraped from public Internet websites, as is the case for all popular large language models (LLMs).

The reason is that the data used for fine-tuning an LLM are meticulously annotated, that is, labelled, which increases the value not only for its primary purpose of improving model performance but also for the attacker who might use such organized and high-quality data sets for malicious purposes. Well-labeled data sets can be exploited to train competing models, conduct targeted disinformation campaigns, or even develop sophisticated algorithms for bypassing security systems. The process of annotating data involves categorizing or tagging them with information that helps the AI model understand and learn from the data more effectively. This requires significant intellectual effort and resources, which is why they become a lucrative target.

Attack Methods

In this section, we delve into various forms of attacks on AI models, highlighting their mechanisms, implications, and the stages at which they occur.

We begin with model attacks to explain how malicious actors may gain access to the data the model has been trained on and proceed to re-identification attacks to clarify how attackers can use data sets to which they have obtained access to identify specific individuals even if the data have been de-identified.

Attacks on AI Models

Attacks on machine learning models seek to unveil the underlying structures, algorithms, or data that inform AI systems. These can have severe implications, especially when proprietary or sensitive data is involved.

Model Extraction Attacks

Model extraction attacks, often categorized as black-box attacks, are predicated on the attacker's ability to observe the outputs generated by a machine learning model in response to chosen inputs. The primary aim is to replicate the functionality of the AI model without access to its internal architecture or training data. By systematically probing the model with inputs and analyzing the corresponding outputs, attackers can infer the model's decision boundaries, logic, and behavior.

The repercussions of a successful model extraction attack include the unauthorized duplication of the model, which could infringe on intellectual property rights or lead to the circumvention of access controls. Moreover, by reconstructing a facsimile of the model, attackers might glean sensitive information implicitly captured by the original model, such as biases or specific data patterns it has learned.

Shadow Model Attacks

Shadow model attacks involve the construction of a parallel "shadow" model. The attacker trains this shadow model on a data set assembled or hypothesized to closely mimic the original model's training data. The objective is not merely to replicate the target model's functionality but to use the shadow model as a probe to understand the target model's training data and infer its internal structure.

This approach enables the attacker to simulate the target model's behavior under controlled conditions, facilitating the exploration of more sophisticated attack vectors. For example, insights gained from the shadow model could be used to design inputs that are more likely to trick the target model into making incorrect predictions or revealing information about the data it was trained on, enhancing the efficacy of subsequent attacks, including membership inference attacks.

Membership Inference Attacks

The goal of a membership inference attack is to determine whether a specific individual's data was used in the training set of a machine learning model. By submitting queries to a model, an attacker may accurately guess that a particular individual's health records were used to train a medical diagnosis model based on the model's responses.

Identifying an individual's data in a model's training set can reveal private information, such as their participation in a particular medical study which allows for an inference regarding their health condition.

Model Inversion Attacks

In a model inversion attack, malicious actors use model predictions to reconstruct training data, threatening the confidentiality of sensitive records. To clarify the difference between model inversion and membership inference, note that model inversion attacks aim at reconstructing input data or attributes, while membership inference attacks focus on identifying data set participation.

Property Inference Attacks

In property inference attacks, attackers analyze the model’s outputs, behavior, or even structure to infer properties of the training data that are not explicitly revealed by the model’s primary function. This can be achieved even without direct access to the training data or detailed knowledge of the model internals. An attacker might use statistical methods or machine learning to analyze patterns in the model’s predictions across various inputs to deduce hidden characteristics of the training data.

Imagine a company developing a machine learning model to predict potential job candidates' success based on their resumes and past job performance data. The model is designed to assist HR in the recruitment process by highlighting candidates likely to perform well in certain roles. However, the model is trained on detailed employment records, including performance evaluations, job histories, and demographic information.

An attacker interested in uncovering discriminatory practices or biases within the company’s hiring process could analyze the model’s predictions across a range of synthetic or real-world resumes. By carefully studying the model’s outputs, the attacker could infer properties about the training data that the company did not intend to disclose. For example, they might discover that the model tends to predict higher success rates for candidates from certain demographic groups or with specific characteristics that are irrelevant to job performance, indicating potential bias in the training data or the model itself. Attackers could exploit this to infer personal information about individuals in the data set.

Gradient Leakage Attack

Gradient leakage attacks constitute a significant privacy threat in the realm of machine learning, particularly under federated learning environments where models are trained across distributed data sets. These attacks exploit the gradient information—derivatives of the loss function with respect to the model parameters—shared across the network to update the global model. By accessing or inferring these gradients, attackers can potentially reconstruct private training data, exposing sensitive information. The risk is notably pronounced in deep learning models trained with optimization algorithms that utilize gradients to minimize loss functions. Although such attacks have primarily been demonstrated on image data sets, emerging research has begun exploring their feasibility against language models, including LLMs.

Re-identification Attacks

Re-identification attacks focus on unauthorized access to or disclosure of personal information. Such access can be gained by intercepting a data transfer because data is inadequately encrypted at rest or as a result of the model attacks discussed above.

As the name suggests, re-identification attacks counter the process of de-identification (more on that below) by matching records that have undergone de-identification with identifiable information contained in the data set in question. For instance, attackers could use publicly available records that have been stripped of direct identifiers and published under the assumption that the remaining data cannot be linked back to individuals. With the help of the remaining indirect- or quasi-identifiers, the attacker can uncover patients' identities by combining these records with the data obtained from the attack victim.

A well-documented real-world example of a re-identification attack involved the personal health information of the governor of Massachusetts in the 1990s. Researchers were given “anonymized” health insurance data for the purpose of public health research. This data set was considered safe for release because it had been stripped of all direct identifiers.

However, the researchers were able to re-identify the governor's records by cross-referencing the allegedly anonymized health data with the voter registration records for the city of Cambridge, Massachusetts. Voter registration lists are public records and contain names, addresses, and birth dates. By matching the quasi-identifiers (zip code, birth date, and gender) present in both the health records and the voter list, the researchers could pinpoint the governor’s health information, effectively demonstrating the vulnerability of the data to re-identification attacks.

Note that this attack has been criticized as overstating the ease with which re-identification is possible. An important aspect in this case was that the governor was a public figure and had undergone hospitalization, about which the media widely reported. Nevertheless, with more data being available online and technologies getting better at making inferences based on such data, re-identification risks must be taken seriously.

Conclusion

We have looked at a myriad of attacks on AI models, some aimed at the model itself and others at the data. Understanding these various forms of attacks on AI models is crucial for developing more secure AI systems. As AI continues to advance, the strategies for protecting against these attacks must evolve accordingly, ensuring that privacy and security considerations keep pace with technological innovation. This ongoing challenge underscores the need for robust privacy-preserving methodologies and a comprehensive approach to AI system design and deployment.

Our favorite technique that has a leg up on most attacks aimed at ML models is the de-identification of data sets. Attackers can gain access to data or do to models whatever they like; no privacy violation is possible if the data does not contain personal information in the first place.

Such data sets remain vulnerable to re-identification attacks. We have seen in the case study involving the governor of Massachusetts that the removal of direct identifiers alone often does not suffice to render data safe. However, stripping direct and indirect identifiers such as names, social security numbers, dates of birth, gender, and race from a data set already improves the security of the data a lot. Sometimes, indirect identifiers cannot be removed as these data points are relevant for data analytics in healthcare or research settings, for example.

Furthermore, with continuing advances in re-identification technologies, the removal of direct and some or even all indirect identifiers does not necessarily mean that individuals can no longer be identified. More is required to achieve irreversible de-identification or proper anonymization. For a deep dive into this topic, download the E-Book Privacy-Enhancing Technologies and the GDPR.

That said, if the use case allows it, that is if data utility can be preserved without direct and indirect identifiers present, it is highly recommended to remove them from the training data. Given the vast amount of data required, this task can be daunting, particularly as it concerns unstructured data. Private AI can help with that.

Private AI’s technology can identify and report on personal identifiers included in large unstructured data sets and replace them with synthetic data or placeholders. There are over 50 entity types for which our technology is optimized, including protected health information. For many use cases, the de-identification approach that relies on context-aware algorithms trained by data experts in multiple languages is able to preserve data utility while maximizing data privacy.

This technology is not only useful for model developers but also further down the value chain. When businesses are concerned about violating privacy rights when employees include personal data in their prompts sent to an external model, Private AI’s PrivateGPT can be deployed to intercept the prompt, filter out the personal data and re-inject it automatically into the response for a seamless user experience. Test PrivateGPT for free here.

Data Left Behind: AI Scribes’ Promises in Healthcare

Data Left Behind: Healthcare’s Untapped Goldmine

The Future of Health Data: How New Tech is Changing the Game

Why is linguistics essential when dealing with healthcare data?

Why Health Data Strategies Fail Before They Start

Private AI to Redefine Enterprise Data Privacy and Compliance with NVIDIA

EDPB’s Pseudonymization Guideline and the Challenge of Unstructured Data

HHS’ proposed HIPAA Amendment to Strengthen Cybersecurity in Healthcare and how Private AI can Support Compliance

Japan's Health Data Anonymization Act: Enabling Large-Scale Health Research

What the International AI Safety Report 2025 has to say about Privacy Risks from General Purpose AI

Private AI 4.0: Your Data’s Potential, Protected and Unlocked

How Private AI Facilitates GDPR Compliance for AI Models: Insights from the EDPB's Latest Opinion

Navigating the New Frontier of Data Privacy: Protecting Confidential Company Information in the Age of AI

Belgium’s Data Protection Authority on the Interplay of the EU AI Act and the GDPR

Enhancing Compliance with US Privacy Regulations for the Insurance Industry Using Private AI

Navigating Compliance with Quebec’s Act Respecting Health and Social Services Information Through Private AI’s De-identification Technology

Unlocking New Levels of Accuracy in Privacy-Preserving AI with Co-Reference Resolution

Strengthened Data Protection Enforcement on the Horizon in Japan

How Private AI Can Help to Comply with Thailand's PDPA

How Private AI Can Help Financial Institutions Comply with OSFI Guidelines

The American Privacy Rights Act – The Next Generation of Privacy Laws

How Private AI Can Help with Compliance under China’s Personal Information Protection Law (PIPL)

PII Redaction for Reviews Data: Ensuring Privacy Compliance when Using Review APIs

Independent Review Certifies Private AI’s PII Identification Model as Secure and Reliable

To Use or Not to Use AI: A Delicate Balance Between Productivity and Privacy

To Use or Not to Use AI: A Delicate Balance Between Productivity and Privacy

News from NIST: Dioptra, AI Risk Management Framework (AI RMF) Generative AI Profile, and How PII Identification and Redaction can Support Suggested Best Practices

Handling Personal Information by Financial Institutions in Japan – The Strict Requirements of the FSA Guidelines

日本における金融機関の個人情報の取り扱い - 金融庁ガイドラインの要件

Leveraging Private AI to Meet the EDPB’s AI Audit Checklist for GDPR-Compliant AI Systems

Who is Responsible for Protecting PII?

How Private AI can help the Public Sector to Comply with the Strengthening Cyber Security and Building Trust in the Public Sector Act, 2024

A Comparison of the Approaches to Generative AI in Japan and China

Updated OECD AI Principles to keep up with novel and increased risks from general purpose and generative AI

Is Consent Required for Processing Personal Data via LLMs?

The evolving landscape of data privacy legislation in healthcare in Germany

The CIO’s and CISO’s Guide for Proactive Reporting and DLP with Private AI and Elastic

The Evolving Landscape of Health Data Protection Laws in the United States

Comparing Privacy and Safety Concerns Around Llama 2, GPT4, and Gemini

How to Safely Redact PII from Segment Events using Destination Insert Functions and Private AI API

WHO’s AI Ethics and Governance Guidance for Large Multi-Modal Models operating in the Health Sector – Data Protection Considerations

How to Protect Confidential Corporate Information in the ChatGPT Era

Unlocking the Power of Retrieval Augmented Generation with Added Privacy: A Comprehensive Guide

Leveraging ChatGPT and other AI Tools for Legal Services

Leveraging ChatGPT and other AI tools for HR

Leveraging ChatGPT in the Banking Industry

Law 25 and Data Transfers Outside of Quebec

The Colorado and Connecticut Data Privacy Acts

Unlocking Compliance with the Japanese Data Privacy Act (APPI) using Private AI

Tokenization and Its Benefits for Data Protection

Private AI Launches Cloud API to Streamline Data Privacy

Processing of Special Categories of Data in Germany

End-to-end Privacy Management

Privacy Breach Reporting Requirements under Law25

Migrating Your Privacy Workflows from Amazon Comprehend to Private AI

A Comparison of the Approaches to Generative AI in the US and EU

Benefits of AI in Healthcare and Data Sources (Part 1)

Privacy Attacks against Data and AI Models (Part 3)

Risks of Noncompliance and Challenges around Privacy-Preserving Techniques (Part 2)

Enhancing Data Lake Security: A Guide to PII Scanning in S3 buckets

The Costs of a Data Breach in the Healthcare Sector and its Privacy Compliance Implications

Navigating GDPR Compliance in the Life Cycle of LLM-Based Solutions

What’s New in Version 3.8

How to Protect Your Business from Data Leaks: Lessons from Toyota and the Department of Home Affairs

New York's Acceptable Use of AI Policy: A Focus on Privacy Obligations

Safeguarding Personal Data in Sentiment Analysis: A Guide to PII Anonymization

Changes to South Korea’s Personal Information Protection Act to Take Effect on March 15, 2024

Australia’s Plan to Regulate High-Risk AI

How Private AI can help comply with the EU AI Act

Comment la Loi 25 Impacte l'Utilisation de ChatGPT et de l'IA en Général

Endgültiger Entwurf des Gesetzes über Künstliche Intelligenz – Datenschutzpflichten der KI-Modelle mit Allgemeinem Verwendungszweck

How Law25 Impacts the Use of ChatGPT and AI in General

Is Salesforce Law25 Compliant?

Creating De-Identified Embeddings

Exciting Updates in 3.7

EU AI Act Final Draft – Obligations of General-Purpose AI Systems relating to Data Privacy

FTC Privacy Enforcement Actions Against AI Companies

The CCPA, CPRA, and California's Evolving Data Protection Landscape

HIPAA Compliance – Expert Determination Aided by Private AI

Private AI Software As a Service Agreement

EU's Review of Canada's Data Protection Adequacy: Implications for Ongoing Privacy Reform

Acceptable Use Policy

ISO/IEC 42001: A New Standard for Ethical and Responsible AI Management

Reviewing OpenAI's 31st Jan 2024 Privacy and Business Terms Updates

Comparing OpenAI vs. Azure OpenAI Services

Quebec’s Draft Regulation Respecting the Anonymization of Personal Information

Version 3.6 Release: Enhanced Streaming, Auto Model Selection, and More in Our Data Privacy Platform

Brazil's LGPD: Anonymization, Pseudonymization, and Access Requests

LGPD do Brasil: Anonimização, Pseudonimização e Solicitações de Acesso à Informação

Canada’s Principles for Responsible, Trustworthy and Privacy-Protective Generative AI Technologies and How to Comply Using Private AI

Private AI Named One of The Most Innovative RegTech Companies by RegTech100

Data Integrity, Data Security, and the New NIST Cybersecurity Framework

Safeguarding Privacy with Commercial LLMs

Cybersecurity in the Public Sector: Protecting Vital Services

Privacy Impact Assessment (PIA) Requirements under Law25

Elevate Your Experience with Version 3.5

Fine-Tuning LLMs with a Focus on Privacy

GDPR in Germany: Challenges of German Data Privacy (Part 2)

Comply with US Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence using Private AI

How to Comply with EU AI Act using PrivateGPT