Liability & AI Malfunction: An AI System Developer’s Perspective

Sep 1, 2020
Share this post
Sharing to FacebookSharing to LinkedInSharing to XSharing to Email

AI is rapidly being deployed around the world in many different use cases, with few guidelines for manufacturers, developers, and operators to follow. Along with the complexity of creating the technology, there remain many unanswered legal questions. As the CTO of a startup making privacy technologies accessible and easy to integrate, I have spent a significant amount of time thinking at the intersection of privacy, AI, and liability. More recently, I’ve taken a wider lens and have been deliberating over AI and liability overall.

While many developers and engineers would rather stay as far away from thinking about the law as possible (I was one of them), that attitude might very well come back to bite us. As AI becomes more ubiquitous, we will be affected by the way failures in AI systems will be arbitrated, and by who and what is deemed to be at fault for a given failure. I was recently given “Rechtshandbuch Artificial Intelligence und Machine Learning” (in English: Legal Handbook of Artificial Intelligence and Machine Learning”) an excellent book on the legal implications of AI that covers the matter of liability.

Liability & AI: An AI System Developer’s Perspective

The book, edited and published by Markus Kaulartz and Tom Braegelmann, provides an overview of one of the most significant legal conundrums our societies are facing; namely, that laws must keep up with technology, but must do so in such a way that does not hinder innovation. We look specifically at Chapter 4.2, written by Maren Wöbbeking, which explores the careful balance that must exist between legal liability and the technologies behind AI. Key to achieving this balance are accurate risk measurements associated with autonomous systems and carefully attributing responsibility to system manufacturers, system operators, and bystanders.

Risks of Autonomous Systems

When considering legal liability around autonomous vehicles, for example, one must take into account both (1) the processes for autonomous decision-making and (2) the traceability and explainability of the decisions. It becomes particularly obvious why proper decision-making traceability is essential in order to determine liability when we specifically consider the disastrous effects that human-falsified input data might have on an outcome.

Wöbbeking quite rightly points out the implications of human interactions with autonomous systems when it comes to determining liability: the majority of autonomous systems still rely heavily on supervised or semi-supervised learning (meaning that one or many humans have to inform the system on which inputs correspond to which outputs at the training phase), and there is a growing literature in adversarial machine learning that’s dedicated to thwarting the many ways in which inputs to an AI system during both inference and training can lead to completely unexpected outputs/outcomes.

How can one pinpoint failure in such a system?

Broadly speaking, end-to-end approaches using a singular, large network yield the best results. Think of a complex system such as an autonomous car, with multiple sensory inputs such as LIDAR & cameras, what would happen if the camera calibration were to drift? It might be a better idea to build separate networks with clearly defined inputs & outputs for this reason — but then, this might impact accuracy, increasing the risk of failure.

In addition to considering potential attacks to AI models through the use of deceptive inputs or even to slight equipment malfunctions that detrimentally modify input signals, as Wöbbeking mentions, it is a human’s responsibility to determine that an autonomous system is being used in the same environment it was trained to run in (i.e., was created for).

Another aspect of liability law should consider whether appropriate measures were taken in order to mitigate the risks of using AI. Under the constrained environments that AI are currently deployed in, they seem to actually reduce the risk of performing certain tasks when compared to a human performing the same task. Risk mitigation will become increasingly relevant and crucial if autonomous systems are to be deployed in more varied and less constrained environments.

Strict Liability

Wöbbeking proceeds to discuss which questions might be covered by existing or new liability laws. One particularly difficult question to answer is whether the risks inherent in the state-of-the-art autonomous systems which otherwise reduce risk should be borne by the injured party, the operator, or the manufacturer.

One possible framework that could apply to autonomous systems is that of strict liability.

“Strict liability differs from ordinary negligence because strict liability establishes liability without fault. In other words, when a defendant is held strictly liable for harm caused to the plaintiff, he is held liable simply because the injury happened.”

The manufacturer of the autonomous system is the party who has the most knowledge and control over the risks and also the most incentive to cut costs. Allocation of responsibility to the manufacturer therefore becomes an extra incentive for them to thoroughly evaluate and mitigate risk. However, holding the manufacturers strictly liable has the very real risk of inhibiting innovation.

While manufacturers have a lot more control than operators, they cannot always control whether the operators have deployed a system according to the instructions. Operators themselves often have a choice between either using or not using an autonomous system they are provided with. Their obligations, while limited, are crucial: reducing risk by using the system as instructed.

Proportional Liability

An alternative framework that could be applied to autonomous systems is that of proportional liability.“Proportional Liability — refers to an arrangement for the assignment of liability in which each member of a group is held responsible for the financial results of the group in proportion to its participation.”

Programmers, data providers, manufacturers, all further developers, third parties manipulating the system, operators, and users all influence the system and contribute to a possible wrong decision. Taking multi-causality into account, while more complex than blaming a single party, might be the right way to assess liability. Propositional liability might bypass any innovation inhibition by the manufacturers, while still holding irresponsible manufacturers more accountable for their lack of risk mitigation. It would help increase the likelihood that operators will take the necessary precautions around autonomous system use.

Risk recovery

Finally, Wöbbeking postulates that a possible regulation for the allocation of any recourse and liability risks would be to pool the risks through a community insurance solution. Possibly a solution similar to social security law. This would probably avoid the complexities associated with liability law and (among other benefits) would also mitigate the disadvantages of a specific risk allocation.

The future of AI

Whatever the legal future of AI looks like, there are some clear takeaways for AI system developers on what we can do now to prepare ourselves; namely:

  1. Clearly document the design & testing process;
  2. Follow software engineering best practices — e.g. no dynamic allocation of memory and no use of recursion;
  3. Take great care in designing validation and test sets. And make sure a new test set is used after each major system update;
  4. Account for bias.

Join us for more discussions on artificial intelligence on LinkedIn, Twitter, and Youtube

Data Left Behind: AI Scribes’ Promises in Healthcare

Data Left Behind: Healthcare’s Untapped Goldmine

The Future of Health Data: How New Tech is Changing the Game

Why is linguistics essential when dealing with healthcare data?

Why Health Data Strategies Fail Before They Start

Private AI to Redefine Enterprise Data Privacy and Compliance with NVIDIA

EDPB’s Pseudonymization Guideline and the Challenge of Unstructured Data

HHS’ proposed HIPAA Amendment to Strengthen Cybersecurity in Healthcare and how Private AI can Support Compliance

Japan's Health Data Anonymization Act: Enabling Large-Scale Health Research

What the International AI Safety Report 2025 has to say about Privacy Risks from General Purpose AI

Private AI 4.0: Your Data’s Potential, Protected and Unlocked

How Private AI Facilitates GDPR Compliance for AI Models: Insights from the EDPB's Latest Opinion

Navigating the New Frontier of Data Privacy: Protecting Confidential Company Information in the Age of AI

Belgium’s Data Protection Authority on the Interplay of the EU AI Act and the GDPR

Enhancing Compliance with US Privacy Regulations for the Insurance Industry Using Private AI

Navigating Compliance with Quebec’s Act Respecting Health and Social Services Information Through Private AI’s De-identification Technology

Unlocking New Levels of Accuracy in Privacy-Preserving AI with Co-Reference Resolution

Strengthened Data Protection Enforcement on the Horizon in Japan

How Private AI Can Help to Comply with Thailand's PDPA

How Private AI Can Help Financial Institutions Comply with OSFI Guidelines

The American Privacy Rights Act – The Next Generation of Privacy Laws

How Private AI Can Help with Compliance under China’s Personal Information Protection Law (PIPL)

PII Redaction for Reviews Data: Ensuring Privacy Compliance when Using Review APIs

Independent Review Certifies Private AI’s PII Identification Model as Secure and Reliable

To Use or Not to Use AI: A Delicate Balance Between Productivity and Privacy

To Use or Not to Use AI: A Delicate Balance Between Productivity and Privacy

News from NIST: Dioptra, AI Risk Management Framework (AI RMF) Generative AI Profile, and How PII Identification and Redaction can Support Suggested Best Practices

Handling Personal Information by Financial Institutions in Japan – The Strict Requirements of the FSA Guidelines

日本における金融機関の個人情報の取り扱い - 金融庁ガイドラインの要件

Leveraging Private AI to Meet the EDPB’s AI Audit Checklist for GDPR-Compliant AI Systems

Who is Responsible for Protecting PII?

How Private AI can help the Public Sector to Comply with the Strengthening Cyber Security and Building Trust in the Public Sector Act, 2024

A Comparison of the Approaches to Generative AI in Japan and China

Updated OECD AI Principles to keep up with novel and increased risks from general purpose and generative AI

Is Consent Required for Processing Personal Data via LLMs?

The evolving landscape of data privacy legislation in healthcare in Germany

The CIO’s and CISO’s Guide for Proactive Reporting and DLP with Private AI and Elastic

The Evolving Landscape of Health Data Protection Laws in the United States

Comparing Privacy and Safety Concerns Around Llama 2, GPT4, and Gemini

How to Safely Redact PII from Segment Events using Destination Insert Functions and Private AI API

WHO’s AI Ethics and Governance Guidance for Large Multi-Modal Models operating in the Health Sector – Data Protection Considerations

How to Protect Confidential Corporate Information in the ChatGPT Era

Unlocking the Power of Retrieval Augmented Generation with Added Privacy: A Comprehensive Guide

Leveraging ChatGPT and other AI Tools for Legal Services

Leveraging ChatGPT and other AI tools for HR

Leveraging ChatGPT in the Banking Industry

Law 25 and Data Transfers Outside of Quebec

The Colorado and Connecticut Data Privacy Acts

Unlocking Compliance with the Japanese Data Privacy Act (APPI) using Private AI

Tokenization and Its Benefits for Data Protection

Private AI Launches Cloud API to Streamline Data Privacy

Processing of Special Categories of Data in Germany

End-to-end Privacy Management

Privacy Breach Reporting Requirements under Law25

Migrating Your Privacy Workflows from Amazon Comprehend to Private AI

A Comparison of the Approaches to Generative AI in the US and EU

Benefits of AI in Healthcare and Data Sources (Part 1)

Privacy Attacks against Data and AI Models (Part 3)

Risks of Noncompliance and Challenges around Privacy-Preserving Techniques (Part 2)

Enhancing Data Lake Security: A Guide to PII Scanning in S3 buckets

The Costs of a Data Breach in the Healthcare Sector and its Privacy Compliance Implications

Navigating GDPR Compliance in the Life Cycle of LLM-Based Solutions

What’s New in Version 3.8

How to Protect Your Business from Data Leaks: Lessons from Toyota and the Department of Home Affairs

New York's Acceptable Use of AI Policy: A Focus on Privacy Obligations

Safeguarding Personal Data in Sentiment Analysis: A Guide to PII Anonymization

Changes to South Korea’s Personal Information Protection Act to Take Effect on March 15, 2024

Australia’s Plan to Regulate High-Risk AI

How Private AI can help comply with the EU AI Act

Comment la Loi 25 Impacte l'Utilisation de ChatGPT et de l'IA en Général

Endgültiger Entwurf des Gesetzes über Künstliche Intelligenz – Datenschutzpflichten der KI-Modelle mit Allgemeinem Verwendungszweck

How Law25 Impacts the Use of ChatGPT and AI in General

Is Salesforce Law25 Compliant?

Creating De-Identified Embeddings

Exciting Updates in 3.7

EU AI Act Final Draft – Obligations of General-Purpose AI Systems relating to Data Privacy

FTC Privacy Enforcement Actions Against AI Companies

The CCPA, CPRA, and California's Evolving Data Protection Landscape

HIPAA Compliance – Expert Determination Aided by Private AI

Private AI Software As a Service Agreement

EU's Review of Canada's Data Protection Adequacy: Implications for Ongoing Privacy Reform

Acceptable Use Policy

ISO/IEC 42001: A New Standard for Ethical and Responsible AI Management

Reviewing OpenAI's 31st Jan 2024 Privacy and Business Terms Updates

Comparing OpenAI vs. Azure OpenAI Services

Quebec’s Draft Regulation Respecting the Anonymization of Personal Information

Version 3.6 Release: Enhanced Streaming, Auto Model Selection, and More in Our Data Privacy Platform

Brazil's LGPD: Anonymization, Pseudonymization, and Access Requests

LGPD do Brasil: Anonimização, Pseudonimização e Solicitações de Acesso à Informação

Canada’s Principles for Responsible, Trustworthy and Privacy-Protective Generative AI Technologies and How to Comply Using Private AI

Private AI Named One of The Most Innovative RegTech Companies by RegTech100

Data Integrity, Data Security, and the New NIST Cybersecurity Framework

Safeguarding Privacy with Commercial LLMs

Cybersecurity in the Public Sector: Protecting Vital Services

Privacy Impact Assessment (PIA) Requirements under Law25

Elevate Your Experience with Version 3.5

Fine-Tuning LLMs with a Focus on Privacy

GDPR in Germany: Challenges of German Data Privacy (Part 2)

Comply with US Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence using Private AI

How to Comply with EU AI Act using PrivateGPT