How to Safely Redact PII from Segment Events using Destination Insert Functions and Private AI API
Twilio Segment is a great centralized platform for storing customer data. However, it's unlikely Segment will be your only application that needs the data. Most of us will need to download customer information from Segment, store it in the data lake, propagate the data to reporting software, and populate a few backend databases. All of this may end up terribly wrong.
What's the least fun part of data engineering? I bet it's backfilling data when you made a mistake in the pipeline and need to fix months of data and update reports. Do you know what's even worse? When the data you fix contains Personal Identifiable Information (PII) that you were supposed to remove but you didn't. It is better to remove PII as early as possible in your data pipeline and not worry about it ever again.
Why do we need to remove PII from Segment Events?
Data engineering may occasionally feel like the act of pushing problems from one place to another, but when it comes to PII, pushing problems downstream is the worst thing you can do.
One day, you may need to remove a piece of data because of GDPR or other regulations. On that day, you realize PII is propagated to several downstream systems. The systems are scattered across multiple geo locations or belong to third parties with data centers abroad. You aren't entirely sure whether the Data Processing Agreement accepted by your users allows you to send the data to those countries. Ok, enough. You are reading a tech tutorial, not a horror story for data engineers ;)
Let's remove PII immediately instead of pushing data and future problems downstream. If nobody gets access to PII just because PII happened to be included in the dataset they used, you minimize the number of systems you have to touch when someone requests a change or removal of their data.
How Private AI API removes PII
When you send data to the Private AI API (and you can deploy the service in your infrastructure or use cloud SaaS), Private AI will find and mask all of the personal data. The response you receive will contain the processed, redacted data and all of the entities present in the original data. You can use those entities to revert data replacement. Of course, if the downstream systems of your data pipeline shouldn't have access to personal data, you can drop the entities and propagate only the de-identified text.
Right now, Private AI supports over 50 languages and personal data types (in Private AI API, data types are called entity types, such as family name, passport number, location, etc.), so it is most likely the only tool you need to remove PII.
Using Segment Destination Insert Functions to Remove PII while Downloading Data
Of course, we have to call the Private AI API somewhere. While making the API call a separate step in a data pipeline is always a viable option, in the case of downloading data from Segment, we suggest simplifying the pipeline and using Segment Destination Insert Functions.
Insert Functions are a feature allowing you to write custom JavaScript code to transform and enrich data before sending it to the destination. In our case, we will use those functions to redact PII.
Step-by-Step Guide to Redact PII with Destination Insert Functions and Private AI API
Prerequisites
- Before starting, head to portal.private-ai.com and get yourself an API key.
- Naturally, you will need a Segment account. If you don't have one, create it here.
- We will also need some data. This article uses AI and a Python script to generate fake data. We will use Segment API and Cohere AI. Remember to install both as Python dependencies to run our data generation script.
Generating fake data
In the data-generating script, we create a free-form message with the content generated by AI and some hard-coded personal information. First, we have to import the libraries.
```
import datetime
import cohere
import segment.analytics as analytics
```
Next, we ask Cohere AI to generate an example doctor's note:
```
cohere_client = cohere.Client(‘PUT_YOUR_COHERE_API_KEY_HERE’)
response = cohere_client.chat(model='command-r-plus', message='Write an example of a doctors notes about a patient who suffers from many conditions')
```
Finally, we create a user identification record and track an event for that user.
```
analytics.write_key = ‘PUT_YOUR_SEGMENT_API_KEY_HERE’
analytics.identify('f4ca124298', {
'name': 'Michael Bolton',
'email': 'mbolton@example.com',
'created_at': datetime.datetime.now(),
'account_id': 1
})
analytics.track('f4ca124298', 'Message Sent', {
'title': 'Message between Sam and Allan',
'content': response.text,
})
```
Write JavaScript Code for Destination Insert Function
Before we continue, we must prepare the Insert Function. Note that, in the example, we modify only the event's content property.
In the function, we send a request to the Private AI API to transform the data. When we receive a successful response, we change the event's name to indicate it was modified by the insert function and replace its content with the processed text returned from the API.
The processed text property contains the original data with PII replaced by entity placeholders. Those placeholders and their original values are in the response's "entities" property, but we ignore them. Therefore, the downstream system won't have a way to revert de-identification.
Also, we are using the Private AI default settings to redact all supported data types. However, it's possible to customize the settings and keep some of the information in the text. For example, you could remove people's names but keep city names. For details, visit the Customizing Detection guide.
async function onTrack(event, settings) {
// Learn more at https://segment.com/docs/connections/spec/track/
const endpoint = 'https://api.private-ai.com/demo/v3/process/text';
let response;
try {
response = await fetch(endpoint, {
method: 'POST',
headers: {
'x-api-key': `PUT_YOUR_PRIVATE_AI_API_KEY_HERE`,
'Content-Type': 'application/json'
},
body: JSON.stringify({ text: [event.properties.content] })
});
} catch (error) {
// Retry on connection error
throw new RetryError(error.message);
}
if (response.status >= 500 || response.status === 429) {
// Retry on 5xx (server errors) and 429s (rate limits)
throw new RetryError(`Failed with ${response.status}`);
}
const jsonifiedResponse = await response.json();
event.name = 'event modified from insert function';
event.properties.content = jsonifiedResponse[0].processed_text;
console.log(event);
return event;
}Setup the Destination Insert Function
In Segment, navigate to Connections > Catalog > Functions and click Create Function.
Select the Insert function type on the screen and click "Next: Build Function."
Copy-paste the function code we prepared earlier.
When you finish, click "Next: Configure & Create" to add the function metadata and click "Create Function."
Remember to use the "Connect to destination" option and add your function to the destination where you send the data.
Verify Successful PII Redaction
Remember to verify that the PII has been successfully redacted. If something doesn't work, you want to know about it now, not when you gather a few hundred TBs of data. Remember that you can customize the type of entities removed from the data by Private AI.
Conclusion
Private AI API used inside of Segment Destination Insert Functions offers a powerful solution to redact PII from Segment events before sending them to downstream applications. With Private AI, data engineers can ensure data privacy and compliance with data protection regulations and mitigate the risk of exposing sensitive information to applications that don't need it. Check out the Getting Started guide to learn more.
More to Explore
