Inbound Customer Data Ingestion FAQ
Frequently asked questions about bringing offline data into Audience Manager.
Can you summarize the onboarding process?
The onboarding process consists of 2 core components described in Batch Data Transfer Process Described . These involve:
- ID synchronization
- Inbound Data File ( .sync file or .overwrite file)
Below is a list of questions and answers you might find helpful after reviewing the documentation.
The examples in this section are simplified or shortened for brevity and demonstration purposes. Refer to the Inbound Data Ingestion documentation for detailed specifications about file formats and syntax.
Can you summarize the deployment process?
We recommend the following:
- Work with your data provider to format the daily inbound data file according to Adobe specifications.
- Transfer a test data file to Adobe for format verification.
- Work with your Adobe consultant to produce a taxonomy suitable for interpreting the contents of the data file.
- In the staging/development environment, confirm that the ID sync is configured to properly pick up the data provider's visitor ID and transfer it to the Audience Manager servers in realtime.
- Deploy DIL/ID sync to production. The ID sync will already be configured as a module within the DIL code by your Adobe consultant.
- Transfer production data files to Audience Manager. Given the dependencies on ID sync mappings, it might make sense to begin transferring data up to one week after production-code deployment, although you can start transferring the data files as soon as code goes into production.
What FTP mode should I use to transfer compressed or encrypted files?
Can I upload an inbound data file (.sync or .overwrite file) before deploying Audience Manager code into production?
- If the data provider is configured to use Profile Link for cross-device targeting, the data available for targeting shortly after an ID sync identifies to the matching Audience Manager visitor ID.
- If the data provider is not configured to use the Profile Link feature, Audience Manager processes only the data for visitor IDs in the inbound data file that have been previously synced/matched back to an Audience Manager visitor ID.
Consider the following use cases in which the data provider is not configured to use Profile Merge:
On Monday, a visitor identified in the CRM database as visitor ABC logs in, which initiates a client-side ID sync. Audience Manager stores the mapping of visitor ABC to Audience Manager visitor 123.
On Tuesday, the CRM database transfers a data file ( .sync ) to the Audience Manager server with the following record:
In this case, Audience Manager :
On Monday, the CRM database pushes a data file ( .sync ) to the Audience Manager server with the following record:
Audience Manager does not have a record of this visitor (or an associated visitor ID) so this record is not processed.
On Tuesday, visitor DEF logs in. This action initiates the first client-side ID sync for that visitor. This action maps visitor DEF to Audience Manager ID 456. However, this visitor does not have CRM data associated with their profile. As a result, Audience Manager does not go back and reprocess old files.
On Wednesday, the CRM database pushes another data file to the Audience Manager server with the following record:
In this case, Audience Manager :
On Monday, the Audience Manager server receives two files with the following records:
.sync file containing:
.overwrite file containing:
Audience Manager holds a mapped record of visitor JKL to ID 789, from a previous ID sync.
In this case, Audience Manager :
What time of day should I transfer my file?
Audience Manager checks for and processes files multiple times throughout the day. Upload your data whenever you're ready.
How long does it take before data from an uploaded file is available for targeting?
Data is available for targeting after 48 hours. Also, do not interpret the "successful upload" email as a statement that the data is available. This only means that Audience Manager has picked up the file and completed the first step of processing.
How often should I send files and should these be full or incremental files?
As a best practice, send an incremental file once per day for new visitors and for visitors whose data has changed. Many Audience Manager customers send a full file once per month. However, these file intervals and increments are flexible. You should send data in increments and at times that make sense for you.
How long does Audience Manager keep my files on the server?
FTP files are removed after they've been processed. S3 files are removed after 30-days. Files that cannot be processed due to format, syntax, or other errors are removed. See also, Privacy and Data Retention FAQ .
What's the difference between full and incremental files?
- Full: A full file overwrites all of your existing visitor profiles and replaces them with the data in your file. Full files are identified by the .overwrite tag appended to the file name. You can use a .overwrite file to reset visitor traits or remove stale, obsolete traits.The .overwrite files only overwrite Audience Manager profile data associated to this data provider. In other words, all Adobe Analytics data associated to the visitor remains intact after a .overwrite file has been processed.
- Incremental: An incremental file appends new data to your existing visitor profiles. Incremental files are identified by the .sync tag appended to the file name. Sending in an incremental file does not erase or overwrite existing profiles.
The following use cases demonstrate how these file types affect stored visitor profiles.
Incremental and Full
For more information about full and incremental file types, see:
What happens if I send in a file with IDs for visitors that have never performed the on-page ID sync?
During processing, Audience Manager simply skips that record and moves on to the next. If a DPID (Data provider ID) is set up as a cross-device DPID, data that is ingested before an ID sync is saved and is available for use shortly after the ID sync occurs.
What is the time stamp, what is it for, and can you provide an example?
Time stamps are used for logging and record keeping. They are required by the syntax used for a properly formatted inbound file name. See:
What is a Data Provider ID (DPID) and how do I get it?
Your Adobe consultant will assign a three-digit or four-digit DPID to your particular data source. This ID is unique and does not change.
How large can the daily data files be?
Does Audience Manager support file compression?
The primary key in my data source database is an email address. Is that considered personally identifiable information?
Yes. Audience Manager does not store email addresses in our database. Visitors should be assigned a random ID or a one-way-hashed version of the email address prior to initiating ID syncs.
Are the data file contents case-sensitive? How about the ID sync?
There are two basic components of a data file: A User ID (see User ID in File Variables Defined ) and profile data, usually in the form of key-value pairs or codes. The User ID is case-sensitive. Generally, profile or key-value data is not case-sensitive.
Should I use FTP or Amazon S3 to transfer files?
As best practice, we recommend Amazon S3 because the process is simpler. Audience Manager transfers FTP files to S3 regardless, so the process is more streamlined if you drop the files on Amazon S3 yourself. What's more, customers uploading simultaneously to FTP share the FTP's bandwidth, so they should expect slower upload speeds. Amazon S3 is also replicated and distributed, so it is generally safer and more reliable than an FTP server. For more information, see About Amazon S3 .
How does Audience Manger process inbound files?
Audience Manager uses Amazon Simple Queue Service (SQS) for inbound data processing. Here is how this works:
- Audience Manager customers upload their inbound data to an Amazon S3 bucket.
- The data enters the Amazon SQS queue, waiting to be processed by Audience Manager.
- Audience Manager reads up to 119000 entries from the Amazon SQS queue and splits them in up to 3 batches. Files in each batch get processed simultaneously.
I need to upload multiple files at the same time. Will the files be processed simultaneously?
It depends. Audience Manager reads up to 119000 entries from the Amazon SQS queue and splits them in up to 3 batches. Your files will be processed simultaneously only if they end up in the same batch. However, due to the high amount of data ingested by Audience Manager on a daily basis, we cannot guarantee any file processing order.