Data ingestion notifications
The process of ingesting data into Adobe Experience Platform is comprised of multiple steps. Once you identify data files that need to be ingested into Platform, the ingestion process begins and each step occurs consecutively until the data is either successfully ingested or fails. The ingestion process can be initiated using the Adobe Experience Platform Data Ingestion API or using the Experience Platform user interface.
Data loaded into Platform must go through multiple steps in order to reach its destination, the Data Lake or the Real-time Customer Profile data store. Each step involves processing the data, validating the data, and then storing the data before passing it on to the next step. Depending on the amount of data being ingested, this can become a time consuming process and there is always a chance of the process failing due to validation, semantics, or processing errors. In the event of a failure, the data issues need to fixed and then the entire ingestion process must be restarted using the corrected data files.
To assist in monitoring the ingestion process, Experience Platform makes it possible to subscribe to a set of events that are published by each step of the process, notifying you to the status of the ingested data and any possible failures.
Available status notification events
Below is a list of available data ingestion status notifications that you can subscribe to.
There is only one event topic provided for all data ingestion notifications. In order to distinguish between different statuses, the event code can be used.
Ingestion - Batch succeeded
Ingestion - Batch failed
Real-time Customer Profile
Profile service - Data load batch Succeeded
Real-time Customer Profile
Profile service - Data load batch failed
Identity graph - Data load batch succeeded
Identity graph - Data load batch failed
Notification payload schema
The data ingestion notification event schema is an Experience Data Model (XDM) schema containing fields and values that provide details regarding the status of the data being ingested. Please visit the public XDM GitHub repo in order to view the latest notification payload schema .
Subscribe to data ingestion status notifications
Through Adobe I/O Events , you can subscribe to multiple notification types using webhooks. The sections below outline the steps for subscribing to Platform notifications for data ingestion events using Adobe Developer Console.
Add Experience Platform events to the project
Once you have created a new project, navigate to that project's overview screen. From here, click Add event .
The Add events dialog appears. Click Experience Platform to filter the list of available options, then click Platform notifications before clicking Next .
The next screen displays a list of event types to subscribe to. Select Data ingestion notification , then click Next .
The next screen prompts you to create a JSON Web Token (JWT). You are given the option to automatically generate a key pair, or upload your own public key generated in the terminal.
For the purposes of this tutorial, the first option is followed. CLick the option box for Generate a key pair , then click the Generate keypair button in the bottom-right corner.
When the key pair generates, it is automatically downloaded by the browser. You must store this file yourself as it is not persisted in the Developer Console.
The next screen allows you to review the details of the newly generated key pair. Click Next to continue.
In the next screen, provide a name and description for the event registration. Best practice is to create a unique, easily identifiable name to help differentiate this event registration from others on the same project.
Further down on the same screen, you can optionally configure how to receive events. Webhook allows you to provide a custom webhook address to receive events, whereas Runtime action allows you to do the same using Adobe I/O Runtime .
This tutorial skips this optional configuration step. Once you are finished, click Save configured events to complete the event registration.
The details page for the newly created event registration appears, where you can review received events, perform debug tracing, and edit its configuration.
Once you have registered Platform notifications to your project, you can view received events from the project dashboard. Refer to the Tracing Adobe I/O Events guide for detailed instructions on how to trace your events.