Data ingestion notifications

The process of ingesting data into Adobe Experience Platform is comprised of multiple steps. Once you identify data files that need to be ingested into Platform, the ingestion process begins and each step occurs consecutively until the data is either successfully ingested or fails. The ingestion process can be initiated using the Adobe Experience Platform Batch Ingestion API or using the Experience Platform user interface.

Data loaded into Platform must go through multiple steps in order to reach its destination, the Data Lake or the Real-Time Customer Profile data store. Each step involves processing the data, validating the data, and then storing the data before passing it on to the next step. Depending on the amount of data being ingested, this can become a time consuming process and there is always a chance of the process failing due to validation, semantics, or processing errors. In the event of a failure, the data issues need to fixed and then the entire ingestion process must be restarted using the corrected data files.

To assist in monitoring the ingestion process, Experience Platform makes it possible to subscribe to a set of events that are published by each step of the process, notifying you to the status of the ingested data and any possible failures.

Register a webhook for data ingestion notifications

In order to receive data ingestion notifications, you must use Adobe Developer Console to register a webhook to your Experience Platform integration.

Follow the tutorial on subscribing to Adobe I/O Event notifications for detailed steps on how to accomplish this.

Receive data ingestion notifications

Once you have successfully registered your webhook and new data has been ingested, you can start receiving event notifications. These events can be viewed using the webhook itself, or by selecting the Debug Tracing tab in your project’s event registration overview in Adobe Developer Console.

The following JSON is an example of a notification payload that would be sent to your webhook in the case of a failed batch ingestion event:

{
  "event_id": "93a5b11a-b0e6-4b29-ad82-81b1499cb4f2",
  "event": {
    "xdm:ingestionId": "01EGK8H8HF9JGFKNDCABHGA24G",
    "xdm:customerIngestionId": "01EGK8H8HF9JGFKNDCABHGA24G",
    "xdm:imsOrg": "{ORG_ID}",
    "xdm:completed": 1598374341560,
    "xdm:datasetId": "5e55b556c2ae4418a8446037",
    "xdm:eventCode": "ing_load_failure",
    "xdm:sandboxName": "prod",
    "sentTime": "1598374341595",
    "processStartTime": 1598374342614,
    "transformedTime": 1598374342621,
    "header": {
      "_adobeio": {
        "imsOrgId": "{ORG_ID}",
        "providerMetadata": "aep_observability_catalog_events",
        "eventCode": "platform_event"
      }
    }
  }
}

To view the full schema for event notifications, refer to the public GitHub repository.

Next steps

Once you have registered Platform notifications to your project, you can view received events from the Project overview. Refer to the guide on tracing Adobe I/O Events for detailed instructions on how to trace your events.

Appendix

The following section contains additional information about interpreting data ingestion notification payloads.

Available status notification events event-codes

The following table lists the available data ingestion status notifications that you can subscribe to.