Create datasets and ingest data

Hi there, data ingestion gives you the ability to bring your data together in one open and scalable Platform. When your data is marked as schemas, it becomes easy to combine data from multiple sources and do things like creating a Real-time Customer Profile. In this video, we will ingest Loyalty data from our Luma brand and map it to the Luma Loyalty members schema that we created in a separate video. Let’s take a look at how we can ingest Loyalty data into Adobe Experience Platform. When you first log into Platform, you land on the homepage from here, you will select Datasets on the left to navigate the Dataset workspace. Once you arrive in Dataset workspace, you will be presented with a list of Datasets already in Platform. From here you can view and manage all Datasets in your organization. To create a new Dataset, you will need to click on the Create Dataset button in the upper right corner of the page. Once you click Create Dataset, you will be given two options, Create Dataset from schema, and Create Dataset from CSV file. Since we have already defined the Luma Loyalty members schema in another video let’s select Create Dataset from schema option. Next, you need to select the Luma Loyalty schema from a list of available schemas, and then click Next. Now you need to give this Dataset a friendly name. We will call it Luma Loyalty Data, click on Finish, and an empty Dataset gets created for our Loyalty Data. Adobe Experience Platform lets users Ingest Data into a Dataset through batch ingestion and streaming ingestion. Batch ingestion lets you import data in a Batch from any number of data sources. Streaming ingestion allows users to send data to a Platform in real-time from client and server-side devices. For batch ingestion you could select the Dataset we created when setting up the source connect of workflow. Data can also be ingested into a Dataset using the data ingestion API. Add Data UI tool lets you perform some initial testing of the data to ensure it looks right before configuring the source or using the API for data ingestion. In the right panel under the Add Data section in performing a Batch data ingestion into a Dataset, partial ingestion enables ingestion of valid records with a specific error threshold for failed records before the entire Batch fails. Enabling partial ingestion also allows you to perform an error diagnosis or error download using the API for failed records. The error threshold will allow you to set the percentage of acceptable errors before the entire Batch fails. By default, this value is set to 5%. Next let’s add data to this Dataset. Let’s drag the file containing Loyalty Data stored in JSON format into the panel. As soon as you drop the file in the interface a Batch gets created with the initial status as loading and then moves to a processing state. You can also see that no batches have been added, message gets replaced with Batch metrics. To find more information about the Batch, you can click on the Batch ID to get more details. The Batch overview page shows the current status, the number of records ingested, file size and a few additional details. In our case, we have zero failed records. When there is a record failure it’s associated error code and description gets displayed within the Batch overview page. Let’s navigate to the Dataset activity page, and you can notice that the Batch status changed from processing to success. Let’s also preview the Dataset to make sure that the data ingestion was successful. If you want to use your Loyalty data in Real-time Customer Profile, you will need to enable it by toggling the button in the right panel. This lets the Real-time Customer Profile service know to start enriching customer profiles with any data in this Dataset. Next, you need to confirm that you want to enable this Dataset for Real-time Customer Profile. Once you click Enable, this Dataset is now ready to enrich profiles stored in Real-time Customer Profile. Let’s upload the Loyalty data again to our Dataset with the Real-time Customer Profile enabled. In the next successful Batch run, data ingested into our Dataset will be used to create real-time customer profiles. Our Dataset contains identity fields that will be populated and then be used to build identities in Platform. Platform Identity Service helps you to gain a better view of your customer and their behavior by bridging identities across devices and systems, allowing you to deliver impactful personal digital experiences in real time. I hope this video provides you an overview of data ingestion in Adobe Experience Platform. -