Monitor accounts and dataflows in the UI
Source connectors in Adobe Experience Platform provide the ability to ingest externally sourced data on a scheduled basis. This tutorial provides steps for viewing existing accounts and dataflows from the Sources workspace.
This tutorial requires a working understanding of the following components of Adobe Experience Platform:
- Experience Data Model (XDM) System : The standardized framework by which Experience Platform organizes customer experience data.
- Real-time Customer Profile : Provides a unified, real-time consumer profile based on aggregated data from multiple sources.
Log in to Adobe Experience Platform and then select Sources from the left navigation bar to access the Sources workspace. The Catalog screen displays a variety of sources for which you can create accounts and dataflows with. Each source shows the number of existing accounts and dataflows associated to them.
Select Accounts from the top header to view existing accounts.
The Accounts pages appears. On this page is a list of viewable accounts, including information about their source, username, number of dataflows, and date of creation.
Select the funnel icon on the top left to launch the sort window.
The sorting panel allows you to access accounts from a specific source. Select the source you wish to work with and select the account from the list on the right.
Use the button in the Name column to create a new source dataflow for the selected account.
Additionally, you can edit existing account information and update your account credentials. Select the pencil icon for the account information you want to edit.
The Edit account details modal appears. From this page, you can update your existing account details and authentication credentials.
Editing account details is available on all batch source connectors.
From the Accounts page, you can view a list of existing dataflows or target datasets associated with the account you accessed. Select the ellipses ( ... ) button to bring up more available options for your selected dataflow. These options are further described below:
Allows you to edit the ingestion schedule of the dataflow.
Allows you to disable data ingestion for the selected dataflow.
Allows you to delete the selected dataflow.
Dataflows can be accessed directly from the Catalog page without viewing Accounts . Select Dataflows from the top header to view a list of dataflows.
A list of existing dataflows appears. On this page is a list of viewable dataflows, including information about their source, username, number of dataflows, and status.
See the following table for more information on statuses:
The Enabled status indicates that a dataflow is active and is ingesting data according to the schedule it was provided.
The Disabled status indicates that a dataflow is inactive and is not ingesting any data.
The Processing status indicates that a dataflow is not yet active. This status is often encountered immediately after a new dataflow is created.
The Error status indicates that the activation process of a dataflow has been disrupted.
Select the funnel icon on the top left to sort.
The sorting panel appears. Select the source you wish to access from the scroll menu and select the dataflow from the list on the right. You can also select the ellipses ( ... ) button to bring up more available options for your selected dataflow.
The Dataflow activity page contains details on the number of records ingested and records failed, as well as information regarding dataflow status and processing time. Select the calendar icon above the dataflow to adjust the time frame of your ingestion records.
The calendar allows you to view the different time frames for ingested records. You can choose to select one of the two pre-set options "Last 7 days" or "Last 30 days". Alternatively, you can set a custom time frame using the calendar. Select your time frame of choice and select Apply to continue.
By default, the Dataflow activity displays the Properties panel associated with the dataflow. Select the flow run from the list to see its associated meta data, including information about its unique run ID.
Select Dataflow run start to access the Dataflow run overview .
The Dataflow run overview displays information on the dataflow including its metadata, partial ingestion status, and assigned error threshold. The upper header also includes an error summary. The Error summary contains the specific top-level error that shows at which step the ingestion process encountered an error.
Refer to the following table for errors that can be seen in the Error summary .
An error occurred while data is being copied from a source.
An error occurred while copied data is being processed to Platform. This error could be regarding parsing, validating, or transforming.
The lower half of the screen contains information on Dataflow run errors . From here, you can also view the files ingested, preview and download error diagnostics, or download the file manifest.
The Dataflow run errors section displays the error code, the number of records failed, and information describing the error.
Select Preview error diagnostics to see more information on the ingestion error.
The Error diagnostics preview panel appears. This screen displays specific information regarding the ingestion failure, including the file name, error code, the name of the column in which the error occurred, and a description of the error.
This section also includes a preview of the column that contains the error.
To enable Error diagnostics preview you must activate Partial ingestion and Error diagnostics when configuring a dataflow. Doing so will allow the system to scan all of the records ingested during the flow run.
After previewing the errors, you can select Download from within the dataflow runs overview panel to access full error diagnostics and download the file manifest. See the documents on error diagnostics and downloading metadata for more information.
For more information on monitoring dataflows and ingestion, refer to the tutorial on monitoring streaming dataflows .
By following this tutorial, you have successfully accessed existing accounts and dataflows from the Sources workspace. Incoming data can now be used by downstream Platform services such as Real-time Customer Profile and Data Science Workspace. See the following documents for more details: