The Automated, No-Code Data Stack
Learn how Astera Data Stack can simplify and streamline your enterprise’s data management.
Streamline Real-Time Data Integration by Leveraging Workflows
A study by IDC predicts that the worldwide data volume will grow to an enormous 175 zettabytes (ZB) by 2025. Managing growing volumes of data from diverse sources can be a tricky feat. For this reason, many organizations leverage data integration tools with workflow automation capabilities to accelerate their data processes and streamline integration.
Whether you want to consolidate transactional data, migrate data from legacy systems, or integrate partner or vendor data, the workflow component in Astera Centerprise automates the execution of a sequence of tasks, in serial or parallel, on multiple servers. This helps minimize error probability, optimize business processes, and improve time-to-value by eliminating the manual steps involved in designing and deploying data integration flows.
Integrate Data with Workflow Automation in Centerprise
Designed to offer ease-of-use and flexibility, the workflow component helps visualize and automate the entire process, from the point data enters an organization to when it is cleaned, validated, and loaded into the preferred destination.
To illustrate the data integration capabilities of Astera Centerprise with workflow automation, let us consider a scenario in which a business receives customer data in spreadsheet every month. The requirement is to cleanse incoming data and load the processed data into the company’s CRM, Salesforce.com, for a unified view.
The screenshot below shows how the workflow functionality in Astera Centerprise helped accomplish this task by automating the ETL process, and sending email notifications to information users on successful completion of the job.
Figure 1: Using a Workflow Component
Let’s look at the steps involved in the workflow in detail.
Step 1: Looping Through the Source Directory
First, a File System object is used for looping through the source directory to pick up the source file path.
Figure 2: Specifying the Source Directory
Step 2: Performing ETL
The Run Dataflow task is used to call a dataflow within a workflow. In this scenario, we will call an existing dataflow for executing the ETL process.
Figure 3: Run Dataflow Object Properties
Figure 4: Orchestrating the ETL Process in a Workflow
The dataflow is used to extract data from source files, cleanse the raw data to create a standardized structure, process it according to the business requirements, and load the transformed data into the company’s CRM, Salesforce.com, as shown in figure 5.
Moreover, the input and output variables are defined in the Variables object. The former is used to provide the source file path to the Excel source object and the latter is used for passing the job status value to the containing workflow for decision-making, as shown in the image below.
Figure 5: ETL Process to streamline workflows
Step 3: Sending Email Notification
A Decision task invokes one of the two paths in the workflow, depending on whether the logical expression inside the Decision object returns a Yes (True) or a No (False). In this scenario, the value of output variable ‘Job Status’ is being passed on from the dataflow into the workflow for decision-making. A Decision object is used to send an email to notify the users when the job is completed or copy the file to a directory if the job is terminated.
Figure 6: Decision Properties
Figure 7: Using a Decision Object in the Workflow
Step 4: Automating the ETL Flow
Astera Centerprise has a built-in job scheduler that allows you to automate ETL flows and streamline database integration by specifying job frequency. This eliminates the need to manually run the flow every time a file is received.
In this case, the flow runs every time a file is dropped in the source directory, as shown below.
Figure 8: Scheduling a Job
Streamline Integration with Centerprise
Data integration jobs involve complex workflows that extract, cleanse, and validate structured and unstructured data. Automation plays a crucial role in data event streaming, as it helps increase throughput and productivity.
Using Astera’s workflow component, you can visually piece together workflows of any complexity, and scale and automate the entire real-time data integration process – from the extraction of source data to transformation and loading in the data warehouse. In case of real-time streaming, this data can then be displayed through reports simultaneously.
Learn more about data integration and workflow automation capabilities of Astera Centerprise. Download a free 14-day trial version and experience it first-hand!
Authors:
Iqbal Ahmed
