Targets Overview
  • 25 Jan 2021
  • 2 Minutes To Read
  • Print
  • Share
  • Dark
    Light

Targets Overview

  • Print
  • Share
  • Dark
    Light

Overview


Rivery targets include data warehouses and data storage. You can configure as many targets as you would like.

See below for specific guidance on connecting to each type of target:


Amazon Redshift

Amazon S3

Azure Blob Storage

Azure Data Lake Analytics

Azure SQL Data Warehouse

Google BigQuery

Google Cloud Storage

Snowflake

Treasure Data




Flows and Concepts when loading into DWH Targets


Some Key Concepts:

ELT

Rivery loading and transforming methods are running based on the concepts of ELT (Extract, Load, Transform). 
Thus, in order to reduce the bottleneck in transforming the data during the load, and ensure using the target DBs high performances in order to transform the data in the DB itself.

Running Over Data Images

In order to ensure the data represents the last complete truth, Rivery loads and transforms the data over the pipeline in think of the target tables. 
That means, Rivery never drops/changes/alter/modify the target table, or its data, in case of failures. 
Rivery platform ensure the data in the table, and the metadata of the table, will be complete as it defined in the River (pipeline), so there will be no data loss, incomplete data, or changes in the table structure in cases of the river did not complete its mission.

File Zone

In order to prevent data loss, and reduce the dependence in the source when loading data into the target is failed for some reason, Rivery stores the data in a cloud storage service such as AWS S3, Google Cloud Storage, or Azure Blob Storage - termed as File Zone
Therefore any failures in loading data into the target databases won’t cause a data loss and in case of failures.
Rivery platform knows to make the retry in the loading step only, without pulling again the same data from the source.


DWH River Types

Source To Target  Rivers

Source to Target rivers are pipelines that pull data from sources, and load it, by the key concepts described above, into the target databases, via the File Zone. Source to Target river is responsible for the Extract and Load phases in the ELT process. Each pipeline configuration can load the data using Overwrite, Append Only or Upsert Merge mode.

Logic River

A smart engine that is responsible for the Transform phase in the ELT process. This river type gives the ability to the user running SQL queries steps over the DB he chooses, and therefore manage and select queries results into tables in the DWH or into file(s) in the File Zone. 
The steps in the logic river can run in parallel, by a loop over a list, by condition or step by step, and also available using smart variables.

Loading Flowcharts

Source To Target River: Loading Data Into Target Table Flowchart



Logic River: Selecting Into Target Table Flowchart



Was This Article Helpful?