Rivery loading and transforming methods are running based on the concepts of ELT (Extract, Load, Transform).
Thus, in order to reduce the bottleneck in transforming the data during the load, and ensure using the target DBs high performances in order to transform the data in the DB itself.
Running Over Data Images
In order to ensure the data represents the last complete truth, Rivery loads and transforms the data over the pipeline in think of the target tables.
That means, Rivery never drops/changes/alter/modify the target table, or its data, in case of failures.
Rivery platform ensure the data in the table, and the metadata of the table, will be complete as it defined in the River (pipeline), so there will be no data loss, incomplete data, or changes in the table structure in cases of the river did not complete its mission.
In order to prevent data loss, and reduce the dependence in the source when loading data into the target is failed for some reason, Rivery stores the data in a cloud storage service such as AWS S3, Google Cloud Storage, or Azure Blob Storage - termed as File Zone.
Therefore any failures in loading data into the target databases won’t cause a data loss and in case of failures.
Rivery platform knows to make the retry in the loading step only, without pulling again the same data from the source.
DWH River Types
Source To Target Rivers
Source to Target rivers are pipelines that pull data from sources, and load it, by the key concepts described above, into the target databases, via the File Zone. Source to Target river is responsible for the Extract and Load phases in the ELT process. Each pipeline configuration can load the data using Overwrite, Append Only or Upsert Merge mode.
A smart engine that is responsible for the Transform phase in the ELT process. This river type gives the ability to the user running SQL queries steps over the DB he chooses, and therefore manage and select queries results into tables in the DWH or into file(s) in the File Zone.
The steps in the logic river can run in parallel, by a loop over a list, by condition or step by step, and also available using smart variables.
Source To Target River: Loading Data Into Target Table Flowchart
Logic River: Selecting Into Target Table Flowchart
Was This Article Helpful?
Table Of Contents
We use our own and third-party cookies to understand how you interact with our knowledge base. This helps us show you more relevant content and ads based on your browsing and navigation history.