Setting up Amazon S3 as a Target
  • Updated on 25 Mar 2020
  • 2 minutes to read
  • Contributors
  • Print
  • Share
  • Dark
    Light

Setting up Amazon S3 as a Target

  • Print
  • Share
  • Dark
    Light

Overview

Getting started with Rivery and S3:

This guide will show you how to create an S3 bucket and how to get the right credentials for using S3 with Rivery.

Before you use this guide, please make sure you’ve signed up for AWS and that you have a console admin user.

If you don’t have one of these prerequisites, you can start here .

Create an S3 Bucket

Rivery needs an S3 bucket to serve as a FileZone. You can either use the FileZone bucket or objects as a base to other Hadoop or Spark operations by Amazon EMR, or by your other services.

Please follow the instructions in Creating an S3 bucket in AWS to assist in creating the bucket.

Create an AWS User for Rivery

Rivery uses an Amazon S3 bucket as a target for your source data. You therefore must create a user in the AWS Console that has access to the relevant bucket.

Please follow the instructions in Create an AWS IAM user to assist in creating this user.

Configure your S3 in Rivery

How to setup S3 for use in Rivery:

  1. Log into Rivery.

  2. Next, set your bucket as the default S3 bucket in Rivery:

    • In the main menu, go to Variables

    • Set your {aws_file_zone} variable value to the s3 bucket name that you’ve created. This will be saved automatically by clicking anywhere on the screen.

    • If you don’t have the { aws_file_zone } variable, you should add a new variable with that name and the s3 bucket name that you’ve created as the value. Press on + Add Variable .

      image34.png

  3. Let’s create a new connection for your S3:

    • Go to Connections.

    • Click on New Connection .

    • From the source list, choose S3.

      Screen_Shot_2018-01-01_at_12.08.34_PM.png

    • Pick the region from the drop down list. Default: us-east-1 (N.Virginia).

    • Next, choose the desired authentication option. There are 3 options to chose from:

      1. AWS Keys:

      aws_keys1.PNG

      Enter the credentials information that you created earlier in this guide

      • AWS_access_key_id
      • AWS_secret_access_key
    1. IAM Role -Automatic:

      Follow the detailed instructions in the connection window to create the Role ARN Key

    2. IAM Role - Manual:

      Follow the detailed instructions in the connection window to create the Role ARN Key

    • You can test your connection by clicking the Test Connection button.
    • Give your connection a name and click Save .
  4. You can now use this connection in any river that has S3 as the data target.

Summary

This guide showed you how to create:

  • An IAM user for Rivery
  • An S3 Bucket
  • A connection to the S3 bucket in Rivery

You can now send data to your S3 bucket in any Source to Target river in Rivery.

Was this article helpful?