Home

S3

Create and manage S3 buckets

Overview

If you’re running in AWS, odds are you’re using S3. Cumulus makes configuring your buckets much simpler through its easy-to-use JSON configuration. You can also share CORS and policy configuration between buckets. Read the following sections to learn about configuring your S3 environment with Cumulus. Example configuration can be found in the Cumulus repo.

Bucket Definition

Each S3 bucket is defined in its own file, and the folder distributions are located in is configurable. File names for S3 bucket configuration correspond to the name of that bucket and are also the name that Cumulus will use to refer to your bucket in its output, and as input into its command line interface (minus the .json, of course). This section describes the base Bucket configuration, and the following sections describe how to configure the various features available to S3 Buckets. Buckets are JSON objects with the following attributes:

CORS Templates

CORS controls which origins have access to the content in your bucket. A CORS template has the following properties:

Here is an example CORS configuration:

[
  {
    "headers": [
      "Authorization"
    ],
    "methods": [
      "GET"
    ],
    "origins": [
      "*"
    ],
    "exposed-headers": [],
    "max-age-seconds": 86400
  }
]

Policy Templates

Policies control access to the S3 bucket and its resources. Cumulus mirrors the format used by AWS for configuring policies. A detailed explanation of bucket policies by AWS can be found here.

Here is an example policy configuration which allows public access to objects in whatever bucket you passed in as the value of the bucket variable:

{
  "Version":"201=5-08-28",
  "Statement":[
    {
      "Sid":"PublicGets",
      "Effect":"Allow",
      "Principal": "*",
      "Action":["s3:GetObject"],
    }
    "Resource":["arn:aws:s3:::{{bucket}}/*"]
  ]
}

Grants

Grants control which accounts have permissions to perform actions on your bucket. A grants configuration has the following properties:

Additionally, the following names match the ones in the AWS console, and, when used, do not need to be specified with an email or an id:

Here is an example grant:

[
  {
  "name": "your-account",
    "id": "328ead623599bb1a2fc207d606cec78f54d201ed6e19db5f056a406bac48b77c",
    "permissions": ["list"]
  },
  {
    "name": "loggly",
    "email": "aws@loggly.com",
    "permissions": ["list", "update"]
  },
  {
    "name": "AuthenticatedUsers",
    "permissions": ["view-permissions"]
  },
  ...
]

Static Website Configuration

S3 allows you to serve a static website from your bucket. To do so, add a JSON object called "website" to your bucket with the following attributes:

Here are two website configuration examples (the first configures the bucket as a website, the second just redirects all traffic to the bucket):

{
  "website": {
    "index": "index.html",
    "error": "error-pages/error.html"
  }
}

{
  "website": {
    "redirect": "https://myawesomesite.com"
  }
}

Bucket Logging

S3 allows you to log access information about your bucket. To enable logging, provide a JSON object named "logging" to your bucket configuration with the following attributes:

Here’s an example:

{
  "target-bucket": "logging-bucket",
  "prefix": "logs/"
}

Bucket Notifications

When certain events happen in a bucket, S3 allows you to post an event to an SNS topic, SQS queue, or Lambda function. To do so, add an array named "notifications" to your bucket configuration, and fill it with JSON objects containing the following attributes:

Here’s an example of a notification configuration object:

{
  "name": "example-notification",
  "triggers": [
    "ObjectCreated:*"
  ],
  "prefix": "a",
  "suffix": "z",
  "type": "sns",
  "target": "example-topic"
}

Lifecycle Configuration

S3 lets you configure lifecycle rules for the objects in a bucket. To configure lifecycle rules, add an array named "lifecycle" that contains JSON objects with the following attributes:

Here’s an example lifecycle rule:

{
  "name": "lifecycle-rule",
  "prefix": "logs",
  "days-until-glacier": 10,
  "days-until-delete": 100,
  "past-versions": {
    "days-until-glacier": 5,
    "days-until-delete": 120
  }
}

Bucket Replication

To enable replication of a bucket to another bucket in another region, supply a JSON object named "replication" to your bucket configuration and give it the following attributes:

Here’s an example replication rule:

{
  "iam-role": "a-role",
  "prefixes": [
    "a",
    "z"
  ],
  "destination": "backup-bucket"
}

A Full Example

Here’s an example of a full bucket configuration with all features specified.

{
  "region": "us-east-1",
  "permissions": {
    "cors": {
      "template": "example-cors",
      "vars": {}
    },
    "policy": {
      "template": "example-policy",
      "vars": {
        "bucket": "example-bucket"
      }
    },
    "grants": [
      {
        "name": "your-account",
        "id": "328ead623599bb1a2fc207d606cec78f54d201ed6e19db5f056a406bac48b77c",
        "permissions": ["list"]
      }
    ]
  },
  "website": {
    "index": "index.html",
    "error": "error-pages/error.html"
  },
  "logging": {
    "target-bucket": "logging-bucket",
    "prefix": "logs/"
  },
  "notifications": [
    {
      "name": "example-notification",
      "triggers": [
        "ObjectCreated:*"
      ],
      "prefix": "a",
      "suffix": "z",
      "type": "sns",
      "target": "example-topic"
    }
  ],
  "lifecycle": [
    {
      "name": "lifecycle-rule",
      "prefix": "a",
      "days-until-glacier": 10,
      "days-until-delete": 100,
      "past-versions": {
        "days-until-glacier": 5,
        "days-until-delete": 120
      }
    }
  ],
  "versioning": true,
  "replication": {
    "iam-role": "a-role",
    "prefixes": [
      "a",
      "z"
    ],
    "destination": "backup-bucket"
  },
  "tags": {
    "a key": "a value"
  }
}

Additional Information

There are some configuration options for S3 buckets that Cumulus does not handle because we do not use them at Lucid or do not want them managed by Cumulus at this time. These include:

If you would like any of these limitations changed, please submit a pull request.

Diffing and Syncing S3

Cumulus’s S3 module has the following usage:

cumulus s3 [diff|help|list|migrate|sync] <asset>

S3 buckets can be diffed, listed, and synced (migration is covered in the following section). The three actions do the following:

Migration

If your environment is anything like ours, you have dozens of S3 buckets, and would rather not write Cumulus configuration for them by hand. Luckily, Cumulus provides a migrate task that will pull down your buckets and produce configuration for them. It will also pull down your CORS rules and bucket policies, and, where buckets are using the same CORS rules or policies, reference the same file.

Configuration

Cumulus reads configuration from a configuration file, conf/configuration.json (the location of this file can also be specified by running cumulus with the --config option). The values in configuration.json can be changed to customized to change some of Cumulus’s behavior. The following is a list of the values in configuration.json that pertain to the S3 module:

➜