Amazon S3 Watch Publisher

The Amazon S3 (Simple Storage Service) Watch publisher polls activity on an Amazon S3 bucket, and publishes messages about that activity to topics. Activity can include adding, changing, and deleting files.

Note: When setting up an Amazon S3 Watch publisher, you can specify how often to poll an S3 bucket. However, keep in mind that Amazon S3 is a pay-per-use service. Every poll is one request to this service.

Configuring the Publisher

Tip: To test that your parameters work as expected, click Validate.

  1. In the FME Server Web User Interface, select Notifications > Publications, and click New.
  2. Provide a Name for the Publication
  3. Protocol: Select Amazon S3 Watch.

  4. Specify the Topics you want to publish to using the drop-down selector. To create a topic and use it right away, click +.

  5. Bucket to Watch: The bucket name in the Amazon S3 account, case sensitive.

    6. Note: For more information about locating buckets in the Amazon S3 Console, see Support for Viewing Data in the Console User Guide.

  6. AWS Access Key ID and AWS Secret Access Key: Provide your security credentials. For more information, see AWS Security Credentials.
  7. Region: The region of the specified bucket. This field is optional; however, in some cases the Amazon S3 Watch publisher may be unable to validate the bucket without it.
  8. Path to Watch (Optional): Provide a folder location in the specified bucket to watch, using forward slashes.

    9. WARNING: Make sure the specified path is valid. The S3 Watch Publisher does not currently perform any verification that the path exists. If it does not, polling activity may produce false positive results.

    For example, any of the following is acceptable:

    • myfolders/testfolder
    • /myfolders/testfolder
    • //myfolders/testfolder
    • myfolders/testfolder/
    • /myfolders/testfolder/

    If left blank, the publisher watches everything in the bucket. If the path does not exist, the publisher will not start.

    Note: For more information about locating folders in the Amazon S3 Console, see Support for Folders in the Console User Guide.

  9. Watch Subdirectories: To watch activity in the selected directory and all subdirectories, specify Yes. To watch activity only in the selected directory and not in its subdirectories, specify No.

  10. Poll Interval: Specify how often to poll from the S3 bucket, in minutes.

    11. Note: Amazon S3 is a pay-per-use service. Every poll is one request to this service.

  11. Events to Watch for: By default, all actions on the specified directory are monitored, including adding (CREATE), changing (MODIFY), and deleting (DELETE) files (and folders, if applicable). To monitor only one or two of these actions, click the "x" beside the action you do not want to monitor to remove it. To add an action after removing it, click inside the dropbox and select it.

    12. Note: Events trigger based on changes to the entity Tag (ETag), which is a representation of file contents. No message is triggered if a change to an existing file does not update the ETag. If a file is overwritten and the ETag value changes, a MODIFY event is triggered.

  12. Click OK.

Sample Notification Schemas

When a file is deleted

{

"s3watch_publisher_content": "ENTRY_DELETE test/sample_file.txt",

"s3watch_publisher_bucket": "test-bucket",

"fns_type": "s3watch_publisher"

}

When a file is changed (for example, added content)

{

"s3watch_publisher_content": "ENTRY_MODIFY test/sample_file.txt",

"s3watch_publisher_bucket": "test-bucket",

"fns_type": "s3watch_publisher"

}

When a file is added

{

"s3watch_publisher_content": "ENTRY_CREATE test/sample_file.txt",

"s3watch_publisher_bucket": "test-bucket",

"fns_type": "s3watch_publisher"

}