Amazon S3

The Amazon S3 Connector lists objects in a bucket, reads their contents, writes and replaces objects, changes their metadata and storage class, deletes them, copies them, and restores archived objects. It works the same way as the Google Cloud Storage Connector: buckets are where files arrive from other systems and where a Program leaves its exports.

SourcesCopy Anchor Link

• s3.objects — objects in the bucket: resource, version (the ETag, or the version_id on a versioned bucket), key, size, content_type, etag, version_id, last_modified, storage_class, metadata (user metadata as a JSON object). The key is key. Filter by prefix to list one "folder". LIST_FILES(s3, glob => r'inbound/*.pdf') is the same listing by path, with resource, version and path.
• s3.contents — the content of objects: resource, version, key, media_type and content, a content_ref. Join it on resource and version from the listing; the Connector reads only the versions the join names. For typed rows from CSV, Parquet or JSON-lines objects, use the Local files Connector's table sources pointed at an s3:// path.

version identifies the bytes that were listed: on a versioned bucket a replaced object is a different version_id, and on an unversioned bucket a changed ETag is read as a conflict rather than as the file that was listed.

TypesCopy Anchor Link

Effects and operationsCopy Anchor Link

Record Effects write by key:

• update — EFFECT UPDATE s3.objects ID key SET (content_type = …, storage_class = …, metadata = …) rewrites metadata or storage class. S3 has no metadata-only update, so the Connector copies the object onto itself with MetadataDirective: REPLACE, conditioned on the ETag seen in the Preview; the content is unchanged.
• delete — EFFECT DELETE s3.objects ID key deletes the object; on a versioned bucket, the version_id it was previewed at.

Content:

• upload — EFFECT STORE s3.objects SET (key = …, content = content [, content_type = …, storage_class = …, overwrite = TRUE]) creates an object from the row's content (a content_ref from SERIALIZE or from a contents source). Without overwrite the write is sent with If-None-Match: *; with it, with If-Match on the ETag seen in the Preview.

Operations:

• copy — EFFECT COMMAND s3.copy TARGET key ID key INPUT (destination = 'other-bucket/path/file.pdf') copies an object within or across buckets the account can write.
• restore — EFFECT COMMAND s3.restore TARGET key ID key INPUT (days = 7, tier = 'Standard' | 'Bulk' | 'Expedited') requests a temporary restore of a Glacier or Deep Archive object.

SetupCopy Anchor Link

"s3": {
  "connector": "s3",
  "bucket": "acme-exports",
  "region": "eu-west-1",
  "prefix": "actql/",
  "credentials": {
    "access_key_id": {"env": "AWS_ACCESS_KEY_ID"},
    "secret_access_key": {"env": "AWS_SECRET_ACCESS_KEY"}
  }
}

An IAM role assumed by the Host can be used instead of keys by omitting credentials. The identity needs s3:ListBucket on the bucket, s3:GetObject (and s3:GetObjectVersion on versioned buckets) to read, s3:PutObject to upload and rewrite, s3:DeleteObject / s3:DeleteObjectVersion to delete, and s3:RestoreObject to restore; a copy also needs read on the source and s3:PutObject on the destination. prefix limits the connection to part of the bucket, and a Program cannot name a key outside it. An endpoint field selects an S3-compatible service. On ActQL Cloud the keys live in the Cloud's secret store, or the Cloud assumes a role you grant it; the connection fields are the same.

Capabilities and pushdownCopy Anchor Link

Preview and reconciliationCopy Anchor Link

Preview. Metadata updates and deletes are full_diff: the Connector reads the object's metadata (HEAD), records its ETag and version_id, and the Preview shows current and new values; the request carries If-Match, so an object changed in between makes the item failed with conflict, never a silent overwrite. Uploads are targeted: the Preview shows the key, media type, byte count and content identity (the SHA-256 the Connector sends as the checksum) that would be written, and whether an existing object would be replaced. Copies preview source and destination; restores preview the object, the tier and the number of days.

Reconciliation. Each item is bound to the key, the expected checksum and, for multipart uploads, the upload id. Every upload, copy and metadata rewrite carries the request's marker in user metadata (x-amz-meta-actql-request); a lost reply is reconciled by HEAD on the destination for that marker, and a multipart upload by its recorded upload id. A matching checksum or matching fields without the marker are not evidence. Deletes and restores leave no marker S3 keeps, so a lost reply to either stays unknown. An in-progress multipart upload is listed by its upload id and either completed from its recorded parts or aborted — never completed blindly. An object with the right key but a different checksum is not evidence; the item stays unknown and nothing is resent. Uploads, copies, updates and deletes end applied on confirmation; a restore ends accepted and stays so while Glacier works, with explain --run and status showing the expected completion.

Limits and costCopy Anchor Link

• Listing is paged 1,000 keys at a time, within the Program's row limit.
• Contents are bounded by the Program's byte limit; a large object is refused rather than partially read. Reads verify the ETag or checksum.
• Objects above 8 MiB are uploaded as multipart uploads; a single object may be up to 5 TiB. Copies of objects above 5 GiB are multipart copies.
• S3 charges per request class (LIST and GET, PUT and COPY), for storage by class, for restores by tier, and for data transfer out of the region; the Connector's requests are ordinary billed operations.

ExampleCopy Anchor Link

Files that arrived in the last hour:

FROM s3.objects
|> WHERE key LIKE 'inbound/%' AND last_modified >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
|> SELECT key, size, last_modified
|> ORDER BY last_modified DESC;

Write a daily export of open deals:

FROM hubspot.deals
|> WHERE stage NOT IN ('closedwon', 'closedlost')
|> SELECT id, dealname, amount, stage
|> SERIALIZE FORMAT CSV ORDER BY id
     OPTIONS (name = CONCAT('open-deals-', FORMAT_DATE('%Y%m%d', CURRENT_DATE()), '.csv'),
              header = TRUE, null_marker = '')
|> EFFECT STORE s3.objects SET (key = 'exports/' || name, content = content);

Copy the PDFs that arrived into an archive bucket (a second S3 connection, archive), unchanged:

FROM LIST_FILES(s3, glob => r'inbound/*.pdf') AS f
|> JOIN s3.contents AS c ON f.resource = c.resource AND f.version = c.version
|> EFFECT STORE archive.objects SET (key = 'pdf/' || f.path, content = c.content);

Bring last year's archived statements back for a week:

FROM s3.objects
|> WHERE key LIKE 'statements/2025/%' AND storage_class = 'DEEP_ARCHIVE'
|> EFFECT COMMAND s3.restore TARGET key ID key INPUT (days = 7, tier = 'Bulk');