S3 audit log export¶
Enterprise, from 0.4.0
Archive the TDB audit log to Amazon S3 — or any S3-compatible store (MinIO, Ceph, Cloudflare R2) — as newline-delimited JSON objects.
S3 is the natural destination for a tamper-evident log: pair it with Object Lock or a write-once bucket policy and the archive becomes something neither an operator nor an attacker on the TDB host can rewrite.
How it works¶
Each export writes the entries S3 has not yet received as one object per batch, keyed by the sequence range it contains:
s3://your-bucket/tdb-audit/tdb-audit-000000001-000000500.jsonl
s3://your-bucket/tdb-audit/tdb-audit-000000501-000001000.jsonl
Keying by seq range rather than by wall-clock time means a re-run writes the
same key and overwrites rather than accumulating a second copy, and a missing
range is visible just by listing the bucket.
The export is incremental — TDB records how far S3 has consumed and resumes from there, so a scheduled export does not re-upload history. It is also on-demand: TDB does not stream in real time. Run it from cron or your orchestration tool.
Configuration¶
| Variable | Default | Description |
|---|---|---|
TDB_S3_BUCKET |
— | Target bucket. Unset means the exporter is disabled and the endpoint reports disabled. |
TDB_S3_PREFIX |
tdb-audit |
Key prefix within the bucket |
TDB_S3_REGION |
SDK default | AWS region |
TDB_S3_ENDPOINT_URL |
— | Override for S3-compatible storage |
TDB_S3_SSE |
— | Server-side encryption: AES256 or aws:kms |
TDB_S3_SSE_KMS_KEY_ID |
— | KMS key ID when TDB_S3_SSE is aws:kms |
Server-side encryption¶
TDB_S3_SSE asks the storage service to encrypt the object. TDB does not
encrypt the body itself, so the setting is a request the server can refuse.
If the server refuses the algorithm, the export fails rather than storing the
batch unencrypted. The object is not written, the batch is reported in
errors, and the export cursor does not advance — so those entries are retried
on the next run instead of being silently archived as plaintext:
{"exported": 0, "skipped": 500, "errors": ["S3 put_object failed: ClientError: An error occurred (NotImplemented) when calling the PutObject operation: Server side encryption specified but KMS is not configured (KMS not configured for a server side encrypted objects)"], "disabled": false}
This matters most on S3-compatible storage. MinIO, for example, rejects AES256
unless a KMS is configured for it; AWS S3 accepts both AES256 and aws:kms
directly. If you set TDB_S3_SSE against storage that cannot honour it, every
export fails until either the server is configured for encryption or the variable
is unset — which is the intended outcome, since the alternative is an audit
archive you believe is encrypted and is not.
Credentials¶
TDB takes no access keys of its own. Credentials resolve through the standard AWS chain — environment variables, shared config, or an instance/task role. On EC2, ECS or EKS, attach a role with write-only access to the bucket and TDB never holds a long-lived secret:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::your-bucket/tdb-audit/*"
}]
}
s3:PutObject alone is enough. TDB never reads or deletes what it has written —
so a compromised TDB host cannot erase the archive it has already produced.
Running an export¶
curl -fsS -X POST "http://localhost:8000/v1/audit/export?destination=s3" \
-H "Authorization: Bearer $ADMIN_KEY"
Long exports
Add &wait=false to return 202 immediately and run the export in
the background, then read the result from
GET /v1/audit/export/status. A destination runs one export at a
time — an overlapping call gets 409. See
Exporting to a SIEM.
A second run immediately after sends nothing:
Check progress for every destination:
{"destinations": [
{"name": "splunk", "configured": false, "last_exported_seq": 0, "last_run": null},
{"name": "s3", "configured": true, "last_exported_seq": 1000,
"last_run": {"state": "succeeded", "started_at": "2026-08-12T04:10:02+00:00",
"finished_at": "2026-08-12T04:10:39+00:00",
"exported": 500, "skipped": 0, "errors": []}}
]}
Both endpoints require the admin role.
Scheduling¶
# Archive to S3 every hour
0 * * * * curl -fsS -X POST "http://localhost:8000/v1/audit/export?destination=s3" \
-H "Authorization: Bearer $ADMIN_KEY" >/dev/null
Re-sending history¶
curl -fsS -X POST "http://localhost:8000/v1/audit/export?destination=s3&since_seq=0" \
-H "Authorization: Bearer $ADMIN_KEY"
Because keys are derived from the seq range, this overwrites the existing objects
rather than creating duplicates — unless the bucket has Object Lock enabled, in
which case the overwrite is refused and reported in errors.
Failure behaviour¶
The cursor advances only over entries S3 accepted. If a batch fails, the export stops there, reports the error, and the next run resumes from the last confirmed batch. Nothing is skipped: a gap in an audit archive is worse than a late one.
Local audit logging is entirely independent of export. An S3 outage never affects TDB's ability to serve queries or write its own log.
Testing against MinIO¶
docker run -d --name minio -p 9000:9000 \
-e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
minio/minio server /data
export TDB_S3_BUCKET=tdb-audit-test
export TDB_S3_ENDPOINT_URL=http://127.0.0.1:9000
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_DEFAULT_REGION=us-east-1
Create the bucket, then run the export as above.
Troubleshooting¶
| Symptom | Cause |
|---|---|
"disabled": true |
TDB_S3_BUCKET is not set |
NoCredentialsError |
No credentials in the AWS chain — check the instance role or environment |
AccessDenied |
The role lacks s3:PutObject on the prefix |
NoSuchBucket |
Bucket missing, or wrong region/endpoint |
| Export always sends everything | The cursor could not be recorded — check that the registry database is writable; the response errors will say so |