Rotating applications' Ceph S3 credentials

At $DAYJOB, some concern has been expressed that S3 client applications have no credential rotation. Which is probably fair. Here are some thoughts on the subject:

• S3 bucket and object access is assigned to an abstraction called a "user."
• Users identify themselves to the storage not by presenting their name, but my means of an "access key" and "secret key." The secret key is not sent across the wire to the S3 server. Instead it is used to sign any given request.
• Users may have more than one access key/secret key pair active at any given time.
• This ability to have multiple key pairs active permits us storage administrators to create a new key pair for an application, store it somewhere it can be retrieved by the application, and some time later, revoke the previous keypair.

It turns out to not be super complicated to do this. Let's walk through the process with newly provisioned application:

$ sudo radosgw-admin user create --display-name "My application test user" --uid my-app-test-user
{
    "user_id": "my-app-test-user",
    "display_name": "My application test user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "my-app-test-user",
            "access_key": "98PBNESEUJZD49GKGOEM",
            "secret_key": "AtKHe8NTIXWMvKUZJMiIhuqVqVFPoUFsciNvr1Di",
            "active": true,
            "create_date": "2025-07-23T21:50:05.710896Z"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": [],
    "account_id": "",
    "path": "/",
    "create_date": "2025-07-23T21:50:05.709612Z",
    "tags": [],
    "group_ids": []
}

$

This is just the standard way of creating an S3 user with Ceph. Of note here, the access key and secret key are both random strings. See, specifically, the values in the keys section of the JSON output. At this point, the access key and secret key can be stored in a HashiCorp vault secret store for the application to retrieve at start time.

For discussion purposes, let us assume that an applications' keypair needs rotating every 90 days. At day 89, a new key can be added to this user like so:

$ sudo radosgw-admin key create --display-name "My application test user" --uid my-app-test-user --gen-access-key
{
    "user_id": "my-app-test-user",
    "display_name": "My application test user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "my-app-test-user",
            "access_key": "98PBNESEUJZD49GKGOEM",
            "secret_key": "AtKHe8NTIXWMvKUZJMiIhuqVqVFPoUFsciNvr1Di",
            "active": true,
            "create_date": "2025-07-23T21:50:05.710896Z"
        },
        {
            "user": "my-app-test-user",
            "access_key": "APENQS50ZRMJA3S5AFIS",
            "secret_key": "Rnz5aEc3IW6rQq5jSBR65QWtEvVPUgLuZAZ1NHIX",
            "active": true,
            "create_date": "2025-07-23T21:59:22.073250Z"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": [],
    "account_id": "",
    "path": "/",
    "create_date": "2025-07-23T21:50:05.709612Z",
    "tags": [],
    "group_ids": []
}

$ 

Note that a new access key and secret key were added to the user's key in the JSON output.