Rotating applications' Ceph S3 credentials

At $DAYJOB, some concern has been expressed that S3 client applications have no credential rotation. Which is probably fair. Here are some thoughts on the subject:

• S3 bucket and object access is assigned to an abstraction called a "user."
• Users identify themselves to the storage not by presenting their name, but my means of an "access key" and "secret key." The secret key is not sent across the wire to the S3 server. Instead it is used to sign any given request.
• Users may have more than one access key/secret key pair active at any given time.
• This ability to have multiple key pairs active permits the storage administrators to create a new key pair for an application, store it somewhere it can be retrieved by the application, and some time later, revoke the previous keypair.

It turns out to not be super complicated to do this. Let's walk through the process with a newly provisioned application:

$ sudo radosgw-admin user create --display-name "My application test user" --uid my-app-test-user
{
    "user_id": "my-app-test-user",
    "display_name": "My application test user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "my-app-test-user",
            "access_key": "98PBNESEUJZD49GKGOEM",
            "secret_key": "AtKHe8NTIXWMvKUZJMiIhuqVqVFPoUFsciNvr1Di",
            "active": true,
            "create_date": "2025-07-23T21:50:05.710896Z"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": [],
    "account_id": "",
    "path": "/",
    "create_date": "2025-07-23T21:50:05.709612Z",
    "tags": [],
    "group_ids": []
}

$

This is just the standard way of creating an S3 user with Ceph. Of note here, the access key and secret key are both random strings. See, specifically, the values in the keys section of the JSON output. At this point, the access key and secret key can be stored in a HashiCorp vault secret store for the application to retrieve at start time.

For discussion purposes, let us assume that an applications' keypair needs rotating every 90 days. At day 89, a new key can be added to this user like so:

$ sudo radosgw-admin key create --display-name "My application test user" --uid my-app-test-user --gen-access-key
{
    "user_id": "my-app-test-user",
    "display_name": "My application test user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "my-app-test-user",
            "access_key": "98PBNESEUJZD49GKGOEM",
            "secret_key": "AtKHe8NTIXWMvKUZJMiIhuqVqVFPoUFsciNvr1Di",
            "active": true,
            "create_date": "2025-07-23T21:50:05.710896Z"
        },
        {
            "user": "my-app-test-user",
            "access_key": "APENQS50ZRMJA3S5AFIS",
            "secret_key": "Rnz5aEc3IW6rQq5jSBR65QWtEvVPUgLuZAZ1NHIX",
            "active": true,
            "create_date": "2025-07-23T21:59:22.073250Z"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": [],
    "account_id": "",
    "path": "/",
    "create_date": "2025-07-23T21:50:05.709612Z",
    "tags": [],
    "group_ids": []
}

$ 

At this point, the newly generated keypair also needs to be stashed in Hashicorp Vault for consumption by the application.

Note that a new access key and secret key were added to the user's keys attribute in the JSON output. If a need ever comes up to see all keys associated with a specific user, simply run

$ sudo radosgw-admin user info --uid my-app-test-user | jq .keys
[
  {
    "user": "my-app-test-user",
    "access_key": "98PBNESEUJZD49GKGOEM",
    "secret_key": "AtKHe8NTIXWMvKUZJMiIhuqVqVFPoUFsciNvr1Di",
    "active": true,
    "create_date": "2025-07-23T21:50:05.710896Z"
  },
  {
    "user": "my-app-test-user",
    "access_key": "APENQS50ZRMJA3S5AFIS",
    "secret_key": "Rnz5aEc3IW6rQq5jSBR65QWtEvVPUgLuZAZ1NHIX",
    "active": true,
    "create_date": "2025-07-23T21:59:22.073250Z"
  }
]
$

After some reasonable grace period (perhaps 7 days), the older keypair can be marked inactive:

$ sudo radosgw-admin user modify --uid my-app-test-user --access-key 98PBNESEUJZD49GKGOEM --key-active false
{
    "user_id": "my-app-test-user",
    "display_name": "My application test user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "my-app-test-user",
            "access_key": "98PBNESEUJZD49GKGOEM",
            "secret_key": "AtKHe8NTIXWMvKUZJMiIhuqVqVFPoUFsciNvr1Di",
            "active": false,
            "create_date": "2025-07-23T21:50:05.710896Z"
        },
        {
            "user": "my-app-test-user",
            "access_key": "APENQS50ZRMJA3S5AFIS",
            "secret_key": "Rnz5aEc3IW6rQq5jSBR65QWtEvVPUgLuZAZ1NHIX",
            "active": true,
            "create_date": "2025-07-23T21:59:22.073250Z"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": [],
    "account_id": "",
    "path": "/",
    "create_date": "2025-07-23T21:50:05.709612Z",
    "tags": [],
    "group_ids": []
}

$

Inactive keys can also be deleted (perhaps after another 90 days):

$ sudo radosgw-admin key rm --uid my-app-test-user --access-key 98PBNESEUJZD49GKGOEM
{
    "user_id": "my-app-test-user",
    "display_name": "My application test user",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "my-app-test-user",
            "access_key": "APENQS50ZRMJA3S5AFIS",
            "secret_key": "Rnz5aEc3IW6rQq5jSBR65QWtEvVPUgLuZAZ1NHIX",
            "active": true,
            "create_date": "2025-07-23T21:59:22.073250Z"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": [],
    "account_id": "",
    "path": "/",
    "create_date": "2025-07-23T21:50:05.709612Z",
    "tags": [],
    "group_ids": []
}

$