Loki S3 storage on Ceph RGW: radosgw-admin setup, path-style Loki config, TSDB schema and compactor retention for OpenStack clouds.
Loki writes two kinds of objects into a bucket, compressed log chunks and TSDB index files, and only the Loki Compactor knows which chunks the index still references. That fact drives most decisions in a Loki S3 storage design on Ceph RGW: how retention deletes data, why a bucket lifecycle rule can break queries, and which pool and placement settings must be fixed before the first chunk lands, since some of them cannot be changed later.
The design below assumes an OpenStack cloud whose Ceph cluster already runs the RADOS Gateway (RGW), and its Loki config follows the Loki 3.7 configuration reference. The wider monitoring stack, from Prometheus metrics to alert routing, is covered in the guide to OpenStack monitoring tools, metrics and alert fatigue.
Versions this was checked against
- The example: Loki 3.7.6 and Ceph Tentacle 20.2.4. Every key, default and command below comes from their tagged docs or source.
- On Loki 3.0:
ingester.wal.disk_full_threshold, the loki_ingester_wal_disk_usage_percent metric and compactor.deletion_marker_object_store_prefix do not exist yet. - On Ceph Reef 18.2: dynamic resharding cannot reduce a bucket's shard count; that arrived in Tentacle.
- Gateways managed by an operator such as Rook keep zone and zonegroup settings in their own resources. Check how your operator reconciles placement before adding targets with
radosgw-admin.
When does Loki need object storage at all?
Loki can keep chunks on a local filesystem, and on one node that works. Grafana's filesystem store page lists the limits: clustering needs a shared filesystem such as NFS, Grafana Labs does not support it in production, and one user hit errors at about 5.5 million chunk files. The scalable deployment modes, and a highly available monolithic Loki, need an object store.
Grafana's storage overview lists Amazon S3, GCS, Azure Blob Storage, IBM COS, Baidu and Alibaba OSS as "supported and recommended" chunk stores, and puts "S3 API compatible storage, such as MinIO" in a tier "not typically recommended for production use". Ceph is not named, but as an S3-compatible store RGW falls in that second tier, so validate the design on your own versions.
Which Loki components keep state outside the bucket?
The bucket holds most of Loki's data, and two components still keep state on local disk. Both need a persistent volume; on OpenStack, a Cinder volume is one option.
Ingesters. Ingesters hold chunks in memory and flush one when it nears chunk_target_size (default 1,572,864 bytes compressed), idles for chunk_idle_period (30m), or reaches max_chunk_age (2h). Until then the data lives in memory and in the write-ahead log, enabled by default and replayed on restart. Stateless distributors send each stream to replication_factor ingesters, generally 3, and a write needs two to succeed (components). In Loki 3.7, at disk_full_threshold (default 0.9) WAL disk usage the ingester rejects pushes until usage drops; Loki 3.0 has no threshold and, on a full disk, accepts writes without logging them to the WAL. Alert on loki_ingester_wal_disk_full_failures_total, plus loki_ingester_wal_disk_usage_percent on 3.7.
The Compactor. It compacts indexes and applies retention, and Grafana's retention page says to run it as a singleton StatefulSet with persistent storage for the marker files in its working_directory. Loki 3.7 can also keep markers in the bucket under deletion_marker_object_store_prefix; keep lifecycle rules away from that prefix.
How should Ceph RGW be laid out for a Loki bucket?
RGW maps each bucket to RADOS pools through a placement target: an index_pool for the bucket index, a data_extra_pool for incomplete multipart upload metadata, and a data_pool per storage class (placement).
The constraint with the longest reach: "A bucket's placement target is selected on creation, and cannot be modified." Create the placement target first and the bucket second; changing it later means copying every object into a new bucket.
Erasure coding suits the data pool. Ceph's erasure code page puts the overhead of a k=4, m=2 profile at 1.5x raw capacity, against 3.0x for size-3 replication, and suggests 4+2 when you are unsure. It warns of a performance cost on HDDs and during recovery, discourages m=1 for production, and notes that most EC pools need k+m failure domains: six hosts for 4+2. Loki chunks and TSDB index files are both ordinary S3 objects, so both land on the data pool.
The bucket index pool stays replicated. RGW stores the bucket index in omap (layout), and erasure-coded pools do not support omap. RGW creates any missing zone pool with default PG settings (pools), so create the EC data pool yourself first:
# Erasure-coded data pool for log objects (needs at least 6 hosts with this profile)
ceph osd erasure-code-profile set loki-ec k=4 m=2 crush-failure-domain=host
ceph osd pool create default.rgw.telemetry.data erasure loki-ec
ceph osd pool application enable default.rgw.telemetry.data rgw
# Placement target that maps to it; the index and non-ec pools are replicated
radosgw-admin zonegroup placement add --rgw-zonegroup default --placement-id telemetry
radosgw-admin zone placement add --rgw-zone default --placement-id telemetry \
--data-pool default.rgw.telemetry.data \
--index-pool default.rgw.telemetry.index \
--data-extra-pool default.rgw.telemetry.non-ec
# Restart the gateways, or run `radosgw-admin period update --commit` if a realm is configured
Path A: a native RGW user
Loki uses the keys user create prints, and that user owns the buckets, so placement and quota go on it:
radosgw-admin user create --uid=loki --display-name="Loki log storage"
radosgw-admin user modify --uid=loki --placement-id telemetry
radosgw-admin quota set --quota-scope=user --uid=loki --max-size=<size>
radosgw-admin quota enable --quota-scope=user --uid=loki
Path B: Keystone EC2 credentials
With rgw s3 auth use keystone = true, Loki uses EC2 keys from openstack ec2 credentials create, run as a dedicated Keystone user in its own project (S3 authentication). With rgw keystone implicit tenants = true, RGW creates a user for that project on first access, <project_id>$<project_id> (multitenancy), and it owns the buckets, so settings on a separate --uid=loki never apply. Make one S3 request with the keys, then target that user:
radosgw-admin user modify --uid='<project_id>$<project_id>' --placement-id telemetry
radosgw-admin quota set --quota-scope=user --uid='<project_id>$<project_id>' --max-size=<size>
radosgw-admin quota enable --quota-scope=user --uid='<project_id>$<project_id>'
Create the buckets and check the owner
radosgw-admin has no bucket create subcommand, so use the S3 API. A LocationConstraint of <api_name>:<placement-id> picks the placement target for either path:
EP="--endpoint-url https://rgw.example.internal --region default"
aws configure set default.s3.addressing_style path
aws $EP s3api create-bucket --bucket loki-chunks \
--create-bucket-configuration LocationConstraint=default:telemetry
aws $EP s3api create-bucket --bucket loki-ruler
radosgw-admin bucket stats --bucket=loki-chunks # Path A
radosgw-admin bucket stats --bucket='<project_id>/loki-chunks' # Path B
In the bucket stats output, owner should read loki or <project_id>$<project_id>, and placement_rule should read telemetry.
What does the Loki S3 config look like for Ceph RGW?
Merge these Loki 3.7 storage blocks into your existing server and ring configuration.
auth_enabled: false # single tenant: chunks are written under fake/
schema_config:
configs:
- from: "2026-08-01" # new install: a date in the past
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
path_prefix: index/ # the default, stated so lifecycle rules can exclude it
period: 24h # retention requires a 24h index period
storage_config:
tsdb_shipper:
active_index_directory: /loki/tsdb-index
cache_location: /loki/tsdb-cache
aws:
endpoint: rgw.example.internal
bucketnames: loki-chunks
region: default # the zonegroup api_name
access_key_id: ${LOKI_S3_ACCESS_KEY}
secret_access_key: ${LOKI_S3_SECRET_KEY}
s3forcepathstyle: true
insecure: false # HTTPS
http_config:
ca_file: /etc/loki/tls/rgw-ca.pem
ruler:
storage:
type: s3
s3:
endpoint: rgw.example.internal
bucketnames: loki-ruler
region: default
access_key_id: ${LOKI_S3_ACCESS_KEY}
secret_access_key: ${LOKI_S3_SECRET_KEY}
s3forcepathstyle: true
insecure: false
http_config:
ca_file: /etc/loki/tls/rgw-ca.pem
ingester:
wal:
enabled: true
dir: /loki/wal # persistent volume
compactor:
working_directory: /loki/compactor # persistent volume
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
delete_request_store: s3
limits_config:
retention_period: 744h # 31 days
max_query_lookback: 744h
Notes on the keys that cause trouble:
region. RGW 20.2.4 builds the SigV4 signing key from the region the client sends (rgw_auth_s3.cc), and Loki sends dummy when region is empty (s3_storage_client.go). Bucket creation checks the LocationConstraint against the zonegroup's api_name (rgw_op.cc), so use that api_name (from radosgw-admin zonegroup get) everywhere. On Path B, Keystone checks the signature; test it there.s3forcepathstyle: true. Virtual-hosted addressing needs rgw_dns_name or zonegroup hostnames plus a wildcard certificate (S3 common); path style needs neither.- The schema
from date. In the past for a new install; for an install with data, add a future-dated entry and leave active ones alone. A schema change cannot be rolled back (schema). - The ruler.
type: s3 only selects the backend; ruler.storage.s3 carries the bucket, endpoint, credentials and TLS. The 3.7.6 reference marks ruler.storage deprecated for ruler_storage, which applies only with use_thanos_objstore: true. ${...} placeholders expand only with -config.expand-env=true.- Permissions:
s3:ListBucket, s3:PutObject, s3:GetObject, and s3:DeleteObject for retention. - Delete requests default to the
index/ prefix (delete_request_store_key_prefix).
How does Loki retention work on S3 storage?
Retention is the Compactor's job, and it is off by default: with retention_enabled unset, logs are kept forever (retention). Four rules catch people:
- Retention needs a 24h index period. The page gives 24h as the minimum retention, but 3.7.6's
Limits.Validate() (limits.go) enforces it only on retention_stream periods, so a shorter global retention_period passes validation. Keep both at 24h or more. delete_request_store is required once retention is enabled.- The page says retention changes do not apply to logs already ingested, while the 3.7.6 Compactor (
expiration.go) checks each chunk's end time against the period in force at every pass. Test a shortened period on staging data before relying on either reading. Deleted chunks do not return when you lengthen it. max_query_lookback should not exceed the retention period, or queries reach for deleted data.
On each pass the Compactor writes one index file per tenant per day without the expired chunk references and records those chunks in marker files. A sweeper deletes them after retention_delete_delay, which lets index gateways fetch the new index and leaves a window to cancel a bad change.
RGW supports S3 lifecycle rules (S3 API), which delete by age and prefix with no view of the Loki index. Grafana says Compactor retention is enough for most deployments. A lifecycle backstop should target only the tenant chunk prefix (fake/ here) and expire later than retention plus delete delay (35 days covers 31 days plus 2h). A whole-bucket rule eventually deletes TSDB index files, delete requests or loki_cluster_seed.json, and queries fail.
What does a Loki bucket do to RGW over time?
With the defaults above, an active stream flushes at least every 2h: 12 chunk objects a day. At 5,000 streams that is 60,000 objects a day and 1.86 million across 31 days, before replicas add more. Redo the arithmetic with your own stream count.
Dynamic bucket index resharding raises the shard count automatically, up to 1999 by default; writes can block briefly during a reshard while reads continue. Add radosgw-admin bucket limit check and the non-ec pool to capacity review. Once Loki reaches its RGW quota, uploads are rejected and flushes fail, so alert well below the limit.
When does another log store fit better?
Retention logic stays in Loki, and a bucket in the same Ceph cluster shares its failure domain; a second site takes RGW multisite or a second store.
- You search log bodies more than labels. Loki indexes only labels, so a 30-day free-text search scans chunks. OpenSearch or Elasticsearch keeps a full-text index for that query shape, at the cost of a larger index to run.
- Nobody on the team operates Ceph. A managed logging service, or Amazon S3 from Grafana's recommended tier, moves storage on-call to someone else.
- One node and one disk hold your retention. The filesystem store on a single persistent volume is simpler, within the limits Grafana lists.
Before it carries production logs
- Record the Loki and Ceph versions, and re-test on upgrade of either.
- Kill an ingester and confirm WAL replay; restart the Compactor mid-deletion and confirm markers survive.
- With a short staging retention, record chunk keys older than it, confirm those keys are gone after retention plus
retention_delete_delay, and confirm a LogQL query for a line inside retention still returns it:
# Before: the 20 oldest chunk objects and their upload times
aws $EP s3api list-objects-v2 --bucket loki-chunks --prefix fake/ \
--query 'Contents[].[LastModified,Key]' --output text | sort | head -20 > oldest.txt
# After retention_period + retention_delete_delay: print any key that survived
cut -f2 oldest.txt | while read -r key; do
aws $EP s3api head-object --bucket loki-chunks --key "$key" >/dev/null 2>&1 && echo "kept: $key"
done
- Stop one gateway and lose one OSD host while ingesting.
- Time queries over the last hour and over 30 days, and keep the numbers as a baseline.
Where VEXXHOST fits
Atmosphere, VEXXHOST's OpenStack distribution, ships its own platform logging: Vector collects host and container logs and sends them to Loki, a stack first described in February 2024. In Atmosphere 7.8.0 that Loki is grafana/loki:3.0.0 (roles/defaults/vars/main.yml) running in SingleBinary mode with one replica, on the filesystem store with a 256Gi persistent volume and 30-day compactor retention (roles/loki/vars/main.yml). That is the single-volume side of the decision above, sized for the platform's own logs. How Ceph comes into an Atmosphere deployment is covered in the storage integration guide.
If logs should live outside the cloud that produces them, VEXXHOST's object storage service is Ceph-backed with S3 and Swift APIs, the service the Kubernetes backup storage post describes as a Velero or restic target.
Next steps: run the radosgw-admin and bucket-owner checks above against a staging zone, then the retention test, before any production logs land. If VEXXHOST runs your Ceph or OpenStack, the managed Ceph page and the OpenStack private cloud page describe what the platform team operates for you.
