Compactor Failing to remove Chunks from the GCS while index is getting cleaned up properly

**Describe the bug**
Running self hosted loki version 3.4.2 on GKE and backed by GCS. I was using GCS LCM for the retention. However, I have enabled stream based retention and was able to get it done using compactor.

Now what I see is, compactor able to delete older index only, the chunks are not getting deleted. Because of this, GCS has grown to huge amounts of data.

**To Reproduce**
Steps to reproduce the behavior:
1. Install Loki 3.4.2 with GCS as backend store with tsdb.
2. Enable retention using the compactor alone for retention.

**Expected behavior**
Its should delete the older indexes from index director in GCS and the older chunks from the fake directory in GCS

**Environment:**
- Infrastructure: Kubernetes
- Deployment tool: helm

**Screenshots, Promtail config, or terminal output**
auth_enabled: false
bloom_build:
builder:
planner_address: glance-pd-logging-sg-loki-bloom-planner-headless.loki.svc.cluster.local:9095
enabled: true
bloom_gateway:
client:
addresses: dnssrvnoa+_grpc._tcp.glance-pd-logging-sg-loki-bloom-gateway-headless.loki.svc.cluster.local
enabled: true
chunk_store_config:
chunk_cache_config:
background:
writeback_buffer: 500000
writeback_goroutines: 1
writeback_size_limit: 500MB
default_validity: 0s
memcached:
batch_size: 8
parallelism: 10
memcached_client:
addresses: dnssrvnoa+_memcached-client._tcp.glance-pd-logging-sg-loki-chunks-cache.loki.svc
consistent_hash: true
max_idle_conns: 72
timeout: 2000ms
common:
compactor_address: 'http://glance-pd-logging-sg-loki-compactor:3100'
path_prefix: /var/loki
replication_factor: 3
storage:
gcs:
bucket_name: glance-logging-gcs-prod
chunk_buffer_size: 0
enable_http2: true
request_timeout: 30m
compactor:
delete_request_store: gcs
retention_enabled: true
frontend:
compress_responses: true
max_body_size: 504857600
max_outstanding_per_tenant: 20480
query_stats_enabled: true
scheduler_address: glance-pd-logging-sg-loki-query-scheduler.loki.svc.cluster.local:9095
tail_proxy_url: http://glance-pd-logging-sg-loki-querier.loki.svc.cluster.local:3100
frontend_worker:
scheduler_address: glance-pd-logging-sg-loki-query-scheduler.loki.svc.cluster.local:9095
index_gateway:
mode: ring
ingester:
chunk_block_size: 524288
chunk_encoding: snappy
chunk_idle_period: 5m
chunk_retain_period: 5s
chunk_target_size: 1048576
max_chunk_age: 6m
limits_config:
deletion_mode: filter-and-delete
ingestion_burst_size_mb: 500
ingestion_rate_mb: 500
max_cache_freshness_per_query: 1m
max_global_streams_per_user: 0
max_line_size: 0
max_query_length: 168h
max_query_parallelism: 180
max_query_series: 200000
max_streams_per_user: 100000000
per_stream_rate_limit: 500MB
per_stream_rate_limit_burst: 5000MB
query_timeout: 900s
reject_old_samples: true
reject_old_samples_max_age: 168h
retention_period: 168h
retention_stream:
- period: 9120h
priority: 1
selector: '{job="glancetv-studio-api-audit"}'
split_queries_by_interval: 5m
volume_enabled: true
memberlist:
join_members:
- loki-memberlist
pattern_ingester:
enabled: false
querier:
max_concurrent: 6
query_ingesters_within: 5m
query_range:
align_queries_with_step: true
cache_label_results: true
cache_results: true
parallelise_shardable_queries: true
results_cache:
cache:
background:
writeback_buffer: 500000
writeback_goroutines: 1
writeback_size_limit: 500MB
default_validity: 168h
memcached_client:
addresses: dnssrvnoa+_memcached-client._tcp.glance-pd-logging-sg-loki-results-cache.loki.svc
consistent_hash: true
timeout: 500ms
update_interval: 1m
query_scheduler:
max_outstanding_requests_per_tenant: 48000
runtime_config:
file: /etc/loki/runtime-config/runtime-config.yaml
schema_config:
configs:
- from: "2024-02-21"
index:
period: 24h
prefix: tsdb_index_
object_store: gcs
schema: v13
store: tsdb
server:
grpc_listen_port: 9095
grpc_server_max_concurrent_streams: 100000000
grpc_server_max_recv_msg_size: 6665610280
grpc_server_max_send_msg_size: 6665610280
grpc_server_min_time_between_pings: 10s
grpc_server_ping_without_stream_allowed: true
http_listen_port: 3100
http_server_read_timeout: 600s
http_server_write_timeout: 600s
log_level: debug
storage_config:
bloom_shipper:
working_directory: /var/loki/data/bloomshipper
boltdb_shipper:
index_gateway_client:
server_address: dns+glance-pd-logging-sg-loki-index-gateway-headless.loki.svc.cluster.local:9095
hedging:
at: 250ms
max_per_second: 20
up_to: 3
tsdb_shipper:
index_gateway_client:
server_address: dns+glance-pd-logging-sg-loki-index-gateway-headless.loki.svc.cluster.local:9095
query_ready_num_days: 7
tracing:
enabled: false

Start with the compactor retention configuration and the GCS-backed TSDB setup described in the issue, then reproduce the behavior on Loki 3.4.2 while checking compactor logs and GCS contents. Done means retention removes both the older TSDB indexes and their associated chunks, not just the indexes.

Written by the indexing model from the issue text.