Support FSx for Lustre MetadataConfiguration on ParallelCluster-managed Persistent 2 filesystems
- Dominant language
- Python
- Stars
- 888
- Forks
- 314
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 43
Description
## Feature request
Please expose Amazon FSx for Lustre enhanced metadata performance configuration for ParallelCluster-managed Persistent 2 filesystems.
### Use case
We create short-lived sequencing-analysis clusters with FSx Lustre storage and S3 data repository associations. We want ParallelCluster and its CloudFormation stack to own filesystem creation and deletion so that the storage lifecycle follows the cluster. Creating FSx separately and attaching it by FileSystemId makes that lifecycle harder to manage and increases the risk of leaving billable resources behind.
### Current limitation
In ParallelCluster 3.16.1, FsxLustreSettings does not expose MetadataConfiguration, and the managed FSx CloudFormation renderer does not pass it to the filesystem resource.
FSx requires the metadata configuration to be specified when the filesystem is created. It cannot be added later to a filesystem created without that configuration. Consequently, creating a filesystem through ParallelCluster and updating it after creation does not recover this capability.
AWS documentation: https://docs.aws.amazon.com/fsx/latest/LustreGuide/managing-metadata-performance.html
### Requested behavior
Support a configuration such as the following (proposed syntax):
```yaml
SharedStorage:
- Name: analysis-fsx
StorageType: FsxLustre
MountDir: /fsx
FsxLustreSettings:
DeploymentType: PERSISTENT_2
FileSystemTypeVersion: "2.15"
StorageType: SSD
StorageCapacity: 12000
PerUnitStorageThroughput: 250
MetadataConfiguration:
Mode: AUTOMATIC
```
Also support USER_PROVISIONED with an explicit Iops value, subject to FSx service constraints.
Please include:
- Schema/model support and rendering into AWS::FSx::FileSystem LustreConfiguration.MetadataConfiguration.
- Validation for supported deployment types, Lustre versions, storage types, modes, and IOPS combinations.
- Documented update behavior consistent with FSx restrictions.
- No change to existing behavior when the field is omitted.
This is about metadata operation throughput and later metadata scalability; it is separate from S3 lazy hydration, data import policies, and cache release.
Contributor guide
Research direction
Start with the FsxLustreSettings schema/model and the managed FSx CloudFormation renderer mentioned in the issue. Trace how existing Persistent 2 settings reach AWS::FSx::FileSystem, then review the linked FSx metadata-performance documentation for supported modes and IOPS constraints. Done means the field is rendered, invalid combinations are rejected, updates respect FSx restrictions, and omitted configuration remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100