Allow Setting Both Memory Requests and Limits for HdfsCluster Pods

Current Behavior

The HdfsCluster Custom Resource Definition (CRD) currently only allows specifying memory.limit under HdfsCluster.spec.nameNodes.config.resources.memory. This configuration automatically sets memory.request equal to memory.limit for pods.

Problem

This behavior is problematic for CI platforms with limited memory resources. Forcing the memory request to match the limit can lead to resource scheduling issues, making it difficult to run HdfsCluster pods in resource-constrained environments.

Desired Behavior

Allow users to independently set both memory.request and memory.limit under HdfsCluster.spec.nameNodes.config.resources. For example:

spec:
  nameNodes:
    config:
      resources:
        memory:
          request: "512Mi"
          limit: "1Gi"

This flexibility ensures that pods can be scheduled even on CI platforms with tight resource constraints.

Proposed Solution

• Update the HdfsCluster CRD to support separate request and limit fields for memory under resources.
• Ensure backward compatibility by defaulting request to the same value as limit if request is not explicitly set.

Impact

This change would make the HdfsCluster CRD more usable in diverse environments, including CI platforms with restricted memory resources.

Additional Context

This enhancement would enable more efficient use of resources while maintaining compatibility with existing configurations.

Locate the HdfsCluster CRD definition and the resource defaulting or reconciliation path that builds pod resources. Check how the current memory limit is represented and propagated, then verify the schema and behavior for separate request and limit values. Done means explicit requests are honored while omitted requests retain the existing default behavior.