Job submit lua plugin failing

I need to enable Lua job submit plugin on cluster creation. This plugin enables custom job submission logic programmed through a Lua script `/opt/slurm/etc/job_submit.lua`.

The problem is that this plugin requires `/opt/slurm/etc/job_submit.lua` file to be in place before `slurmctld` starts, otherwise `slurmctld` startup will fail.

I need a mechanism to inject a file to the head node right after all the software is installed, but before it's started. So that we can inject the `job_submit.lua` file into the head node correctly.

**Required Info:**
- AWS ParallelCluster version [e.g. 3.1.1]: 3.12.0
- Full cluster configuration without any credentials or personal data.
```
HeadNode:
CustomActions:
OnNodeConfigured:
Args:
- head
Script: >-
s3://MYBUCKET/scripts/postscript.sh
Iam:
AdditionalIamPolicies:
- Policy: '***MASKED_CUSTOM_POLICY_ARN***'
- Policy: 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore'
- Policy: 'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy'
S3Access:
- BucketName: MYBUCKET
EnableWriteAccess: false
Image:
CustomAmi: ami-1234455677
InstanceType: t3.large
LocalStorage:
RootVolume:
Encrypted: true
Size: 150
Networking:
AdditionalSecurityGroups:
- sg-1122344
SubnetId: subnet-11223344
SharedStorageType: Ebs
Ssh:
AllowedIps: 10.0.0.0/8
KeyName: pcluster_deployer_key-dev
Iam:
ResourcePrefix: pcluster-devtest-
Image:
CustomAmi: ami-1234455677
Os: rocky8
Imds:
ImdsSupport: v1.0
LoginNodes:
Pools:
- Count: 1
CustomActions:
OnNodeConfigured:
Args:
- login
Script: >-
s3://MYBUCKET/scripts/postscript.sh
Iam:
AdditionalIamPolicies:
- Policy: '***MASKED_CUSTOM_POLICY_ARN***'
- Policy: 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore'
- Policy: 'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy'
Image:
CustomAmi: ami-1234455677
InstanceType: t3.medium
Name: login
Networking:
AdditionalSecurityGroups:
- sg-1122344
SubnetIds:
- subnet-11223344
Ssh:
AllowedIps: 10.0.0.0/8
KeyName: pcluster_deployer_key-dev
Monitoring:
Alarms:
Enabled: true
Dashboards:
CloudWatch:
Enabled: true
DetailedMonitoring: true
Logs:
CloudWatch:
DeletionPolicy: Delete
Enabled: true
RetentionInDays: 90
Rotation:
Enabled: true
Region: us-west-2
Scheduling:
Scheduler: slurm
SlurmQueues:
- AllocationStrategy: lowest-price
CapacityType: ONDEMAND
ComputeResources:
- CustomSlurmSettings:
CPUSpecList: 1
DisableSimultaneousMultithreading: true
DynamicNodePriority: 14000
Instances:
- InstanceType: t3.xlarge
MaxCount: 3
MinCount: 1
Name: devtest-xl
StaticNodePriority: 14000
- CustomSlurmSettings:
CPUSpecList: 1
DisableSimultaneousMultithreading: true
DynamicNodePriority: 32000
Instances:
- InstanceType: t3.2xlarge
MaxCount: 3
MinCount: 1
Name: devtest-2xl
StaticNodePriority: 32000
ComputeSettings:
LocalStorage:
EphemeralVolume:
MountDir: /ephemeral
RootVolume:
Encrypted: true
Size: 200
CustomActions:
OnNodeConfigured:
Args:
- compute
Script: >-
s3://MYBUCKET/scripts/postscript.sh
CustomSlurmSettings:
AllowQos: gold
Iam:
AdditionalIamPolicies:
- Policy: '***MASKED_CUSTOM_POLICY_ARN***'
- Policy: 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore'
- Policy: 'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy'
S3Access:
- BucketName: MYBUCKET
EnableWriteAccess: false
Image:
CustomAmi: ami-1234455677
JobExclusiveAllocation: false
Name: devtest1
Networking:
AdditionalSecurityGroups:
- sg-1122344
SubnetIds:
- subnet-11223344
- subnet-11223345
- subnet-11223346
- subnet-11223347
SlurmSettings:
CustomSlurmSettings:
- TaskPluginParam: SlurmdOffSpec
- PrologFlags: 'Alloc,Contain,X11'
- DefMemPerCPU: 4096
- JobSubmitPlugins: "lua"
Dns:
HostedZoneId: ZAAAAAAAAAAAAAAAAA
EnableMemoryBasedScheduling: true
MungeKeySecretArn: '********MASKED********'
QueueUpdateStrategy: DRAIN
ScaledownIdletime: 30

```
- Cluster name: devtest
- Output of `pcluster describe-cluster` command.

**Bug description and how to reproduce:**

1. Create the cluster from config above using `pcluster create-cluster`
2. Cluster creation fails:
- in the `slurmd.log` ther eare messages about "job_submit.lua not found"