aws / aws/aws-parallelcluster

DataRepositoryAssociations reports sucess but files don't show up

Open
#6,029 2 comments 0 reactions 0 assignees View on GitHub
3.x Backlog Bug
Dominant language
Python
Stars
888
Forks
314
Avg merge
1d 10h
Merged PRs (30d)
43

Description

If you have an active AWS support contract, please open a case with AWS Premium Support team using the below documentation to report the issue:
https://docs.aws.amazon.com/awssupport/latest/user/case-management.html

Before submitting a new issue, please search through open [GitHub Issues](https://github.com/aws/aws-parallelcluster/issues) and check out the [troubleshooting documentation](https://docs.aws.amazon.com/parallelcluster/latest/ug/troubleshooting-v3.html).

Please make sure to add the following data in order to facilitate the root cause detection.

**Required Info:**
- AWS ParallelCluster version [e.g. 3.1.1]: 3.8.0
- Full cluster configuration without any credentials or personal data.
[config_180124.txt](https://github.com/aws/aws-parallelcluster/files/14035973/config_180124.txt)
- Cluster name: pingu
- Output of `pcluster describe-cluster` command.
{
"creationTime": "2024-01-24T08:15:03.919Z",
"headNode": {
"launchTime": "2024-01-24T08:24:43.000Z",
"instanceId": "i-0d4908c5ce11c0d3d",
"publicIpAddress": "16.171.226.150",
"instanceType": "c5n.large",
"state": "running",
"privateIpAddress": "10.0.0.228"
},
"version": "3.8.0",
"clusterConfiguration": {
"url": "https://parallelcluster-21a2b8a6db2e042c-v1-do-not-delete.s3.eu-north-1.amazonaws.com/parallelcluster/3.8.0/clusters/pingu-7l9fj77yzjl1b3wm/configs/cluster-config.yaml?versionId=9PiFICNIiOq.Na8c1DRP5lB6adTQOCIO&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIARP4E2QSTIOX7FEXY%2F20240124%2Feu-north-1%2Fs3%2Faws4_request&X-Amz-Date=20240124T091625Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=2c0831e5d780b6e5113b4f61c99421ffab265b7105ce00a7a5fb4fe9db706562"
},
"tags": [
{
"value": "hpc-cluster",
"key": "CFD"
},
{
"value": "3.8.0",
"key": "parallelcluster:version"
},
{
"value": "pingu",
"key": "parallelcluster:cluster-name"
}
],
"cloudFormationStackStatus": "CREATE_COMPLETE",
"clusterName": "pingu",
"computeFleetStatus": "RUNNING",
"cloudformationStackArn": "arn:aws:cloudformation:eu-north-1:102820906150:stack/pingu/ace48b20-ba90-11ee-8eef-0ec568337136",
"lastUpdatedTime": "2024-01-24T08:15:03.919Z",
"region": "eu-north-1",
"clusterStatus": "CREATE_COMPLETE",
"scheduler": {
"type": "slurm"
}
}
- [Optional] Arn of the cluster CloudFormation main stack: See output above

**Bug description and how to reproduce:**
We are building a cluster through the config and use the DataRepositoryAssociations to set up links to s3 buckets. The creation seem to report no errors, but we have issues with files not showing up for one of the DRAs.

Currently we have 2 DRAs in the config. The first specified executes the Data repository task specified properly and the files are available when logging in. The second (CFD-setup) does report success for the Data repository task but the folder is empty on the instance. When creating a new Data repository task it executes successfully and updates the folder content - thus it seems like the issue is not with the configuration of the DRA but some process not fully completed during setup.

Please be sure to attach the following logs:
* From Head node: `/var/log/cloud-init.log`, `/var/log/cfn-init.log` and `/var/log/chef-client.log`
[cfn-init.log](https://github.com/aws/aws-parallelcluster/files/14035899/cfn-init.log)
[chef-client.log](https://github.com/aws/aws-parallelcluster/files/14035900/chef-client.log)
[cloud-init.log](https://github.com/aws/aws-parallelcluster/files/14035901/cloud-init.log)
* From Compute node: `/var/log/cloud-init-output.log`.

**Additional context:**
Any other context about the problem. E.g.:
- CLI logs: `~/.parallelcluster/pcluster-cli.log`
[pcluster-cli.log](https://github.com/aws/aws-parallelcluster/files/14035927/pcluster-cli.log)

- Custom bootstrap scripts, if any
- Screenshots, if useful.
![image](https://github.com/aws/aws-parallelcluster/assets/74775652/c8e77d2a-0498-4166-b332-7a2e4bd0a352)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.