Multi-node parallelism on slurm clusters

Hi,

Let's say, I have a slurm cluster that contains 100 nodes, each node has 100 cores. Assuming I have 10000 tasks.
This is my current code:

```
dist_executor = SlurmPipelineExecutor(
job_name=f"filter",
pipeline=[
JsonlReader(
args.input_dataset,
limit=args.data_limit,
file_progress=True,
doc_progress=True,
text_key=args.text_key,
glob_pattern="*.jsonl.zst",
),
LanguageFilter(
language_threshold=0.65,
languages=[Languages.english],
exclusion_writer=JsonlWriter(
f"{LOCAL_PATH}/removed/1_non_english/",
output_filename="data/${rank}.jsonl",
compression=None
# folder structure: language/dump/file
),
),
JsonlWriter(
output_folder=f"{LOCAL_PATH}/{args.output_name}/intermediate_filtered",
output_filename="${rank}.jsonl",
compression=None
),
],
tasks=args.n_tasks,
workers=args.n_workers,
time="20:00:00",
partition=args.partition1,
account=args.account,
logging_dir=f"{LOCAL_LOGS_PATH}/intermediate_filtered",
slurm_logs_folder=f"{LOCAL_LOGS_PATH}/intermediate_filtered_slurm_logs",
randomize_start_duration=180, # don't hit the bucket all at once with the list requests
cpus_per_task=12,
# qos="high",
mem_per_cpu_gb=3,
)
```

I find that the `workers` is the number of nodes instead of the number of CPU cores. Is my understanding correct?
Then, it seems to me that my 10000 tasks will be executed node by node. For example, the node_1 will process task_1 and then task_101, then task_201. It does not fully utilize all the CPU cores in a single node. I expect all the tasks (task_1, task_101, task_201 ...) to be assigned to node_1 at the beginning and **executed parallelly**.

If I want to distribute my jobs to each node and fully utilize every core, what should I change to my code?

Thanks!