llnl / llnl/dtcmp

Wrong number of groups with `DTCMP_Rankv()`

Open
#19 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
10
Forks
9
PR merge metrics
No merged PRs in 30d

Description

Dear DTCMP developers,

I encountered a weird behavior with DTCMP_Rankv() with some combinations of number of values and tasks. Consider this example program:

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <dtcmp.h>
#include <mpi.h>
#include <stdint.h>

int rankv(uint64_t incount, uint64_t* values) {
    int rank, ranks;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &ranks);


    /* get extent of key type */
    MPI_Aint value_lb, value_extent;
    MPI_Type_get_extent(MPI_UINT64_T, &value_lb, &value_extent);

    /* compute size of sort element and allocate buffer */
    size_t rankbufsize = (size_t)value_extent * incount;
    void* rankbuf = malloc(rankbufsize);
    char* rankptr = (char*)rankbuf;

    for(int i=0; i<incount; i++) {
        memcpy(rankptr, &values[i], sizeof(uint64_t));
        rankptr += value_extent;
    }

    /* rank hardlinks by inode */
    uint64_t groups;
    uint64_t output_bytes = incount * sizeof(uint64_t);
    uint64_t* group_id    = (uint64_t*) malloc(output_bytes);
    uint64_t* group_ranks = (uint64_t*) malloc(output_bytes);
    uint64_t* group_rank  = (uint64_t*) malloc(output_bytes);
    int rank_rc = DTCMP_Rankv(
                      (int)incount, rankbuf, &groups, group_id, group_ranks,
                      group_rank, MPI_UINT64_T, MPI_UINT64_T, DTCMP_OP_UINT64T_ASCEND, DTCMP_FLAG_NONE,
                      MPI_COMM_WORLD);

    if (rank_rc != DTCMP_SUCCESS) {
        printf("Failed to rank values\n");
        return -1;
    }

    printf("rank: %d DTCMP_Rankv incount: %lu rank_rc: %d groups: %lu\n", rank, incount, rank_rc, groups);

    /* free off temporary memory */
    free(group_id);
    free(group_ranks);
    free(group_rank);
    /* free input buffer holding rank elements */
    free(rankbuf);

    return 0;
}

int main(int argc, char* argv[]) {

    MPI_Init(NULL, NULL);
    DTCMP_Init();

    uint64_t incount, *values = NULL;

    int rank, ranks;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &ranks);

    if (rank== 0 || rank==3) {
        incount = 1;
        values = (uint64_t*) malloc(sizeof(uint64_t));
	values[0] = 64; // or anything else
    } else {
        incount = 0;
        values = NULL;
    }
    rankv(incount, values);

    DTCMP_Finalize();
    MPI_Finalize();

    return EXIT_SUCCESS;

}

There is 1 value (64) on rank 0 and 3, no value on other tasks. When I run with 4 tasks, it finds 2 groups:

$ mpirun -N 4 rankv 
rank: 1 DTCMP_Rankv incount: 0 rank_rc: 0 groups: 2
rank: 2 DTCMP_Rankv incount: 0 rank_rc: 0 groups: 2
rank: 3 DTCMP_Rankv incount: 1 rank_rc: 0 groups: 2
rank: 0 DTCMP_Rankv incount: 1 rank_rc: 0 groups: 2

I would expect it to find only one group (for value 64 on ranks 0 and 3) in this case. Any idea?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided four-task MPI program and inspect the DTCMP_Rankv entry point. Check how zero-count ranks and equal values contribute to the reported groups; done means the example reports one group for value 64 on ranks 0 and 3 while preserving a successful return code.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.