Unusually Long Cali-Query Time

Nobody has claimed this yet.

Greetings,

We are trying to create a timeline of MPI function calls amongst 8 ranks that are performing irregular communication in AMR code, so the MPI whitelist consists of only MPI_Isend, MPI_Irecv, and MPI_Allreduce. The ranks generate cali files that are 45 megabytes each. My cali-query command to generate a timeline is the following:

`cali-query -t -o query.out msgtrace* --sort-by=time.offset
`

The execution of the previous command runs but never completes and is killed after ~50 minutes of running, even when using multiple threads since we have multiple .cali files generated (specified with the --threads option).

We have CALI_MARK_FUNCTION macro's in every function that contains communication to track the parent functions of the MPI_Function calls.

Our caliper configuration within the slurm job script is the following:

`export CALI_SERVICES_ENABLE=event,recorder,timestamp,trace`
`export CALI_EVENT_TRIGGER=function,mpi.function`
`export CALI_TIMER_SNAPSHOT_DURATION=false`
`export CALI_TIMER_INCLUSIVE_DURATION=false`
`export CALI_TIMER_OFFSET=true`
`export CALI_MPI_MSG_TRACING=true`
`export CALI_MPI_WHITELIST=MPI_Isend,MPI_Irecv,MPI_Allreduce`
`export CALI_RECORDER_FILENAME=msgtrace-%mpi.rank%.cali`

We are wondering if this is the appropriate way of using Caliper to create a timeline table of MPI Function calls within our code base. Any suggestions are welcomed.

Thank you!

Start by reproducing the `cali-query -t -o query.out msgtrace* --sort-by=time.offset` command against the recorded `.cali` files, using the reported MPI tracing configuration and thread options. Determine whether the command is an appropriate way to produce the MPI timeline and document a workable resolution or the conditions causing it not to complete.

Written by the indexing model from the issue text.