NVIDIA / NVIDIA/cutlass

[BUG] Cutlass' static library defines fewer symbols than the shared library

Open
#1,828 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

? - Needs Triage bug inactive-30d inactive-90d
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Describe the bug
I've recently been playing around with building cutlass as both a shared and static library. However, I've noticed that the static library often omits some symbols that are present in the shared library.

Steps/Code to reproduce bug
I've encountered this issue on 3a8c01a18b. I've not tried any other cutlass versions, but I don't think there would be a difference.

First, make and navigate to a build directory from the cutlass root directory. Then, configure cutlass' build system using the following cmake command:

cmake .. -DCUTLASS_NVCC_ARCHS=90 -DCUTLASS_ENABLE_TESTS=OFF -DCUTLASS_UNITY_BUILD_ENABLED=ON -DCUTLASS_LIBRARY_OPERATIONS=gemm

The NVCC archs, tests and unity builds aren't that important for reproducibility: I've seen the issue occur in many different configurations of flags.

Then, compile both the static and the shared libraries via make.

make cutlass_library
make cutlass_library_static

This should succeed. Finally, run the following commands and notice the different outputs:

readelf -Ws tools/library/libcutlass.a --demangle | grep cutlass::library::from_string | wc -l ; I get 12
readelf -Ws tools/library/libcutlass.so --demangle | grep cutlass::library::from_string | wc -l ; I get 24

Moreover, if I actually look at the symbols themselves, I see that the static library doesn't actually define all of the symbols.

readelf -Ws tools/library/libcutlass.a --demangle | grep "cutlass::library::initialize_all(cutlass::library::Manifest&)

   9: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND cutlass::library::initialize_all(cutlass::library::Manifest&)
   7: 0000000000000000    27 FUNC    GLOBAL DEFAULT    1 cutlass::library::initialize_all(cutlass::library::Manifest&)

Whereas the shared library does define these:

readelf -Ws tools/library/libcutlass.so --demangle | grep "cutlass::library::intialize_all(cutlass::library::Manifest&)

 13441: 00000000005c5ee0    27 FUNC    GLOBAL DEFAULT   14 cutlass::library::initialize_all(cutlass::library::Manifest&)
 15449: 00000000005c5ee0    27 FUNC    GLOBAL DEFAULT   14 cutlass::library::initialize_all(cutlass::library::Manifest&)

This means that a third-party application that wants to statically link against cutlass is somewhat limited.

Expected behavior
My expectation would be that the static library would export all non-explicitly hidden symbols. I appreciate there's some subtle linker behaviour that can influence how symbols are exported, but this seems unusual.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the CMake targets and build commands for tools/library/libcutlass.a and tools/library/libcutlass.so, using the reported configuration. Compare how the static and shared targets collect symbols, then reproduce the discrepancy with readelf and the from_string or initialize_all queries. Done means the static library defines the same non-hidden symbols as the shared library.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.