AcademySoftwareFoundation / AcademySoftwareFoundation/OpenShadingLanguage

Untracked memory in OSL

Open
#658 3 comments 0 reactions 1 assignee Claimed by @lgritz View on GitHub
Dominant language
C++
Stars
2.3k
Forks
414
Avg merge
3d 1h
Merged PRs (30d)
10

Description

Using valgrind's massif profiling tool, I've found several large blocks of memory that aren't accounted for, where the unaccounted memory is dwarfing the accounted. The biggest is that the ShadingContext memory isn't accounted for, including all the constant pools. The vectors that
point to the values for parameters and such are accounted in things like paramvals, but the actual values in the pools that those point to aren't accounted. The shader objects and their execution space also aren't accounted for. The final big source of missing memory is stuff that LLVM is using, but I can understand not counting that as it's much harder.

Here's what OSL reports.

```

OSL Shading Current Memory 86,549,052
OSL Shading Peak Memory 98,972,980

osl_mem_master_current 6,469,424
osl_mem_master_peak 6,469,424
osl_mem_master_ops_current 3,348,864
osl_mem_master_ops_peak 3,348,864
osl_mem_master_args_current 431,864
osl_mem_master_args_peak 431,864
osl_mem_master_syms_current 2,660,352
osl_mem_master_syms_peak 2,660,352
osl_mem_master_defaults_current 12,372
osl_mem_master_defaults_peak 12,372
osl_mem_master_consts_current 6,996
osl_mem_master_consts_peak 6,996
osl_mem_inst_current 80,079,628
osl_mem_inst_peak 92,503,556
osl_mem_inst_syms_current 57,871,904
osl_mem_inst_syms_peak 70,255,264
osl_mem_inst_paramvals_current 6,811,456
osl_mem_inst_paramvals_peak 6,811,456
osl_mem_inst_connections_current 2,783,484
osl_mem_inst_connections_peak 3,484,800

```

And here is a cleaned up version of the valgrind info for things I'm pretty sure aren't counted anywhere.

```
---- Shading context memory ----
->24.63% (301,596,672B) 0x871AC55: OSL::ShadingContext::ShadingContext(OSL::pvt::ShadingSystemImpl&, OSL::PerThreadInfo*)
->07.70% (94,248,960B) 0x871ABD8: OSL::ShadingContext::ShadingContext(OSL::pvt::ShadingSystemImpl&, OSL::PerThreadInfo*)
| ->07.70% (94,248,960B) 0x86C9F7C: OSL::pvt::ShadingSystemImpl::get_context(OSL::PerThreadInfo*, OpenImageIO::v1_5::TextureSystem::Perthread*)
->07.70% (94,248,960B) 0x871ABD8: OSL::ShadingContext::ShadingContext(OSL::pvt::ShadingSystemImpl&, OSL::PerThreadInfo*)
| ->07.70% (94,248,960B) 0x86C9F7C: OSL::pvt::ShadingSystemImpl::get_context(OSL::PerThreadInfo*, OpenImageIO::v1_5::TextureSystem::Perthread*)
->00.41% (5,043,792B) 0x86C9F68: OSL::pvt::ShadingSystemImpl::get_context(OSL::PerThreadInfo*, OpenImageIO::v1_5::TextureSystem::Perthread*)
->00.38% (4,712,448B) 0x871AA1C: OSL::ShadingContext::ShadingContext(OSL::pvt::ShadingSystemImpl&, OSL::PerThreadInfo*)
| ->00.38% (4,712,448B) 0x86C9F7C: OSL::pvt::ShadingSystemImpl::get_context(OSL::PerThreadInfo*, OpenImageIO::v1_5::TextureSystem::Perthread*)

---- actual value pools for strings, floats, ints, etc -----
->01.10% (13,486,024B) 0x7B132E8: __gnu_cxx::new_allocator::allocate(unsigned long, void const*) (new_allocator.h:89)
| ->01.10% (13,486,024B) 0x7B14468: std::_Vector_base >::_M_allocate(unsigned long) (stl_vector.h:140)
| ->00.65% (8,000,000B) 0x87B442A: OSL::pvt::RuntimeOptimizer::add_constant(OSL::pvt::TypeSpec const&, void const*, OpenImageIO::v1_5::TypeDesc)
| ->00.41% (5,035,640B) 0x7B14188: std::_Vector_base >::_Vector_base(unsigned long, std::allocator const&) (stl_vector.h:113)
| | ->00.41% (5,035,640B) 0x7B24094: std::_Vector_base >::_Vector_base(unsigned long, std::allocator const&) (stl_vector.h:110)
| | ->00.41% (5,035,640B) 0x7B147A8: std::vector >::vector(std::vector > const&) (stl_vector.h:242)
| | ->00.41% (4,985,984B) 0x7B23608: OSL::OSLQuery::Parameter::Parameter(OSL::OSLQuery::Parameter const&) (oslquery.h:61)
| ->00.04% (450,384B) 0x7B152BD: OpenImageIO::v1_5::ustring* std::vector >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator > > >(unsigned long, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >) (stl_vector.h:963)
| ->00.04% (450,384B) 0x7B15D28: std::vector >::operator=(std::vector > const&) (vector.tcc:164)
| ->00.04% (450,384B) 0x8737FD9: OSL::pvt::ShaderInstance::parameters(OpenImageIO::v1_5::ParamValueList const&)

->00.96% (11,764,220B) 0x65FD9EC: __gnu_cxx::new_allocator::allocate(unsigned long, void const*) (new_allocator.h:89)
| ->00.96% (11,764,220B) 0x65FE928: std::_Vector_base >::_M_allocate(unsigned long) (stl_vector.h:140)
| ->00.49% (6,023,576B) 0x68B6319: float* std::vector >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator > > >(unsigned long, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >) (stl_vector.h:963)
| | ->00.49% (6,023,576B) 0x68B6BD4: std::vector >::operator=(std::vector > const&) (vector.tcc:164)
| | ->00.49% (6,023,576B) 0x8737FBB: OSL::pvt::ShaderInstance::parameters(OpenImageIO::v1_5::ParamValueList const&)
| ->00.33% (4,000,000B) 0x68B62A1: float* std::vector >::_M_allocate_and_copy >(unsigned long, std::move_iterator, std::move_iterator) (stl_vector.h:963)
| | ->00.33% (4,000,000B) 0x68B658A: std::vector >::reserve(unsigned long) (vector.tcc:72)
| | ->00.33% (4,000,000B) 0x87C0D25: OSL::pvt::ShadingSystemImpl::alloc_float_constants(unsigned long)
| ->00.14% (1,705,484B) 0x68B5EE4: std::_Vector_base >::_Vector_base(unsigned long, std::allocator const&) (stl_vector.h:113)
| | ->00.14% (1,705,484B) 0x68B9788: std::_Vector_base >::_Vector_base(unsigned long, std::allocator const&) (stl_vector.h:110)
| | ->00.14% (1,705,484B) 0x6CA5504: std::vector >::vector(std::vector > const&) (stl_vector.h:242)
| | ->00.14% (1,705,484B) 0x7B235DF: OSL::OSLQuery::Parameter::Parameter(OSL::OSLQuery::Parameter const&) (oslquery.h:61)

| ->00.44% (5,404,828B) 0x65597EC: std::_Vector_base >::_M_allocate(unsigned long) (stl_vector.h:140)
| | ->00.33% (4,000,000B) 0x66A0411: int* std::vector >::_M_allocate_and_copy(unsigned long, int*, int*) (stl_vector.h:963)
| | | ->00.33% (4,000,000B) 0x66A19D3: std::vector >::reserve(unsigned long) (vector.tcc:72)
| | | ->00.33% (4,000,000B) 0x87B3D7F: OSL::pvt::RuntimeOptimizer::add_constant(OSL::pvt::TypeSpec const&, void const*, OpenImageIO::v1_5::TypeDesc)
| | ->00.07% (874,648B) 0x665847C: std::vector >::_M_insert_aux(__gnu_cxx::__normal_iterator > >, int const&) (vector.tcc:322)
| | | ->00.07% (874,608B) 0x877B2C7: OSL::pvt::OSOReaderToMaster::instruction_arg(char const*)
| | ->00.04% (455,452B) 0x6657455: int* std::vector >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator > > >(unsigned long, __gnu_cxx::__normal_iterator > >, __gnu_cxx::__normal_iterator > >) (stl_vector.h:963)
| | | ->00.04% (455,452B) 0x6657B50: std::vector >::operator=(std::vector > const&) (vector.tcc:164)
| | | ->00.04% (455,452B) 0x8737F9D: OSL::pvt::ShaderInstance::parameters(OpenImageIO::v1_5::ParamValueList const&)

-- the actual shader objects ---
->01.05% (12,800,024B) 0x86D335C: OSL::pvt::ShadingSystemImpl::Shader(OpenImageIO::v1_5::string_view, OpenImageIO::v1_5::string_view, OpenImageIO::v1_5::string_view)
| ->01.05% (12,800,024B) 0x86D38E9: OSL::ShadingSystem::Shader(OpenImageIO::v1_5::string_view, OpenImageIO::v1_5::string_view, OpenImageIO::v1_5::string_view)

--- execution scratch? ---
->00.43% (5,219,886B) 0x6CDA194: __gnu_cxx::new_allocator::allocate(unsigned long, void const*) (new_allocator.h:89)
| ->00.43% (5,219,886B) 0x7870E74: std::_Vector_base >::_M_allocate(unsigned long) (stl_vector.h:140)
| ->00.43% (5,219,886B) 0x78735FB: std::vector >::_M_fill_insert(__gnu_cxx::__normal_iterator > >, unsigned long, char const&) (vector.tcc:414)
| ->00.43% (5,219,044B) 0x871BC68: OSL::ShadingContext::execute(OSL::ShaderGroup&, OSL::ShaderGlobals&, bool)

----- LLVM ----
->01.71% (20,971,520B) 0x912397B: llvm::ValueHandleBase::AddToUseList()
| ->01.03% (12,582,912B) 0x8DAF743: llvm::ValueMap >::operator[](llvm::Value const* const&)
| | ->00.69% (8,388,608B) 0x8E8E488: (anonymous namespace)::PruningFunctionCloner::CloneBlock(llvm::BasicBlock const*, std::vector >&)
| | ->00.34% (4,194,304B) 0x8EE83C1: llvm::MapValue(llvm::Value const*, llvm::ValueMap >&, llvm::RemapFlags, llvm::ValueMapTypeRemapper*)
| | | ->00.34% (4,194,304B) 0x8EE81C6: llvm::MapValue(llvm::Value const*, llvm::ValueMap >&, llvm::RemapFlags, llvm::ValueMapTypeRemapper*)
| | | ->00.34% (4,194,304B) 0x8EE8517: llvm::RemapInstruction(llvm::Instruction*, llvm::ValueMap >&, llvm::RemapFlags, llvm::ValueMapTypeRemapper*)
| ->00.34% (4,194,304B) 0x8E8E4DC: (anonymous namespace)::PruningFunctionCloner::CloneBlock(llvm::BasicBlock const*, std::vector >&)
| | ->00.34% (4,194,304B) 0x8E8EF79: llvm::CloneAndPruneFunctionInto(llvm::Function*, llvm::Function const*, llvm::ValueMap >&, bool, llvm::SmallVectorImpl&, char const*, llvm::ClonedCodeInfo*, llvm::DataLayout const*, llvm::Instruction*)
| ->00.34% (4,194,304B) 0x8B42E96: llvm::BitcodeReaderValueList::push_back(llvm::Value*)
| | ->00.34% (4,194,304B) 0x8B3F5F3: llvm::BitcodeReader::ParseModule(bool)

->00.61% (7,425,672B) 0x9121D11: llvm::User::operator new(unsigned long, unsigned int)
| ->00.35% (4,253,040B) 0x9090BB6: llvm::ConstantInt::get(llvm::LLVMContext&, llvm::APInt const&)
| | ->00.19% (2,339,352B) 0x89B7C6C: llvm::SelectionDAG::getConstant(llvm::APInt const&, llvm::EVT, bool)
| | | ->00.16% (1,913,688B) 0x89B7D7E: llvm::SelectionDAG::getConstant(unsigned long, llvm::EVT, bool)
| | | ->00.03% (421,632B) 0x89B8D87: llvm::SelectionDAG::FoldConstantArithmetic(unsigned int, llvm::EVT, llvm::SDNode*, llvm::SDNode*)
| | ->00.09% (1,161,144B) 0x8770E51: OSL::pvt::LLVM_Util::constant(unsigned long)
| | | ->00.09% (1,161,144B) 0x87730AE: OSL::pvt::LLVM_Util::constant_ptr(void*, llvm::PointerType*)
| | | ->00.07% (893,304B) 0x8759E5E: OSL::pvt::BackendLLVM::llvm_assign_initial_value(OSL::pvt::Symbol const&)
| | ->00.06% (679,392B) 0x9090D66: llvm::ConstantInt::get(llvm::IntegerType*, unsigned long, bool)
| ->00.25% (3,018,640B) 0x9093F54: llvm::ConstantCreator::create(llvm::Type*, llvm::ExprMapKeyType const&, unsigned short)

->00.51% (6,249,363B) 0x91475DA: llvm::MallocSlabAllocator::Allocate(unsigned long)
| ->00.42% (5,193,728B) 0x9147382: llvm::BumpPtrAllocator::StartNewSlab()
| | ->00.42% (5,173,248B) 0x9147492: llvm::BumpPtrAllocator::Allocate(unsigned long, unsigned long)
| | | ->00.16% (2,015,232B) 0x911AF7F: llvm::StructType::setBody(llvm::ArrayRef, bool)
| | | ->00.11% (1,310,720B) 0x911BCF0: llvm::PointerType::get(llvm::Type*, unsigned int)
| | | ->00.07% (892,928B) 0x911B64D: llvm::StructType::create(llvm::LLVMContext&, llvm::StringRef)
| | | ->00.07% (802,816B) 0x911DAC0: llvm::FunctionType::get(llvm::Type*, llvm::ArrayRef, bool)
| | | ->00.01% (135,168B) 0x911C772: llvm::ArrayType::get(llvm::Type*, unsigned long)
| ->00.09% (1,055,635B) 0x91474C8: llvm::BumpPtrAllocator::Allocate(unsigned long, unsigned long)
| ->00.09% (1,055,635B) 0x911AF7F: llvm::StructType::setBody(llvm::ArrayRef, bool)
| ->00.09% (1,055,635B) 0x911B6FE: llvm::StructType::create(llvm::LLVMContext&, llvm::ArrayRef, llvm::StringRef, bool)
| ->00.09% (1,055,635B) 0x875AFC7: OSL::pvt::BackendLLVM::llvm_type_groupdata()

->00.37% (4,527,960B) 0x9094622: std::_Rb_tree, std::pair const, llvm::ConstantExpr*>, std::_Select1st const, llvm::ConstantExpr*> >, std::less >, std::allocator const, llvm::ConstantExpr*> > >::_M_insert_(std::_Rb_tree_node_base const*, std::_Rb_tree_node_base const*, std::pair const, llvm::ConstantExpr*> const&)
| ->00.37% (4,491,720B) 0x909482E: std::_Rb_tree, std::pair const, llvm::ConstantExpr*>, std::_Select1st const, llvm::ConstantExpr*> >, std::less >, std::allocator const, llvm::ConstantExpr*> > >::_M_insert_unique(std::pair const, llvm::ConstantExpr*> const&)
| | ->00.37% (4,491,720B) 0x9094968: std::_Rb_tree, std::pair const, llvm::ConstantExpr*>, std::_Select1st const, llvm::ConstantExpr*> >, std::less >, std::allocator const, llvm::ConstantExpr*> > >::_M_insert_unique_(std::_Rb_tree_const_iterator const, llvm::ConstantExpr*> >, std::pair const, llvm::ConstantExpr*> const&)
| | ->00.37% (4,491,720B) 0x9094C3A: llvm::ConstantUniqueMap::getOrCreate(llvm::Type*, llvm::ExprMapKeyType const&)
| | ->00.37% (4,491,720B) 0x9091CC7: llvm::ConstantExpr::getIntToPtr(llvm::Constant*, llvm::Type*)

->00.30% (3,670,016B) 0x9096FAE: llvm::DenseMapBase, llvm::DenseMapAPIntKeyInfo::KeyTy, llvm::ConstantInt*, llvm::DenseMapAPIntKeyInfo>::grow(unsigned int)
| ->00.30% (3,670,016B) 0x9090C16: llvm::ConstantInt::get(llvm::LLVMContext&, llvm::APInt const&)
| ->00.30% (3,670,016B) 0x8770E51: OSL::pvt::LLVM_Util::constant(unsigned long)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.