microsoft / microsoft/SynapseML
Facing error while training with large dataset : std::bad_alloc
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
Intermittently facing this issue of `std::bad_alloc` while training with the below configuration, can you please advise on the same
No of Executors : 30
No of Cores : 12, total 16 vcpu's
Ram : 128 gb Executor Memory : tried with 48 gb / 64 gb
Driver Memory : 64 gb
No of events : 250 Million
No of Features : 2010 columns
`Model Configuration : useSingleDatasetMode=True, numLeaves=512, featureFraction=0.8, numIterations=1024, useBarrierExecutionMode=True , validationIndicatorCol="validation" (0.4 Million records which can fit easily in driver memory)
`
Version : com.microsoft.azure:synapseml_2.12:0.9.5
22/07/03 13:11:12 INFO MemoryStore: Block broadcast_130_piece105 stored as bytes in memory (estimated size 4.0 MiB, free 30.4 GiB)
22/07/03 13:11:12 INFO MemoryStore: Block broadcast_130_piece151 stored as bytes in memory (estimated size 4.0 MiB, free 30.3 GiB)
22/07/03 13:11:12 INFO TorrentBroadcast: Reading broadcast variable 130 took 2395 ms
22/07/03 13:11:16 INFO MemoryStore: Block broadcast_130 stored as values in memory (estimated size 1884.0 MiB, free 28.5 GiB)
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
AB#1855963
Contributor guide
Assessment
This issue has not been assessed yet.