alibaba / alibaba/x-deeplearning
memory leak in deepctr example
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
I ran the deepctr example with some minor modify( change epoch, batch_size, embedding_size), then I found the memory usage go up to 10GB after 1 minute.
## Here is my mofiy
```diff
diff --git a/xdl/examples/deepctr/deepctr.py b/xdl/examples/deepctr/deepctr.py
index 343f352..2ef9106 100644
--- a/xdl/examples/deepctr/deepctr.py
+++ b/xdl/examples/deepctr/deepctr.py
@@ -20,7 +20,7 @@ reader = xdl.DataReader("r1", # name of reader
paths=["./data.txt"], # file paths
enable_state=False) # enable reader state
-reader.epochs(1).threads(1).batch_size(10).label_count(1)
+reader.epochs(300).threads(1).batch_size(100).label_count(1)
reader.feature(name='sparse0', type=xdl.features.sparse)\
.feature(name='sparse1', type=xdl.features.sparse)\
.feature(name='deep0', type=xdl.features.dense, nvec=256)
@@ -29,11 +29,11 @@ reader.startup()
def train():
batch = reader.read()
sess = xdl.TrainSession()
- emb1 = xdl.embedding('emb1', batch['sparse0'], xdl.TruncatedNormal(stddev=0.001), 8, 1024, vtype='hash')
- emb2 = xdl.embedding('emb2', batch['sparse1'], xdl.TruncatedNormal(stddev=0.001), 8, 1024, vtype='hash')
+ emb1 = xdl.embedding('emb1', batch['sparse0'], xdl.TruncatedNormal(stddev=0.001), 128, 1024, vtype='hash')
+ emb2 = xdl.embedding('emb2', batch['sparse1'], xdl.TruncatedNormal(stddev=0.001), 128, 1024, vtype='hash')
loss = model(batch['deep0'], [emb1, emb2], batch['label'])
train_op = xdl.SGD(0.5).optimize()
- log_hook = xdl.LoggerHook(loss, "loss:{0}", 10)
+ log_hook = xdl.LoggerHook(loss, "loss:{0}", 100)
sess = xdl.TrainSession(hooks=[log_hook])
while not sess.should_stop():
sess.run(train_op)
```
## Here is the output of `top` command
```bash
18142 root 20 0 45.101g 0.011t 61252 S 367.1 4.7 15:16.10 python deepctr.py --run=local
```
Seems the memory grow up about 100MB per epoch.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with xdl/examples/deepctr/deepctr.py and reproduce the report using the changed epochs, batch size, and embedding size, running it with `python deepctr.py --run=local`. Monitor memory across epochs and inspect the training loop and session setup. Done means the example no longer grows by about 100 MB per epoch under the reported configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100