alibaba / alibaba/x-deeplearning

memory leak in deepctr example

Open
#45 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
PureBasic
Stars
4.3k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

I ran the deepctr example with some minor modify( change epoch, batch_size, embedding_size), then I found the memory usage go up to 10GB after 1 minute.
## Here is my mofiy
```diff
diff --git a/xdl/examples/deepctr/deepctr.py b/xdl/examples/deepctr/deepctr.py
index 343f352..2ef9106 100644
--- a/xdl/examples/deepctr/deepctr.py
+++ b/xdl/examples/deepctr/deepctr.py
@@ -20,7 +20,7 @@ reader = xdl.DataReader("r1", # name of reader
paths=["./data.txt"], # file paths
enable_state=False) # enable reader state

-reader.epochs(1).threads(1).batch_size(10).label_count(1)
+reader.epochs(300).threads(1).batch_size(100).label_count(1)
reader.feature(name='sparse0', type=xdl.features.sparse)\
.feature(name='sparse1', type=xdl.features.sparse)\
.feature(name='deep0', type=xdl.features.dense, nvec=256)
@@ -29,11 +29,11 @@ reader.startup()
def train():
batch = reader.read()
sess = xdl.TrainSession()
- emb1 = xdl.embedding('emb1', batch['sparse0'], xdl.TruncatedNormal(stddev=0.001), 8, 1024, vtype='hash')
- emb2 = xdl.embedding('emb2', batch['sparse1'], xdl.TruncatedNormal(stddev=0.001), 8, 1024, vtype='hash')
+ emb1 = xdl.embedding('emb1', batch['sparse0'], xdl.TruncatedNormal(stddev=0.001), 128, 1024, vtype='hash')
+ emb2 = xdl.embedding('emb2', batch['sparse1'], xdl.TruncatedNormal(stddev=0.001), 128, 1024, vtype='hash')
loss = model(batch['deep0'], [emb1, emb2], batch['label'])
train_op = xdl.SGD(0.5).optimize()
- log_hook = xdl.LoggerHook(loss, "loss:{0}", 10)
+ log_hook = xdl.LoggerHook(loss, "loss:{0}", 100)
sess = xdl.TrainSession(hooks=[log_hook])
while not sess.should_stop():
sess.run(train_op)
```
## Here is the output of `top` command
```bash
18142 root 20 0 45.101g 0.011t 61252 S 367.1 4.7 15:16.10 python deepctr.py --run=local
```
Seems the memory grow up about 100MB per epoch.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with xdl/examples/deepctr/deepctr.py and reproduce the report using the changed epochs, batch size, and embedding size, running it with `python deepctr.py --run=local`. Monitor memory across epochs and inspect the training loop and session setup. Done means the example no longer grows by about 100 MB per epoch under the reported configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.