oracle / oracle/oci-cli

`oci os object get` OOMs when the file it is getting is too big

Open
#883 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
669
Forks
236
Avg merge
1m
Merged PRs (30d)
4

Description

While testing our cloudinit scripts with some smaller instances with 2GB of memory I found that my call to oci os object get ends up OOMing. Here's where this happens in the script output:

+ oci os object get -bn <redacted> --name releases/<redacted>.gz --file <redacted>.gz                                                                                   
Downloading object                                                                                                                                                                                                    
/var/lib/cloud/instance/scripts/part-001: line 111: 38203 Killed                  oci os object get -bn <redacted> --name releases/<redacted>.gz --file <redacted>.gz

And the corresponding line in dmesg:

[  354.613716] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/cloud-final.service,task=oci,pid=38203,uid=0                                           
[  354.617651] Out of memory: Killed process 38203 (oci) total-vm:6534412kB, anon-rss:1497088kB, file-rss:5336kB, shmem-rss:0kB, UID:0 pgtables:6484kB oom_score_adj:0                                                

The target file here is 2.9GB. You can see the OOM happens when the oci process is consuming about 1.5GB.

Bumping RAM up to 8GB on these instances results in no OOM.

I see in the source some references to streaming interfaces, but maybe we're not using those, they're bugged, or we have a memory leak somewhere else.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the oci os object get command and inspect the source's streaming interfaces. Reproduce the download with a large object on a low-memory instance, then trace whether the response is buffered or streamed. Done means large downloads complete without memory usage growing to the object size.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.