cameron314 / cameron314/concurrentqueue

Memory Leak

Open
#364 13 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
12.5k
Forks
1.9k
PR merge metrics
No merged PRs in 30d

Description

Hi
Sorry for the previous issue(mistakenly opened with a wrong user)

We are using concurrentqueue in our trading application for transfering data between threads, and we are very pleased about the performance of this queue.
However there is a significant memory leak problem in the library, We are transferring all market data from 1 producer to multiple consumers(approx 2-3 consumers) but creating 10,15 different queues and when we do that the memory usage increases 10mb in every 1 seconds, so at the end of the day we are consuming at least 200-300gb of memory.
When we remove the concurrentqueue and replace it with intel tbb concurrent queue library, memory leak dissapears.
Here is how we use the concurrentqueue: (We also have the same problem on spsc queue) . I'm also attaching the memory usage of our server when we continously use the concurrentqueue (In the graph at the time 16.00 and 17.00 we restarted the app)

#pragma once

#include <memory>
#include "IdGenerator.h"
#include "../Globals/Enumarations.h"
#include "../Containers/concurrentqueue.h"

template<class T = char*, class L = moodycamel::ConcurrentQueue<T>>

class Subscriber: private IdGenerator {
public:
	Subscriber() : id(++idGenerater) {
	}
	virtual ~Subscriber(){
	}

	void add(const T& message) {
		messageList.enqueue(message);
	}
	bool tryPop(T& item) {
		return messageList.try_dequeue(item);
	}
	L& getList() {
		return messageList;
	}
	unsigned short getId()const {
		return id;
	}
	void clear()
	{
	}
protected:
	const short id;
	L messageList;
};

template<class T, class L> using SubscriberPtr = std::shared_ptr<Subscriber<T, L>>;
memory

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the shown Subscriber wrapper and the queue implementation in concurrentqueue.h, including the mentioned SPSC queue, and reproduce the growth with the reported multiple-producer/consumer pattern under a memory profiler. The issue is done when the cause is identified, memory remains bounded after sustained use, and a focused regression test covers the reproducer.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.