golang-queue / golang-queue/queue

Number of workers can be exceeded + Unit Test

Open
#154 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
628
Forks
49
PR merge metrics
No merged PRs in 30d

Description

Library function 'BusyWorkers' can sometimes return more workers than configured.
I am also suspecting that also +1 task can be taken.

Please check the unit test bellow:
11 wokers instead of expected 10.

func TestShouldShowCorrectNumberOfBusyWorkers(t *testing.T) {

	maxTasks := 100
	maxBusyWorkers := int64(10)
	waitAllTasksExecuted := &sync.WaitGroup{}
	waitAllTasksExecuted.Add(maxTasks)

	runningTasksCount := &atomic.Int32{}
	runningTasksCount.Store(0)

	taskGenerator := func(_ int) job.TaskFunc {
		return func(ctx context.Context) error {
			runningTasksCount.Add(1)
			defer runningTasksCount.Add(-1)
			defer waitAllTasksExecuted.Done()
			time.Sleep(100 * time.Millisecond)
			return nil
		}
	}

	tasks := []job.TaskFunc{}
	for i := 0; i < maxTasks; i++ {
		tasks = append(tasks, taskGenerator(i))
	}
	pool := *queue.NewPool(maxBusyWorkers)
	pool.Start()
	defer pool.Shutdown()

	ia := &atomic.Int32{}
	ia.Store(0)

	waitTasksAdded := &sync.WaitGroup{}
	waitTasksAdded.Add(maxTasks)
	// Send a tasks every n ms
	ticker := time.NewTicker(10 * time.Millisecond)
	go func() {

		for range ticker.C {
			i := ia.Load()
			if i >= int32(maxTasks) {
				ticker.Stop()
				return
			}

			go func() {
				defer waitTasksAdded.Done()
				err := pool.QueueTask(job.TaskFunc(tasks[i]))
				if err != nil {
					panic(err)
				}
			}()

			ia.Add(1)
		}
	}()

	// Monitor busy workers count and running tasks
	tickerMonitor := time.NewTicker(10 * time.Millisecond)
	go func() {

		for range tickerMonitor.C {
// -> This is actual test failure
			require.LessOrEqual(t, int32(pool.BusyWorkers()), int32(maxBusyWorkers), "busy workers should be less or equal to max busy workers")
			require.LessOrEqual(t, int32(runningTasksCount.Load()), int32(maxBusyWorkers), "busy workers should be less or equal to max busy workers")
		}
	}()

	// Wait all tasks to be added
	waitTasksAdded.Wait()

	require.Equal(t, uint64(maxTasks), pool.SubmittedTasks(), "all tasks are submitted")

	waitAllTasksExecuted.Wait()
	time.Sleep(1 * time.Second)
	tickerMonitor.Stop()

	require.Equal(t, uint64(maxTasks), pool.SubmittedTasks(), "all tasks are submitted")
	require.Equal(t, uint64(maxTasks), pool.SuccessTasks(), "all tasks are successful")
	require.Equal(t, int64(0), pool.BusyWorkers(), "no busy workers, all tasks are finished")

}


github.com/golang-queue/queue v0.4.0
github.com/stretchr/testify v1.11.1
@appleboy

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with queue.NewPool, BusyWorkers, and the supplied TestShouldShowCorrectNumberOfBusyWorkers reproduction. Run the test while monitoring the reported busy-worker and running-task counts; done means the pool never exceeds the configured limit of 10 and all 100 tasks still complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.