microsoft / microsoft/azure-quantum-python

Job operations are not parallelizable

Open
#300 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
160
Forks
113
Avg merge
6d 23h
Merged PRs (30d)
1

Description

I'm trying to submit and process results of multiple jobs in parallel but I'm noticing that, in many cases, jobs are submitted sequentially.

import datetime
import imp
from multiprocessing import Pool
from azure.quantum import Workspace
from azure.quantum.optimization import Problem, ProblemType, Term
from azure.quantum.target.toshiba import SimulatedBifurcationMachine
import logging


def create_problem():
    problem = Problem('Test_Problem', problem_type=ProblemType.pubo)
    problem.add_terms([
        Term(c=1, indices=[]),
        Term(c=2, indices=[1, 2]),
        Term(c=-3, indices=[1, 2]),
        Term(c=1, indices=[0, 2])
    ])
    return problem


def exec_job(no):
    workspace = Workspace (
        resource_id = "<workspace-id>",
        location= "<location>"
    )

    problem = create_problem()
    solver = SimulatedBifurcationMachine(workspace, loops=0, timeout=10)
    print(str(datetime.datetime.utcnow()), "\t[%d] solver.submit() start" % no)
    job = solver.submit(problem)
    print(str(datetime.datetime.utcnow()), "\t[%d] job.id=%s" % (no, str(job.id)))

    print(str(datetime.datetime.utcnow()), "\t[%d] job.get_results() start" % no)
    result = job.get_results()
    print(str(datetime.datetime.utcnow()), "\t[%d] job.details=%s" % (no, str(job.details)))


if __name__ == '__main__':
    logging.basicConfig(level=logging.DEBUG)

    njobs = 2
    with Pool(processes=njobs) as pool:
        pool.map(func=exec_job, iterable=range(njobs))

For the above, output is:
2022-04-21 18:40:23.466530 [0] solver.submit() start
2022-04-21 18:40:23.542727 [1] solver.submit() start
2022-04-21 18:40:28.752803 [1] job.id=810bc3bb-c1a2-11ec-baa1-b831b575aea6
2022-04-21 18:40:28.752803 [1] job.get_results() start
..........2022-04-21 18:40:52.826348 [1] job.details={..........}
2022-04-21 18:44:19.108243 [0] job.id=81018a8e-c1a2-11ec-adf4-b831b575aea6
2022-04-21 18:44:19.109211 [0] job.get_results() start
..........2022-04-21 18:44:42.799796 [0] job.details={..............}

You can see that job 1 was submitted and awaited while job 0 hasn't been submitted at the same time and was done 4 mins after the first job.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reported multiprocessing.Pool example with exec_job, then trace the solver.submit() and job.get_results() entry points used by SimulatedBifurcationMachine. Compare the timestamps for both workers and identify where submission or result retrieval becomes serialized. Done means independent jobs can be submitted and processed concurrently without the observed multi-minute delay.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.