Add `add_running_callback()` and `remove_running_callback()` methods to `concurrent.futures.Future` and `asyncio.Future`

Feature or enhancement

Add add_running_callback() and remove_running_callback() methods to concurrent.futures.Future and asyncio.Future

Pitch

The concurrent.futures.Future and asyncio.Future module in Python provide a convenient way to run parallel and asynchronous tasks. However, they currently lacks the ability to add and remove callbacks that are executed when a Future is running. This proposal aims to add two new methods to the Future class, add_running_callback() and remove_running_callback(), which will allow developers to add and remove callbacks that are executed when a Future is running. This will provide more flexibility and control over the execution of parallel and asynchronous tasks.

Example usage:

Here is an example of how the new methods will be used:

import concurrent.futures
import time

def some_long_running_task():
    print('Running a long running task')
    time.sleep(10)
    return 42

def running_callback_1(future):
    print(f'{future} is running')

def running_callback_2(future):
    print(f'{future} is still running')

with concurrent.futures.ThreadPoolExecutor() as executor:
    future = executor.submit(some_long_running_task)
    future.add_running_callback(running_callback_1)
    future.add_running_callback(running_callback_2)
    # do something else here like ping a server
    future.remove_running_callback(running_callback_1)
    # do something else here like fetch some data
    future.remove_running_callback(running_callback_1)
    # do something else here 

In the above example, running_callback_1 and running_callback_2 are added and then removed from the list of running callbacks for the Future. Here is how to achieve the same but in blocking manner:

import concurrent.futures
import time

def some_long_running_task():
    print('Running a long running task')
    time.sleep(10)
    return 42

def running_callback(future):
    print(f'{future} is running')
    print(f'ping or fetch something while {future} is running')
    print(f'check something else while {future} is running')

def done_callback(future):
    print(f'{future} is done and returned {future.result()}')

def while_running(future):
    while not future.done():
        print('Running a long running task')
        time.sleep(1)

def main():
    with concurrent.futures.ThreadPoolExecutor() as executor:
        future = executor.submit(some_long_running_task)
        future.add_done_callback(done_callback)
        while_running(future)

if __name__ == '__main__':
    main()

Previous discussion

This feature has not been previously discussed.

Design and Implementation

The proposed add_running_callback() and remove_running_callback() methods will be added to the Future class in both the concurrent.futures.Future and asyncio.Future. The add_running_callback() method will take a single argument, a callable object, and will add it to a list of callbacks to be executed when the Future is running. The remove_running_callback() method will also take a single argument, a callable object, and will remove it from the list of callbacks.

The add_running_callback() and remove_running_callback() methods will be implemented by adding a new attribute, _running_callbacks, to the Future class. This attribute will be a list of callbacks that are executed when the Future is running. The add_running_callback() method will simply append the passed callback to the list, and the remove_running_callback() method will remove it.

In addition, the Future.init method will be updated to initialize the _running_callbacks attribute as an empty list. And the Future._invoke_callbacks method will be updated to invoke the callbacks in the _running_callbacks list, in addition to the done callbacks, when the Future's state changes.

Backwards compatibility

The proposed changes to the Future class will not affect any existing code that uses both the concurrent.futures.Future and asyncio.Future. The new methods, add_running_callback() and remove_running_callback(), are optional and will not be used by any code that does not specifically call them.

Conclusion

Adding the add_running_callback() and remove_running_callback() methods to the Future class in both the concurrent.futures.Future and asyncio.Future will provide greater flexibility and control over the execution of parallel and asynchronous tasks. This proposal is backwards-compatible and provides a simple and straightforward solution to a common problem faced by developers using both the concurrent.futures.Future and asyncio.Future.

P.S. This would be my first contribution and I have already had quick look at Python Developer’s Guide and PEP Purpose and Guidelines, and I believe this kind of proposals falls under enhancements effort but if I missed something or this is not the right place for this proposal, please kindly let me know.

Start by reading the concurrent.futures.Future and asyncio.Future implementations and their existing callback behavior. Define the running-callback semantics, including when callbacks are invoked and how removal behaves, then update both Future classes and verify the new methods and callback invocation paths.