vmware / vmware/pyvmomi

AttributeError: 'DataObject' object has no attribute '__traceback__'

Open
#1,116 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
2.3k
Forks
763
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

Fault objects seem to get wrapped in DataObjects, which don't have a __traceback__ attribute and causes some error handlers to crash.

This has been mentioned in https://github.com/vmware/pyvmomi/issues/1080#issuecomment-4185448819, and I believe it is also related to https://github.com/vmware/pyvmomi/issues/1086.

Reproduction steps
  1. Use a framework or write an error handler that explicitly checks __traceback__. In my case, I'm using Celery.
  2. Allow the fault Exception to raise.

This is similar to my actual usage:

def start_resize(vm, user, **options):
    check_permissions(user, vm)
    result = tasks.resize_vm.s(vm, **options).apply_async()
    result.get()

# tasks.py
@celeryapp.task
def resize_vm(vm, **options):
    vc = get_vsphere_client()
    try:
        vc.change_hardware(vm, **options)
    except Exception:
        logger.exception("...")
        raise

# class VsphereClient
def change_hardware(vm, **options):
    configspec = vim.vm.ConfigSpec(**options)
    if vm.runtime.powerState == "poweredOn":
        self.power_off_vm(vm)  # raises vim.fault.InvalidPowerState if the VM is already stopped
    task = vm.ReconfigVM_Task(spec=configspec)
    WaitForTask(task, si=self.si)
    self.power_on_vm(vm)  # raises vim.fault.InvalidPowerState if the VM is already running

When the InvalidPowerState exception is raised, the resize_vm task correctly logs the exception:

pyVmomi.VmomiSupport.vim.fault.InvalidPowerState: (vim.fault.InvalidPowerState) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'The attempted operation cannot be performed in the current state (Powered on).',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   requestedState = 'poweredOff',
   existingState = 'poweredOn'
}

The problem begins after the call to logger.exception, when the exception is re-raised:

< the InvalidPowerState traceback omitted for brevity >

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", line 605, in trace_task
    I, R, state, retval = on_error(task_request, exc)
                          ~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", line 421, in on_error
    R = I.handle_error_state(
        task, request, eager=eager, call_errbacks=call_errbacks,
    )
  File "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", line 199, in handle_error_state
    return {
           ~
        RETRY: self.handle_retry,
        ~~~~~~~~~~~~~~~~~~~~~~~~~
        FAILURE: self.handle_failure,
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    }[self.state](task, req,
    ~~~~~~~~~~~~~^^^^^^^^^^^
                  store_errors=store_errors,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
                  call_errbacks=call_errbacks)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", line 249, in handle_failure
    if exc.__traceback__ is None:
       ^^^^^^^^^^^^^^^^^
AttributeError: 'DataObject' object has no attribute '__traceback__'

InvalidPowerState happens to be the one that happened today, but I've run into this with other VimFault types. In #1080, it was a NotFound fault.

Expected behavior

Re-raising a vim.fault.VimFault should not pass a DataObject up to the next exception handler. Or, if that can't be avoided, a DataObject that wraps a VimFault should proxy the __traceback__ attribute to the wrapped fault object's __traceback__.

Additional context

While I am running into this with Celery, this could be expected to occur with any exception handler that accesses __traceback__ on exception objects.

I mentioned this same problem in #1080, which was primarily focused on static typing. However, this is a runtime erasure of exception information.

#1086 also points out that VimFault itself is not an exception type. I mentioned the same concern in #1080. It appears Exception doesn't get added to the MRO for faults until runtime, which is certainly a pain for type checkers, but, more importantly, it also seems that there are circumstances where the VimFault is wrapped in a DataObject--which is definitely not a BaseException even at runtime. That would make it impossible to catch, in addition to erasing the __traceback__.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the provided Celery path through resize_vm, change_hardware, and the InvalidPowerState fault, then inspect how VimFault instances are constructed and re-raised. Trace where the fault becomes a DataObject and compare the resulting object with the original exception; done means re-raised VimFaults retain exception behavior and traceback information for handlers such as Celery.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.