deeppavlov / deeppavlov/AutoIntent

Проблема с удалением артефактов (?) в _optimization_info.py

Open
#262 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
54
Forks
16
No merged PRs in 30d

Description

Если указать в пресете:

  - node_type: decision
    target_metric: decision_f1

Получаю ошибку Failed to save optimization info

Трейс ошибки 
[I 2025-08-28 09:45:03,556] A new study created in RDB with name: NodeType.scoring
Failed to save optimization info
Traceback (most recent call last):
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\context\optimization_info\_optimization_info.py", line 245, in dump
  shutil.rmtree(target_path)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
  return _rmtree_unsafe(path, onerror)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 626, in _rmtree_unsafe
  onerror(os.rmdir, path, sys.exc_info())
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 624, in _rmtree_unsafe
  os.rmdir(path)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'data\\runs\\breakable_fly_08-28-2025_09-45-03\\optimization_info\\pydantic'
Failed to save optimization info
Traceback (most recent call last):
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\context\optimization_info\_optimization_info.py", line 245, in dump
  shutil.rmtree(target_path)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
  return _rmtree_unsafe(path, onerror)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 622, in _rmtree_unsafe
  onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 620, in _rmtree_unsafe
  os.unlink(fullname)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'data\\runs\\breakable_fly_08-28-2025_09-45-03\\optimization_info\\pydantic\\trials\\model_dump.json'
Failed to save optimization info
Traceback (most recent call last):
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\context\optimization_info\_optimization_info.py", line 245, in dump
  shutil.rmtree(target_path)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
  return _rmtree_unsafe(path, onerror)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 626, in _rmtree_unsafe
  onerror(os.rmdir, path, sys.exc_info())
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 624, in _rmtree_unsafe
  os.rmdir(path)
OSError: [WinError 145] The directory is not empty: 'data\\runs\\breakable_fly_08-28-2025_09-45-03\\optimization_info\\pydantic'
Failed to save optimization info
Traceback (most recent call last):
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\context\optimization_info\_optimization_info.py", line 245, in dump
  shutil.rmtree(target_path)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
  return _rmtree_unsafe(path, onerror)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 622, in _rmtree_unsafe
  onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 620, in _rmtree_unsafe
  os.unlink(fullname)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'data\\runs\\breakable_fly_08-28-2025_09-45-03\\optimization_info\\pydantic\\artifacts\\class_info.json'
[W 2025-08-28 09:46:16,305] Trial 0 failed with parameters: {'module_idx': 0} because of the following error: FileNotFoundError(2, 'The system cannot find the path specified').
Traceback (most recent call last):
File "D:\intent-classifier\.venv\Lib\site-packages\optuna\study\_optimize.py", line 201, in _run_trial
  value_or_values = func(trial)
                    ^^^^^^^^^^^
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\nodes\_node_optimizer.py", line 163, in objective
  context.dump_optimization_info()
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\context\_context.py", line 88, in dump_optimization_info
  self.optimization_info.dump(self.logging_config.dirpath)
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\context\optimization_info\_optimization_info.py", line 245, in dump
  shutil.rmtree(target_path)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
  return _rmtree_unsafe(path, onerror)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 626, in _rmtree_unsafe
  onerror(os.rmdir, path, sys.exc_info())
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 624, in _rmtree_unsafe
  os.rmdir(path)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'data\\runs\\breakable_fly_08-28-2025_09-45-03\\optimization_info\\pydantic'
[W 2025-08-28 09:46:16,314] Trial 10 failed with parameters: {'module_idx': 0} because of the following error: FileNotFoundError(2, 'The system cannot find the file specified').
Traceback (most recent call last):
File "D:\intent-classifier\.venv\Lib\site-packages\optuna\study\_optimize.py", line 201, in _run_trial
  value_or_values = func(trial)
                    ^^^^^^^^^^^
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\nodes\_node_optimizer.py", line 163, in objective
  context.dump_optimization_info()
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\context\_context.py", line 88, in dump_optimization_info
  self.optimization_info.dump(self.logging_config.dirpath)
File "D:\intent-classifier\.venv\Lib\site-packages\autointent\context\optimization_info\_optimization_info.py", line 245, in dump
  shutil.rmtree(target_path)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
  return _rmtree_unsafe(path, onerror)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
  _rmtree_unsafe(fullname, onerror)
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 622, in _rmtree_unsafe
  onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\Astrov\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 620, in _rmtree_unsafe
  os.unlink(fullname)
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'data\\runs\\breakable_fly_08-28-2025_09-45-03\\optimization_info\\pydantic\\trials\\model_dump.json'

Временный фикс - подавил ошибки FileNotFoundError, PermissionError здесь. В причинах/последствиях не разбирался

        with tempfile.TemporaryDirectory(dir=path) as temp_dir:
            temp_path = Path(temp_dir)

            # Try to save to the temporary directory first
            try:
                Dumper.dump(self, temp_path, exists_ok=False, exclude=exclude)

                # If successful, move the contents to the final destination
                if target_path.exists():
                    for sub in target_path.iterdir():
                        if sub.is_dir():
                            try:
                                shutil.rmtree(sub)
                            except (FileNotFoundError, PermissionError):
                                logger.warning(f"Could not remove directory {sub}")
                                pass
                        else:
                            try:
                                sub.unlink()
                            except (FileNotFoundError, PermissionError):
                                logger.warning(f"Could not remove file {sub}")
                                pass
                else:
                    target_path.mkdir(parents=True, exist_ok=True)
                shutil.move(temp_path, target_path)

            except Exception:
                logger.exception("Failed to save optimization info")
                raise

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/autointent/context/optimization_info/_optimization_info.py around dump and the referenced cleanup block, then inspect src/autointent/nodes/_node_optimizer.py where optimization info is saved. Reproduce the decision_f1 preset on Windows and determine why cleanup encounters missing or locked paths. Done means the optimization run saves its information without the reported FileNotFoundError or PermissionError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.