Azure / Azure/azure-sdk-for-python

OSError: Invalid argument when writing to files like meta.json or pf_version_check.json during local Azure AI Evaluation on Windows

Open
#41,913 4 comments 0 reactions 0 assignees View on GitHub
AI Client customer-reported needs-team-attention question Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

# Bug Report: Azure AI Evaluation OSError on Windows

- **Package Name**: azure-ai-evaluation
- **Package Version**: 1.9.0
- **Operating System**: Windows
- **Python Version**: 3.13.3

## Describe the bug

When running assistant evaluation locally with the Azure AI Evaluation framework on Windows, the process fails with an `OSError: Invalid argument` when attempting to write to files such as `meta.json` or `.promptflow/pf_version_check.json`. This error appears to be related to how Promptflow handles file paths on Windows.

It does not happen consistently. Sometimes, deleting the whole `/.promptflow/` folder helps.

Error stack trace:

```text
OSError: [Errno 22] Invalid argument: '\\.promptflow\\.runs\\azure_ai_evaluation_evaluators_assistant_target_\\meta.json'
```

_This also occurs for `.promptflow/pf_version_check.json`._

Full stack trace excerpt:

```text
Traceback (most recent call last):
...
File "/.venv/Lib/site-packages/promptflow/_sdk/operations/_local_storage_operations.py", line 248, in _dump_meta_file
json_dump({"batch_size": LOCAL_STORAGE_BATCH_SIZE}, self._meta_path)
File "/.venv/Lib/site-packages/promptflow/_sdk/_utilities/general_utils.py", line 872, in json_dump
with write_open(file) as f:
OSError: [Errno 22] Invalid argument: '\\.promptflow\\.runs\\azure_ai_evaluation_evaluators_assistant_target_\\meta.json'
...
```

## To Reproduce

Steps to reproduce the behavior:

1. Set up an evaluation script using the Azure AI Evaluation framework with a local target. Example:
```python
from azure.ai.evaluation import evaluate

result = evaluate(
data=dataset_path,
target=assistant_target,
evaluators=evaluators,
evaluator_config={
"default": {
"column_mapping": {
"query": "${data.query}",
"response": "${target.response}",
"ground_truth": "${data.ground_truth}",
"context": "${target.context}",
}
}
},
output_path=output_path,
azure_ai_project=azure_ai_project,
)
```

3. Run the evaluation on Windows.
4. Occassionally observe the OSError with "Invalid argument" when trying to write to `meta.json` or `.promptflow/pf_version_check.json` files (or sometimes `.promptflow/pf.yaml`)

## Expected behavior

The evaluation script should run successfully without file system errors.

## Additional context

- The error does not occur consistently; sometimes, it helps to first delete the `\.promptflow` folder when running the evaluation script.
- The files in question appear to exist, but the error suggests a problem writing to them.
- This may be related to how Promptflow handles file paths on Windows.

Attempted solutions:
- Changing the `PF_HOME_DIR` to be within another folder
- Setting `PF_WORKER_COUNT` to 1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.