NVIDIA / NVIDIA/cudf

[FEA]: Add trace formatter option to cudf-polars benchmarks

Open
#22,134 0 comments 0 reactions 0 assignees View on GitHub
cudf-polars feature request
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**

cudf-polars tracing can be configured through python's stdlib / structlog's formatting mechanism. In general, how logs are formatted / emitted are up to the application.

But in cudf-polars' benchmarks we include some code to collect traces in memory, to support the `--collect-traces` flag: https://github.com/rapidsai/cudf/blob/3eff41430092e18730b9da8940fbf156c68652a2/python/cudf_polars/cudf_polars/experimental/benchmarks/utils_legacy.py#L1862-L1884

To help debug errors that occur at runtime, it'd be helpful to stream trace logs to a file instead of buffering them in memory.

**Describe the solution you'd like**

A new CLI option to specify how to format logs

```python
stream = io.StringIO()
memory_handler = logging.StreamHandler(stream)
ndjson_handler = logging.FileHandler(
f"logs-{os.getpid()}.ndjson", mode="a", encoding="utf-8"
)
proc_fmt = structlog.stdlib.ProcessorFormatter
memory_handler.setFormatter(
proc_fmt(processor=json_renderer, foreign_pre_chain=shared_processors)
)
ndjson_handler.setFormatter(
proc_fmt(processor=json_renderer, foreign_pre_chain=shared_processors)
)

logging.basicConfig(
level=logging.INFO, handlers=[memory_handler, ndjson_handler]
)
```

Either `--streaming-logs` or perhaps expand `--collect-traces=memory/file/None`.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.