batting_stats() and pitching_stats() errors out for certain players/years
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 428
- PR merge metrics
- No merged PRs in 30d
Description
running pitching_stats with Kent Tekulve's playerid results in this error:
```cmd
>>> pitching_stats(start_season=1979, end_season=1980, players=1012905)
Traceback (most recent call last):
File "/home/yzhang/.envs/deadball_env/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 934, in _finalize_columns_and_data
columns alidate_or_indexify_columns(contents, columns)
File "/home/yzhang/.envs/deadball_env/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 981, in _validate_or_indexify_columns
raise AssertionError(
AssertionError: 334 col= _validate_or_indexify_columns(contents, columns)
File "/home/yzhang/.envs/deadball_env/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 981, in _validate_or_indexify_columns
raise AssertionError(
AssertionError: 334 columns passed, passed data had 1 columns
```
---
same type of error also seems to happen when calling batting_stats for 1970 Roberto Clemente as well. 1969 and 1971 both show up normally:
```python
>>> batting_stats(1969, 1971, players=1002340)
IDfg Season Name Team Age G AB ... Events CStr% CSW% xBA xSLG xwOBA L-WAR
0 1002340 1969 Roberto Clemente PIT 34 138 507 ... 0 NaN NaN NaN NaN NaN 7.0
1 1002340 1971 Roberto Clemente PIT 36 132 522 ... 0 NaN NaN NaN NaN NaN 6.5
[2 rows x 320 columns]
```
whereas 1970 Clemente search returns a similar error (attaching full error trace)
```cmd
>>> batting_stats(1970, players=1002340)
Traceback (most recent call last):
File "", line 1, in
File "/home/yzhang/Downloads/pybaseball/pybaseball/cache/cache.py", line 58, in _cached
result = func(*args, **kwargs)
File "/home/yzhang/Downloads/pybaseball/pybaseball/datasources/fangraphs.py", line 176, in fetch
return super().fetch(*args, **kwargs)
File "/home/yzhang/Downloads/pybaseball/pybaseball/datasources/fangraphs.py", line 154, in fetch
self.html_accessor.get_tabular_data_from_options(
File "/home/yzhang/Downloads/pybaseball/pybaseball/datasources/html_table_processor.py", line 90, in get_tabular_data_from_options
return self.get_tabular_data_from_url(
File "/home/yzhang/Downloads/pybaseball/pybaseball/datasources/html_table_processor.py", line 78, in get_tabular_data_from_url
return self.get_tabular_data_from_html(
File "/home/yzhang/Downloads/pybaseball/pybaseball/datasources/html_table_processor.py", line 59, in get_tabular_data_from_html
return self.get_tabular_data_from_element(
File "/home/yzhang/Downloads/pybaseball/pybaseball/datasources/html_table_processor.py", line 50, in get_tabular_data_from_element
fg_data = pd.DataFrame(data_rows, columns=headings)
File "/home/yzhang/.envs/deadball_env/lib/python3.9/site-packages/pandas/core/frame.py", line 782, in __init__
arrays, columns, index = nested_data_to_arrays(
File "/home/yzhang/.envs/deadball_env/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 498, in nested_data_to_arrays
arrays, columns = to_arrays(data, columns, dtype=dtype)
File "/home/yzhang/.envs/deadball_env/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 840, in to_arrays
content, columns = _finalize_columns_and_data(arr, columns, dtype)
File "/home/yzhang/.envs/deadball_env/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 937, in _finalize_columns_and_data
raise ValueError(err) from err
ValueError: 320 columns passed, passed data had 1 columns
```
looks like an element might not be parsed correctly and so is just returning [[None]] when creating the DataFrame in get_tabular_data_from_element in HTMLTableProcessor. not sure if fangraphs has different DOM elements for different players/years that differs from the hardcoded rows/cells xpath?
Contributor guide
Research direction
Reproduce the Kent Tekulve and Roberto Clemente calls, then read pybaseball/datasources/fangraphs.py and pybaseball/datasources/html_table_processor.py, especially get_tabular_data_from_element where the DataFrame is built. Compare the parsed headings and rows for failing and working player/year requests. Done means both batting_stats() and pitching_stats() return usable data without a pandas column-shape error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100