cdisc-org / cdisc-org/DataExchange-DatasetJson
Clarification on null and empty string handling for decimal and string types
- Dominant language
- Jupyter Notebook
- Stars
- 41
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
While writing type-aware conversion and validation code for the `rows` element, I noticed a couple of points that I would like to clarify.
For decimal values, when the value is missing, examples such as `adlbh` use empty string.
In the specification, missing values are defined as follows:
> Missing values are represented by null. Empty strings are represented by "".
However, the handling of decimal values is not entirely clear.
In my opinion, null would be more appropriate for a missing decimal value, but is it intended that empty strings are also acceptable?
While checking the implementation examples related to this, I also noticed that empty string are generally used for missing string values.
This may reflect the SAS convention where an empty string represents a missing value for character variables. However, in many modern systems, null and empty string are clearly distinguished, and treating empty strings simply as missing values could potentially cause issues.
Possible approaches might include:
• Explicitly define that empty string and null are not distinguished and should be treated equivalently.
• Add a column attribute to specify which values should be treated as missing (e.g., missing="").
• Add a column attribute indicating that empty strings should be preserved separately from missing values (e.g., keep_empty=true).
If there is an intended way to handle this, it would be very helpful to know.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.