[Bug]: File Metadata does not scan for decimals properly
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### Apache Hop version?
2.12 +
### Java version?
17.0.16
### Operating system
Linux
### What happened?
I have a file that has currency values - (example: 12423.25) File Metadata is reading it as a Number with length 15 and precision = 1 and mask #.# - which subsequently drops off any number past the first decimal.
Also - if there is an embedded quote (when all fields are enclosed in quotes), the transform may fail if it is a large dataset. It would be nice to handle that situation - **which this also breaks the enclosure from being identified even if the transform does not throw an error**
File attached
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start with the File Metadata transform using the attached file in the stated Hop 2.12 and Java 17 environment. Reproduce decimal inference and embedded-quote behavior on a large dataset; done means decimal length, precision, and mask retain values such as 12423.25, while quoted fields remain correctly enclosed without transform failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100