apache / apache/hop

[Bug]: File Metadata does not scan for decimals properly

Open
#5,609 8 comments 0 reactions 0 assignees View on GitHub
awaiting triage P3 Transforms
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### Apache Hop version?

2.12 +

### Java version?

17.0.16

### Operating system

Linux

### What happened?

I have a file that has currency values - (example: 12423.25) File Metadata is reading it as a Number with length 15 and precision = 1 and mask #.# - which subsequently drops off any number past the first decimal.

Also - if there is an embedded quote (when all fields are enclosed in quotes), the transform may fail if it is a large dataset. It would be nice to handle that situation - **which this also breaks the enclosure from being identified even if the transform does not throw an error**

File attached

### Issue Priority

Priority: 3

### Issue Component

Component: Transforms

Contributor guide

Open the contributing guide

Research direction

Start with the File Metadata transform using the attached file in the stated Hop 2.12 and Java 17 environment. Reproduce decimal inference and embedded-quote behavior on a large dataset; done means decimal length, precision, and mask retain values such as 12423.25, while quoted fields remain correctly enclosed without transform failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.