isar / isar/hive

how to get column name in udtf?

Open
#931 0 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Dart
Stars
4.4k
Forks
449
PR merge metrics
No merged PRs in 30d

Description

**Question**
Please explain the problem you are running into.

**Code sample**
```dart
Provide a few simple lines of code to show your problem.
```

table info:
col_name data_type comment
user_id bigint NULL
gender string NULL
region string NULL

table data:
user_id gender region
2 m CN
1 f ID

expected :
key attrname attrval
2 gender m
2 region CN
1 gender f
1 region ID

here is my udtf code:
public class Row2Column extends GenericUDTF {

private final static Map tableMap = new HashMap<>();
private static final Logger LOG = LogManager.getLogger(Row2Column.class);

@Override
public StructObjectInspector initialize(StructObjectInspector args) throws UDFArgumentException {
List inputFields = args.getAllStructFieldRefs();
for (int i = 0; i < inputFields.size(); ++i) {
tableMap.put(i + 1, inputFields.get(i).getFieldName());
}
System.out.println("initialize-------");
System.out.println(tableMap);
LOG.info("initialize-------");
LOG.info("---:" + tableMap);
if (tableMap.size() == 0) {
LOG.error("error" + tableMap);
throw new UDFArgumentException("explode() takes only one argument");
}
ObjectInspector[] udtfInputOIs = new ObjectInspector[inputFields.size()];
for (int i = 0; i < inputFields.size(); i++) {
udtfInputOIs[i] = inputFields.get(i).getFieldObjectInspector();
}
return initialize(udtfInputOIs);
}

@Override
public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
System.out.println("initialize++++++++");
System.out.println("++++++++" + tableMap);
LOG.info("initialize++++++++");
LOG.info("++++++++:" + tableMap);
List fieldNames = new ArrayList(3);
List fieldOIs = new ArrayList(3);
fieldNames.add("key");
fieldNames.add("AttrName");
fieldNames.add("AttrVal");
fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
}

@Override
public void process(Object[] record) throws HiveException {
ArrayList results = new ArrayList();
System.out.println("process-------");
System.out.println(tableMap);

for (int i = 1; i < record.length; ++i) {
System.out.println("record[i]"+ record[i]);
results.add(new Object[]{record[0], tableMap.get(i), record[i]});
}
Iterator it = results.iterator();
while (it.hasNext()) {
Object[] r = it.next();
forward(r);
}
}

@Override
public void close() throws HiveException {
// do nothing
}
}

i used spark sql to query and got:
spark-sql> select c2r(user_id,gender, region) from test.test_chenrun_c2r;
initialize++++++++
++++++++{}
key attrname attrval
2 NULL m
2 NULL CN
1 NULL f
1 NULL ID

and tableMap is empty.
It seems
public StructObjectInspector initialize(StructObjectInspector args)
this method is not called.

is there any way to get column name in
public StructObjectInspector initialize(ObjectInspector[] argOIs) ?

**Version**
- Platform: linux
- Hive version: [3.1.2]

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the two initialize overloads and process in the Row2Column UDTF, then reproduce the Spark SQL call shown in the issue. Verify which initialization entry point receives metadata and determine what "done" means for obtaining column names or documenting that they are unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.