how to get column name in udtf?
- Dominant language
- Dart
- Stars
- 4.4k
- Forks
- 449
- PR merge metrics
- No merged PRs in 30d
Description
**Question**
Please explain the problem you are running into.
**Code sample**
```dart
Provide a few simple lines of code to show your problem.
```
table info:
col_name data_type comment
user_id bigint NULL
gender string NULL
region string NULL
table data:
user_id gender region
2 m CN
1 f ID
expected :
key attrname attrval
2 gender m
2 region CN
1 gender f
1 region ID
here is my udtf code:
public class Row2Column extends GenericUDTF {
private final static Map tableMap = new HashMap<>();
private static final Logger LOG = LogManager.getLogger(Row2Column.class);
@Override
public StructObjectInspector initialize(StructObjectInspector args) throws UDFArgumentException {
List inputFields = args.getAllStructFieldRefs();
for (int i = 0; i < inputFields.size(); ++i) {
tableMap.put(i + 1, inputFields.get(i).getFieldName());
}
System.out.println("initialize-------");
System.out.println(tableMap);
LOG.info("initialize-------");
LOG.info("---:" + tableMap);
if (tableMap.size() == 0) {
LOG.error("error" + tableMap);
throw new UDFArgumentException("explode() takes only one argument");
}
ObjectInspector[] udtfInputOIs = new ObjectInspector[inputFields.size()];
for (int i = 0; i < inputFields.size(); i++) {
udtfInputOIs[i] = inputFields.get(i).getFieldObjectInspector();
}
return initialize(udtfInputOIs);
}
@Override
public StructObjectInspector initialize(ObjectInspector[] argOIs) throws UDFArgumentException {
System.out.println("initialize++++++++");
System.out.println("++++++++" + tableMap);
LOG.info("initialize++++++++");
LOG.info("++++++++:" + tableMap);
List fieldNames = new ArrayList(3);
List fieldOIs = new ArrayList(3);
fieldNames.add("key");
fieldNames.add("AttrName");
fieldNames.add("AttrVal");
fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
}
@Override
public void process(Object[] record) throws HiveException {
ArrayList results = new ArrayList();
System.out.println("process-------");
System.out.println(tableMap);
for (int i = 1; i < record.length; ++i) {
System.out.println("record[i]"+ record[i]);
results.add(new Object[]{record[0], tableMap.get(i), record[i]});
}
Iterator it = results.iterator();
while (it.hasNext()) {
Object[] r = it.next();
forward(r);
}
}
@Override
public void close() throws HiveException {
// do nothing
}
}
i used spark sql to query and got:
spark-sql> select c2r(user_id,gender, region) from test.test_chenrun_c2r;
initialize++++++++
++++++++{}
key attrname attrval
2 NULL m
2 NULL CN
1 NULL f
1 NULL ID
and tableMap is empty.
It seems
public StructObjectInspector initialize(StructObjectInspector args)
this method is not called.
is there any way to get column name in
public StructObjectInspector initialize(ObjectInspector[] argOIs) ?
**Version**
- Platform: linux
- Hive version: [3.1.2]
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the two initialize overloads and process in the Row2Column UDTF, then reproduce the Spark SQL call shown in the issue. Verify which initialization entry point receives metadata and determine what "done" means for obtaining column names or documenting that they are unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100