apache / apache/accumulo

Failed to create a new table thru Accumulo Client library (org.apache.accumulo.core.client.AccumuloClient) in Kerberos-enabled Accumulo cluster

Open
#2,029 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
487
Avg merge
4d 5h
Merged PRs (30d)
13

Description

**Describe the bug**
Accumulo Client should be able to support the same API in both non-kerberos and kerberos enabled accumulo cluster. I was able to use Kerberos key tab and principal to create a table, inject data, drop the table thru Accumulo shell. When I do the same in Java Accumulo Client, my java program may list user, tables, and inject data to an existing table, but failed to create a new table with the following error in client side.

```
[main} DEBUG org.apache.accumulo.examples.helloworld.InsertWithKerberos - Failed to create table
org.apache.accumulo.core.client.AccumuloException: Internal error processing beginFateOperation
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:402)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:354)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1692)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(TableOperationsImpl.java:245)
at org.apache.accumulo.examples.helloworld.InsertWithKerberos.lambda$main$1(InsertWithKerberos.java:105)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
at org.apache.accumulo.examples.helloworld.InsertWithKerberos.main(InsertWithKerberos.java:98)
Caused by: org.apache.thrift.TApplicationException: Internal error processing beginFateOperation
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.accumulo.core.manager.thrift.FateService$Client.recv_beginFateOperation(FateService.java:91)
at org.apache.accumulo.core.manager.thrift.FateService$Client.beginFateOperation(FateService.java:77)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.beginFateOperation(TableOperationsImpl.java:258)
at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:364)
... 8 more

```
On the manager side, I see the matching error with NPE.

```
2021-04-23T00:26:04,450 [thrift.ProcessFunction] ERROR: Internal error processing beginFateOperation
java.lang.NullPointerException: null
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?]
at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.getTokenClassFromName(TCredentialsUpdatingInvocationHandler.java:158) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.updateArgs(TCredentialsUpdatingInvocationHandler.java:95) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:62) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at com.sun.proxy.$Proxy41.beginFateOperation(Unknown Source) ~[?:?]
at org.apache.accumulo.core.master.thrift.FateService$Processor$beginFateOperation.getResult(FateService.java:421) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.core.master.thrift.FateService$Processor$beginFateOperation.getResult(FateService.java:400) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.12.0.jar:0.12.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.12.0.jar:0.12.0]
at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:104) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) ~[libthrift-0.12.0.jar:0.12.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
```

**Versions (OS, Maven, Java, and others, as appropriate):**
- Affected version(s) of this project: [e.g. Accumulo 2.1.0-SNAPSHOT]
- OS: [e.g. CentOS 7.5]
- Others:

**To Reproduce**
Steps to reproduce the behavior (or a link to an example repository that reproduces the problem):
1. Start with 'https://github.com/apache/accumulo-examples/blob/main/src/main/java/org/apache/accumulo/examples/helloworld/Insert.java'.
2. Change code like the following to use kerberos principal and keytab file you have setup for kerberos authentication.
```
try {

final Configuration hadoopConf = new Configuration(false);
hadoopConf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, "kerberos");
UserGroupInformation.setConfiguration(hadoopConf);
UserGroupInformation.loginUserFromKeytab(principal, keytabPath);

final File keytabFile = new File(keytabPath);
final KerberosToken ktoken = new KerberosToken(principal, keytabFile);
AccumuloClient aclient = Accumulo.newClient().from(opts.getClientPropsPath())
.as(principal, ktoken).useSasl().build();
boolean authenticated = aclient.securityOperations().authenticateUser(principal, ktoken);
log.debug("AccumuloClient instance created with : " + aclient.whoami() + " for Kerberos, "
+ authenticated);

for (SystemPermission perm : SystemPermission.values()) {
log.debug("Expected user to have permission: " + perm,
aclient.securityOperations().hasSystemPermission(aclient.whoami(), perm));
}
aclient.tableOperations().create("hellotable");
aclient.close();
} catch (Exception e) {
log.debug("Exception caught in Main....");
log.debug(e.getMessage());
}
```
3. Change accumulo-client.properties to have kerberos authentication case. Test environment and configuration with 'accumulo shell' to verify. After verification, drop the test table.

4. Compile the changed code and run the same while you tail the manager log on your head node. You will see the similar failure exceptions I shared above with NPE.

**Expected behavior**
Accumulo client should be able to create a table in Accumulo.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Additional context**
I reviewed the Accumulo Shell code and it is also using the same client library, but not sure what additional thing(s) Shell class is doing in order to make the creation work.

Contributor guide

Open the contributing guide

Research direction

Start with the reproduced call in accumulo-examples/.../Insert.java and trace table creation through TableOperationsImpl.beginFateOperation and the manager-side TCredentialsUpdatingInvocationHandler. Run the Kerberos setup and client example while tailing the manager log; done means the Java AccumuloClient can create a table successfully in a Kerberos-enabled cluster without the reported NPE.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
authentication, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.