Very large chunk sizes causes net.schmizz.sshj.connection.ConnectionException: Timeout when trying to expand the window size
- Dominant language
- Java
- Stars
- 2.7k
- Forks
- 620
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 11
Description
This issue is also posted on [stackoverflow chunk question](https://stackoverflow.com/questions/69023031/why-is-the-sshj-maximum-write-size-32kb)
Hello everyone here is the code snippet where when I am writing a very large byte[] into a file at a specific start position. I get this exception ```net.schmizz.sshj.connection.ConnectionException: Timeout when trying to expand the window size```
The sshj version I am using is 0.31.0.
Here is the function that is throwing the exception, specifically on the line where .write() is called.
```
@Override
public void write(List items) throws Exception {
for(DataChunk chunk : items){
Path path = Paths.get(chunk.getFileName());
RemoteFile remoteFile = getRemoteFile(path.toString());
if(remoteFile == null) throw new FileNotFoundException();
remoteFile.write(chunk.getStartPosition(), chunk.getData(), 0, chunk.getData().length);
}
}
```
Here every DataChunk object contains a byte[] with a size of 11MB. I understand this is a large chunk to write over SFTP but still I think it should be possible to do no? I have tried this type of upload using Jsch and it worked just fine which makes me wonder why it fails for sshj.
Here is how I create the client currently
```
public SSHClient createConnectedAndAuthenticatedSSHClient(AccountEndpointCredential credential, long size) throws IOException {
String[] destCredUri = accountCredentialURIFormated(credential);
SSHClient sshClient = new SSHClient();
sshClient.addHostKeyVerifier(new PromiscuousVerifier());
sshClient.connect(destCredUri[0], Integer.parseInt(destCredUri[1]));
try {
sshClient.authPassword(credential.getUsername(), credential.getSecret());
if(sshClient.isAuthenticated() && sshClient.isConnected()) logger.info("Connected & authenticated client");
return sshClient;
} catch (UserAuthException | TransportException e) {
logger.debug("sshj failed to do basic auth with the account endpoint credential");
}
try {
logger.info("Doing pem key auth sftp -- connected the remote ssh server");
KeyPair kp = this.createKeyPair(credential.getSecret());
KeyProvider provider = sshClient.loadKeys(kp);
sshClient.authPublickey(credential.getUsername(), provider);
if(sshClient.isAuthenticated() && sshClient.isConnected()) logger.info("Connected & authenticated client");
return sshClient;
} catch (InvalidKeySpecException | NoSuchAlgorithmException e) {
logger.error("sshj failed to do pem auth with the account endpoint credential");
e.printStackTrace();
}
return null;
}
```
Here is the entire exception that I get,
```
net.schmizz.sshj.connection.ConnectionException: Timeout when trying to expand the window size
at net.schmizz.sshj.connection.channel.Window$Remote.awaitExpansion(Window.java:93) ~[sshj-0.31.0.jar!/:0.31.0]
at net.schmizz.sshj.connection.channel.ChannelOutputStream$DataBuffer.flush(ChannelOutputStream.java:79) ~[sshj-0.31.0.jar!/:0.31.0]
at net.schmizz.sshj.connection.channel.ChannelOutputStream$DataBuffer.write(ChannelOutputStream.java:60) ~[sshj-0.31.0.jar!/:0.31.0]
at net.schmizz.sshj.connection.channel.ChannelOutputStream.write(ChannelOutputStream.java:141) ~[sshj-0.31.0.jar!/:0.31.0]
at net.schmizz.sshj.sftp.SFTPEngine.transmit(SFTPEngine.java:304) ~[sshj-0.31.0.jar!/:0.31.0]
at net.schmizz.sshj.sftp.SFTPEngine.request(SFTPEngine.java:138) ~[sshj-0.31.0.jar!/:0.31.0]
at net.schmizz.sshj.sftp.RemoteFile.asyncWrite(RemoteFile.java:83) ~[sshj-0.31.0.jar!/:0.31.0]
at net.schmizz.sshj.sftp.RemoteFile.write(RemoteFile.java:78) ~[sshj-0.31.0.jar!/:0.31.0]
at org.onedatashare.transferservice.odstransferservice.service.step.sftp.SshJWriter.write(SshJWriter.java:101) ~[classes!/:0.0.1-SNAPSHOT]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:193) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:159) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.write(SimpleChunkProcessor.java:294) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:217) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:77) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) ~[spring-tx-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91) ~[spring-batch-core-4.2.4.RELEASE.jar!/:4.2.4.RELEASE]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
net.schmizz.sshj.connection.ConnectionException: Stream closed
at net.schmizz.sshj.connection.channel.ChannelOutputStream.checkClose(ChannelOutputStream.java:158)
at net.schmizz.sshj.connection.channel.ChannelOutputStream.write(ChannelOutputStream.java:137)
at net.schmizz.sshj.connection.channel.ChannelOutputStream.write(ChannelOutputStream.java:131)
at net.schmizz.sshj.sftp.SFTPEngine.transmit(SFTPEngine.java:300)
at net.schmizz.sshj.sftp.SFTPEngine.request(SFTPEngine.java:138)
at net.schmizz.sshj.sftp.RemoteResource.close(RemoteResource.java:53)
at org.onedatashare.transferservice.odstransferservice.service.step.sftp.SshJWriter.afterStep(SshJWriter.java:59)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.batch.support.SimpleMethodInvoker.invokeMethod(SimpleMethodInvoker.java:108)
at org.springframework.batch.core.listener.MethodInvokerMethodInterceptor.invoke(MethodInvokerMethodInterceptor.java:69)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy146.afterStep(Unknown Source)
at org.springframework.batch.core.listener.CompositeStepExecutionListener.afterStep(CompositeStepExecutionListener.java:64)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:247)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:94)
at org.springframework.batch.core.job.flow.support.state.SplitState$1.call(SplitState.java:91)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
```
Any advice on how to pass very large(11MB+) byte[] to the RemoteFile would be great. As I only ever experience this issue with chunk sizes above 32KB+.
Thank you in advance.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.