embulk / embulk/embulk-input-sftp

Support new OpenSSH format private keys with passphrase

Open
#41 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Hi, all

I faced a problem when connecting my SFTP server with new OpenSSH format.

### Environment

```
% embulk gem list | grep embulk
embulk (0.9.18 java)
embulk-input-sftp (0.3.2)
```

### Overview

Connecting to SFTP server failed when executing `embulk guess seed.yml -o config.yml` by using OpenSSH private key accompanying passphrase.

### Problem

[OpenSSH 6.5](https://lwn.net/Articles/583485/) released new private key format when `ssh-keygen` and the format has been default in [OpenSSH 7.8](https://lwn.net/Articles/763444/) since last year.

And then, if new default format is set, `embulk` processes are failed.
For example, when I setup SFTP server and tried executing Embulk, I received `rg.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server` and `Could not connect to SFTP server`.

```
% embulk guess seed.yml -o config.yml

java.lang.RuntimeException: org.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server at "sftp://prev_header@{host}/".
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.embulk.input.sftp.SftpFileInput.listFilesByPrefix(SftpFileInput.java:303)
at org.embulk.input.sftp.SftpFileInputPlugin.transaction(SftpFileInputPlugin.java:23)
at org.embulk.spi.FileInputRunner.transaction(FileInputRunner.java:62)
at org.embulk.exec.SamplingParserPlugin.runFileInputSampling(SamplingParserPlugin.java:47)
at org.embulk.spi.FileInputRunner.guess(FileInputRunner.java:80)
at org.embulk.exec.GuessExecutor.doGuess(GuessExecutor.java:108)
at org.embulk.exec.GuessExecutor.access$000(GuessExecutor.java:32)
at org.embulk.exec.GuessExecutor$1.run(GuessExecutor.java:81)
at org.embulk.exec.GuessExecutor$1.run(GuessExecutor.java:78)
at org.embulk.spi.Exec.doWith(Exec.java:22)
at org.embulk.exec.GuessExecutor.guess(GuessExecutor.java:78)
at org.embulk.EmbulkEmbed.guess(EmbulkEmbed.java:183)
at org.embulk.EmbulkRunner.guessInternal(EmbulkRunner.java:203)
at org.embulk.EmbulkRunner.guess(EmbulkRunner.java:60)
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:427)
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
at org.embulk.cli.Main.main(Main.java:64)
Caused by: org.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server at "sftp://prev_header@{host}/".
at org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:86)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:92)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:71)
at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:55)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:711)
at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:648)
at org.embulk.input.sftp.SftpFileInput$1.call(SftpFileInput.java:226)
at org.embulk.input.sftp.SftpFileInput$1.call(SftpFileInput.java:205)
at org.embulk.spi.util.RetryExecutor.run(RetryExecutor.java:81)
at org.embulk.spi.util.RetryExecutor.runInterruptible(RetryExecutor.java:62)
at org.embulk.input.sftp.SftpFileInput.listFilesByPrefix(SftpFileInput.java:205)
... 16 more
Caused by: org.apache.commons.vfs2.FileSystemException: Could not load private key from "org.apache.commons.vfs2.provider.sftp.IdentityInfo@59cde35".
at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.addIndentity(SftpClientFactory.java:174)
at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.addIdentities(SftpClientFactory.java:157)
at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.createConnection(SftpClientFactory.java:83)
at org.apache.commons.vfs2.provider.sftp.SftpFileProvider.doCreateFileSystem(SftpFileProvider.java:79)
... 26 more
Caused by: com.jcraft.jsch.JSchException: invalid privatekey: [B@c29b0dc
at com.jcraft.jsch.KeyPair.load(KeyPair.java:664)
at com.jcraft.jsch.KeyPair.load(KeyPair.java:561)
at com.jcraft.jsch.IdentityFile.newInstance(IdentityFile.java:40)
at com.jcraft.jsch.JSch.addIdentity(JSch.java:423)
at org.apache.commons.vfs2.provider.sftp.SftpClientFactory.addIndentity(SftpClientFactory.java:172)
... 29 more

Error: org.apache.commons.vfs2.FileSystemException: Could not connect to SFTP server at "sftp://prev_header@{host}/".
```

The `in` directive of `seed.yml` file used at that time is below:

```
in:
type: sftp
host: HOST
port: PORT
user: USER
secret_key_file:
content: |
-----BEGIN OPENSSH PRIVATE KEY-----
...
-----END OPENSSH PRIVATE KEY-----
secret_key_passphrase: *****
user_directory_is_root: false
timeout: 300
path_prefix: /path/to/file
```

Thus, we will be happy to new key format available in this plugin.

### Workaround

I succeeded to generate `config.yml` by using previous default format, PEM style keys.
The previous style format can be generated by adding `-m PEM` option to `ssh-keygen` command or using OpenSSH before version 7.7.

```
in:
type: sftp
host: HOST
port: PORT
user: USER
secret_key_file:
content: |
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
...
-----END RSA PRIVATE KEY-----
secret_key_passphrase: *****
user_directory_is_root: false
timeout: 300
path_prefix: /path/to/file
```

Thanks,

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.