jenkinsci / jenkinsci/docker-workflow-plugin
[JENKINS-65404] Command line limit hit on windows nodes when using docker agent
- Dominant language
- Java
- Stars
- 527
- Forks
- 422
- Avg merge
- 10m
- Merged PRs (30d)
- 1
Description
When Gitlab branch source plugin triggers a job, all commits and commit messages since the last build gets added as environment variables. Docker workflow plugin appends all environment variables to a command line to run the docker container, i.e.
docker run -d -t -w [working directory] [volume mappings] -e ******** -e ******** [image] cmd.exe
With large enough change sets, we encounter
```
java.io.IOException: CreateProcess error=206, The filename or extension is too long
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.100.33.107/10.100.33.107:55682
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1001)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1121)
at hudson.Launcher$ProcStarter.start(Launcher.java:508)
at org.jenkinsci.plugins.docker.workflow.client.WindowsDockerClient.launch(WindowsDockerClient.java:131)
at org.jenkinsci.plugins.docker.workflow.client.WindowsDockerClient.run(WindowsDockerClient.java:54)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:198)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: Cannot run program "docker": CreateProcess error=206, The filename or extension is too long
at java.lang.ProcessBuilder.start(Unknown Source)
at hudson.Proc$LocalProc.(Proc.java:252)
at hudson.Proc$LocalProc.(Proc.java:221)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
at hudson.Launcher$ProcStarter.start(Launcher.java:508)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1390)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1333)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:376)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:119)
at java.lang.Thread.run(Unknown Source)
```
We have only observed this on windows agents, running windows containers.
With only two commits and two changed files we have close to 6000 characters in the environment, and the command line limit on windows is 8191 characters: https://docs.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation
A possible solution could be to use --env-file, as described here: https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file
The issue is reproduced with the attached pipeline.groovy, containing a large environment block, (no need for gitlab to reproduce the issue). The cause is the large amount of data in the environment variables and names.
---
Originally reported by
r01ne, imported from: Command line limit hit on windows nodes when using docker agent
Raw content of original issue
When Gitlab branch source plugin triggers a job, all commits and commit messages since the last build gets added as environment variables. Docker workflow plugin appends all environment variables to a command line to run the docker container, i.e.
docker run -d -t -w [working directory] [volume mappings] -e ******** -e ******** [image] cmd.exeWith large enough change sets, we encounter
java.io.IOException: CreateProcess error=206, The filename or extension is too long
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.100.33.107/10.100.33.107:55682
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1001)
at hudson.Launcher$RemoteLauncher.launch(Launcher.java:1121)
at hudson.Launcher$ProcStarter.start(Launcher.java:508)
at org.jenkinsci.plugins.docker.workflow.client.WindowsDockerClient.launch(WindowsDockerClient.java:131)
at org.jenkinsci.plugins.docker.workflow.client.WindowsDockerClient.run(WindowsDockerClient.java:54)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:198)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: Cannot run program "docker": CreateProcess error=206, The filename or extension is too long
at java.lang.ProcessBuilder.start(Unknown Source)
at hudson.Proc$LocalProc.<init>(Proc.java:252)
at hudson.Proc$LocalProc.<init>(Proc.java:221)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:996)
at hudson.Launcher$ProcStarter.start(Launcher.java:508)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1390)
at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:1333)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:376)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:119)
at java.lang.Thread.run(Unknown Source)We have only observed this on windows agents, running windows containers.
With only two commits and two changed files we have close to 6000 characters in the environment, and the command line limit on windows is 8191 characters: https://docs.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation
A possible solution could be to use --env-file, as described here: https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file
The issue is reproduced with the attached pipeline.groovy, containing a large environment block, (no need for gitlab to reproduce the issue). The cause is the large amount of data in the environment variables and names.
environment
```
Jenkins 2.277.2
Docker workflow plugin 1.26
Gitlab branch source plugin 1.5.5 (Not required to reproduce issue)
```
1 attachment
- [pipeline.groovy](https://issues.jenkins.io/secure/attachment/54631/pipeline.groovy)
Contributor guide
Assessment
This issue has not been assessed yet.