Remaining cmd.exe dependencies hang the session when the command prompt is disabled by Group Policy
- Dominant language
- Java
- Stars
- 5.1k
- Forks
- 1.2k
- PR merge metrics
- PR metrics pending
Description
#18751 removed `cmd.exe` from the git and svn startup probes, which is what kept RStudio from starting at all under `DisableCMD` (#18735). The underlying mechanism is still present everywhere else: on Windows every `core::system::runCommand()` is `cmd.exe /S /C`, and a `cmd.exe` disabled by Group Policy prints its message and then blocks on console input that nothing ever writes.
## Remaining automatic path
`src/cpp/session/modules/clang/CompilationDatabase.cpp:354` and `:367` run `R CMD config CXX` and ` --version` through `runCommand()` with no `_WIN32` fork. For any package project with `src/` files, `initializeDefinitionIndex()` schedules indexing shortly after the file monitor comes up, and `updateForCurrentPackage()` evaluates `computeCompilerHash()` (line 565) *before* the `isCurrent` check (line 567), so the spawn happens even when the cached compilation database is fresh. The result is that client init succeeds and the IDE renders, then the session goes unresponsive on first idle. The same chain fires on any `.cpp` edit or save.
These two sites are not a mechanical conversion to `runProgram()`: `R CMD config CXX` can return a command with flags (e.g. `g++ -std=gnu++17`), so `compilerCommand + " --version"` would need command-line splitting.
## Remaining user-initiated paths
- `src/cpp/session/modules/SessionSVN.cpp` `createConsoleProc()` still builds `svn() << args.args()` as a shell string for `ConsoleProcess::create(command, ...)`, so every async SVN operation (update, commit, checkout, history, show) spawns a hidden `cmd.exe` that never exits and leaves the progress dialog spinning.
The NOTE above that call site justifies the shell-string form with "so that we can redirect standard output to a file", which is not accurate on Windows: `ChildProcess::run()` applies `options_.stdOutFile` via `STARTUPINFO` regardless of which `init()` overload was used, and `init(exe, args, options)` only emits `LOG_ERROR_MESSAGE("stdOutFile/stdErrFile options cannot be used with runProgram")` without clearing the field. Relaxing that guard on Win32 and using `ConsoleProcess::create(s_svnExePath, args.args(), ...)` -- as git's `createConsoleProc()` already does -- would close this one.
- `src/cpp/session/modules/SessionWorkbench.cpp:343` (ssh-keygen)
- `src/cpp/session/modules/SessionSVN.cpp` (patch, ~line 1207)
- `src/cpp/session/modules/clang/CodeCompletion.cpp:252` and `:304`
- `src/cpp/session/modules/build/SessionBuild.cpp:2447`
## Suggested approach
A core-level fail-fast would cover all of them at once. In `ChildProcess::run()` (`src/cpp/core/system/Win32ChildProcess.cpp`, which already returns `Error`), when `exe_` has resolved to `cmd.exe`, read `DisableCMD` from `HKCU`/`HKLM\Software\Policies\Microsoft\Windows\System` and return a descriptive `Error` instead of spawning. A value of 1 blocks `/C` scripts. `core::system::RegistryKey` already exists (used by `RToolsInfo.cpp`) and would need a DWORD getter. That also covers `resolveCommand()`'s own `.cmd` -> `cmd.exe` rewrite, so `runProgram()` callers benefit too.
Turning the hang into an actionable error is a smaller and more complete change than forking each call site, though the clang sites may still want a real `runProgram()` conversion so compiler detection keeps working under the policy.
Reported by @kevinushey in review of #18751.
Contributor guide
Research direction
Start in src/cpp/core/system/Win32ChildProcess.cpp at ChildProcess::run(), and inspect the RegistryKey usage in RToolsInfo.cpp. Trace the listed clang, SVN, ssh-keygen, patch, code-completion, and build call sites to confirm their process paths. Done means a disabled cmd.exe fails descriptively without hanging, with relevant callers still handled under the policy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, r
- Domain
- desktop, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100