boot-clj / boot-clj/boot

`boot.pod/make-pod` (and thus `boot.core/aot`) fail with envs that are too large

Open
#733 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Clojure
Stars
1.7k
Forks
178
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
`boot.pod/make-pod` (and thus its dependents like `boot.core/aot`) fails with environments that are sufficiently large. I came across this bug while working on a project with a lot of dependencies (760 jars) and thus a `:boot-class-path` of 83324 chars. The reason for the failure is that environments beyond a certain size cannot be embedded in code within `boot.pod/init-pod!`:
https://github.com/boot-clj/boot/blob/8b5fd047bd1c6fd33438e31062780084aca37dee/boot/pod/src/boot/pod.clj#L850

**To Reproduce**
Steps to reproduce the behavior:
1. From the REPL (though this bug also occurs from the commandline, it's easier to reproduce at the REPL), run `(boot.pod/make-pod env)` where `env` is a "sufficiently large" Boot environment. In my case it failed with a `:boot-class-path` of 83324 chars (also failed with `:fake-class-path`), but didn't care about the `:dependencies` size (it failed when I kept `:boot-class-path` in the env, whether or not I set `:dependencies` to `[]`).
2. Observe that the following exception occurs:
```
;; Given a dependency of Clojure 1.10
java.lang.IllegalArgumentException:
clojure.lang.Compiler$CompilerException: Syntax error compiling fn* at (0:0).

;; Given a dependency of Clojure 1.8 or 1.9
java.lang.ClassFormatError: Unknown constant tag 112 in class file clojure/core$eval21
clojure.lang.Compiler$CompilerException: java.lang.ClassFormatError: Unknown constant tag 112 in class file clojure/core$eval21, compiling:(NO_SOURCE_PATH:0:0)
```

**Expected behavior**
`boot.pod/make-pod` and dependents (`boot.core/aot` among them) should gracefully handle very large Boot environments.

**Desktop**
- OS: Mac OS 10.14.1
- Java: 1.8

**Additional context**
This is not due to the one classpath string being too big, as I tried splitting it up at compile-time via `partition-all`, embedding that in code, and recombining at runtime.

The workaround I'm using right now is to spit the environment to a file and read it in again:
```clojure
(defn- init-pod!
[env pod]
;; This was added
(let [temp-file (doto (java.io.File/createTempFile "env" ".edn") (.deleteOnExit))]
;; This was added
(spit temp-file env)
(doto pod
(require-in "boot.pod")
(with-invoke-in (boot.pod/set-worker-pod! worker-pod))
(with-eval-in
(require 'boot.util 'boot.pod)
(reset! boot.util/*verbosity* ~(deref util/*verbosity*))
;; This was changed
(alter-var-root #'boot.pod/env (constantly (slurp ~(.getPath temp-file))))
(create-ns 'pod)
(dosync (alter @#'clojure.core/*loaded-libs* conj 'pod))
(alter-var-root #'*ns* (constantly (the-ns 'pod)))
(clojure.core/refer-clojure)))))
```

Contributor guide

Open the contributing guide

Research direction

Start in boot/pod/src/boot/pod.clj at init-pod! around the referenced line, then reproduce the failure with boot.pod/make-pod and a large environment such as one containing an 83324-character :boot-class-path. Compare this path with the reported temporary-file workaround; done means make-pod and dependents such as boot.core/aot handle very large environments without the shown compiler or class-format errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
build-system, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.