Pinning files smaller than the Navie limit but larger than the OpenAI limit causes an error

Using VS Code:

Version: 1.93.0 (Universal)
Commit: 4849ca9bdf9666755eb463db297b69e5385090e3
Date: 2024-09-04T13:02:38.431Z
Electron: 30.4.0
ElectronBuildId: 10073054
Chromium: 124.0.6367.243
Node.js: 20.15.1
V8: 12.4.254.20-electron.0
OS: Darwin arm64 23.6.0

AppMap extension v0.129.0+pin-files-20240905 from #1979

Project: https://github.com/land-of-apps/sample_rails_app - using mainline branch

Loom recreation of issue: https://www.loom.com/share/f35943f69a5f4b0dbd00cf6ecde75542

Sample files used:
otel_traces.txt
server_output.txt

When you open Navie and pin these 2 files into the context, then ask a question, you'll get an error in the chat window saying:

SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON

The JSON file is valid JSON confirmed with jq.

In AppMap services, the error says:

68416 [Stderr] PayloadTooLargeError: request entity too large
68416 [Stderr]     at readStream (/snapshot/appmap-js/node_modules/body-parser/node_modules/raw-body/index.js:163:17)
68416 [Stderr]     at getRawBody (/snapshot/appmap-js/node_modules/body-parser/node_modules/raw-body/index.js:116:12)
68416 [Stderr]     at read (/snapshot/appmap-js/node_modules/body-parser/lib/read.js:79:3)
68416 [Stderr]     at jsonParser (/snapshot/appmap-js/node_modules/body-parser/lib/types/json.js:138:5)
68416 [Stderr]     at call (/snapshot/appmap-js/node_modules/connect/index.js:239:7)
68416 [Stderr]     at next (/snapshot/appmap-js/node_modules/connect/index.js:183:5)
68416 [Stderr]     at corsHandler (/snapshot/appmap-js/node_modules/connect-cors/lib/connect-cors.js:137:14)
68416 [Stderr]     at call (/snapshot/appmap-js/node_modules/connect/index.js:239:7)
68416 [Stderr]     at next (/snapshot/appmap-js/node_modules/connect/index.js:183:5)
68416 [Stderr]     at Function.handle (/snapshot/appmap-js/node_modules/connect/index.js:186:3)
68416 [Stderr]     at Server.app (/snapshot/appmap-js/node_modules/connect/index.js:51:37)
68416 [Stderr]     at Server.emit (node:events:537:28)
68416 [Stderr]     at parserOnIncoming (node:_http_server:993:12)
68416 [Stderr]     at HTTPParser.parserOnHeadersComplete (node:_http_common:117:17)

If I cut the file sizes down to total of under 100kb then i'm able to get a response from Navie. We should return a better error in the chat window so the user knows to reduce the size of the files added to the context.

Start by reproducing the Navie flow with the two pinned files described in the issue and inspect the AppMap services request handling shown in the body-parser stack trace. The work is done when an oversized context produces a clear chat-window message telling the user to reduce the pinned file sizes instead of exposing a JSON parsing error.