job_chat: code replacement might fail with duplicate code sections

The PR #259 which gives job_chat the ability to return edited code can sometimes fail to apply the code edits suggested by the LLM. If the user's job code has multiple consecutive sections of identical code and only some parts are targeted with the code edit, the LLM calls can struggle to a) output enough context to select the correct passage of old code to replace, or b) to output the correct number of instances of the duplicate passages in the edited code, leading to deletions.

Example input with six identical passages to replicate the issue:

{ "history": [], "content": "I need to add error handling only to the fifth POST request to retry once if it fails.", "context": { "expression": "// Process and prepare data\nfn(state => {\n const items = state.data.items.map(item => ({\n id: item.id,\n name: item.name,\n status: 'pending'\n }));\n \n return { ...state, items };\n});\n\npost('https://api.example.com/endpoint', state => state.items);\n\npost('https://api.example.com/endpoint', state => state.items);\n\npost('https://api.example.com/endpoint', state => state.items);\n\npost('https://api.example.com/endpoint', state => state.items);\n\npost('https://api.example.com/endpoint', state => state.items);\n\npost('https://api.example.com/endpoint', state => state.items);" }, "meta": {} }

A version of this example is also included in the tests in job_chat as test_duplicate_sections_additional

Example failure output, where the corrector step still doesn't provide enough context for string replacement to work:

INFO:job_chat:Corrector response: { "explanation": "Added unique surrounding context to target only the fifth POST request by including the fourth POST request before it and the sixth POST request after it.", "corrected_old_code": "post('https://api.example.com/endpoint', state => state.items);\n\npost('https://api.example.com/endpoint', state => state.items);\n\npost('https://api.example.com/endpoint', state => state.items);", "corrected_new_code": "post('https://api.example.com/endpoint', state => state.items);\n\nfn(async state => {\n try {\n const response = await post('https://api.example.com/endpoint', state => state.items)(state);\n return response;\n } catch (error) {\n console.log('First attempt failed, retrying once...');\n return post('https://api.example.com/endpoint', state => state.items)(state);\n }\n});\n\npost('https://api.example.com/endpoint', state => state.items);" } WARNING:job_chat:Corrected old code appears 2 times in the code. Applying edit to first occurrence only.

We've already mitigated this problem for most cases with a separate model call that only deals with code edit corrections. However, as LLMs struggle to keep count accurately, this can still fail. The error correction step can also introduce new errors and make things worse (e.g. through malformed text that will not match the existing code). However, on balance, in repeated tests, it improves results in at least 90% of cases. We think it's not worth trying to improve it further unless this surfaces in user testing.

Start with job_chat's test_duplicate_sections_additional and reproduce the six-identical-passage example; compare the correction step's output with the string-replacement warning. Done would require a demonstrated improvement that avoids incorrect or destructive edits in this scenario, but the issue notes that further work is not currently considered worthwhile without user-testing evidence.