The serialized result of document modification loses or duplicates comments

I've tried to replicate the problem we have in our language with an existing test model, but the issue seems quite specific to the grammar involved.

The org.eclipse.xtext.ui.tests.editor.model.edit.XtextDocumentModifyTest test is similar in structure to what is failing, but I was not able to replicate "our" problem with that test case's model, though I did encounter a different problem via this additional test in that class:

	@Test
	public void testComments() throws Exception {
		// @formatter:off
		final String grammar = text(
				"grammar foo.Foo", 
				"generate foo \"http://foo.net/foo\"",
				"@Ann // Ann",
				"@Ann2 // Ann2",
				" Foo: 'foo';"
				);
		// @formatter:on
		final IXtextDocument document = createDocument(grammar);
		document.modify(new IUnitOfWork.Void<XtextResource>() {
			@Override
			public void process(XtextResource state) throws Exception {
				Grammar grammar = (Grammar) state.getContents().get(0);
				AbstractRule abstractRule = grammar.getRules().get(0);
				EList<Annotation> annotations = abstractRule.getAnnotations();
				annotations.get(1).setName("Ann3");
			}
		});
		String result = document.get();
		assertEquals(grammar.replace("@Ann2", "@Ann3"), result);
	}

It fails like this where the result is not even syntactically correct:

This test class is more representative of the type of problem I'm seeing.

package org.eclipse.xtext.ui.tests.editor.model.edit;

import org.eclipse.emf.common.util.EList;
import org.eclipse.emf.ecore.EObject;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.xtext.junit4.AbstractXtextTests;
import org.eclipse.xtext.parser.antlr.Lexer;
import org.eclipse.xtext.parser.antlr.internal.InternalXtextLexer;
import org.eclipse.xtext.resource.OutdatedStateManager;
import org.eclipse.xtext.resource.XtextResource;
import org.eclipse.xtext.service.OperationCanceledManager;
import org.eclipse.xtext.ui.editor.model.DocumentTokenSource;
import org.eclipse.xtext.ui.editor.model.IXtextDocument;
import org.eclipse.xtext.ui.editor.model.XtextDocument;
import org.eclipse.xtext.ui.editor.model.edit.ITextEditComposer;
import org.eclipse.xtext.ui.tests.FoldingTestLanguageStandaloneSetup;
import org.eclipse.xtext.ui.tests.folding.Element;
import org.eclipse.xtext.ui.tests.folding.FoldingFactory;
import org.eclipse.xtext.ui.tests.folding.FoldingModel;
import org.eclipse.xtext.util.StringInputStream;
import org.eclipse.xtext.util.concurrent.IUnitOfWork;
import org.junit.Test;

import com.google.common.base.Joiner;
import com.google.inject.Provider;

public class FoldingTestLanguageDocumentModifyTest extends AbstractXtextTests {

	private Resource resource;

	@Override
	public void setUp() throws Exception {
		super.setUp();
		with(FoldingTestLanguageStandaloneSetup.class);
	}

	@Test
	public void testAddElement() throws Exception {
		final String model = text( //
				"element Root", //
				" element child1 end // C1", //
				" element child2 end // C2", //
				" // C3", //
				"end" //
		);
		final IXtextDocument document = createDocument(model);
		document.modify(new IUnitOfWork.Void<XtextResource>() {
			@Override
			public void process(XtextResource state) throws Exception {
				EList<EObject> contents = state.getContents();
				FoldingModel foldingModel = (FoldingModel) contents.get(0);
				EList<Element> elements = foldingModel.getElements();
				Element rootElement = elements.get(0);
				EList<Element> subelements = rootElement.getSubelements();
				Element newElement = FoldingFactory.eINSTANCE.createElement();
				newElement.setName("newElement");
				subelements.add(1, newElement);
			}
		});
		final String expected = text( //
				"element Root", //
				" element child1 end // C1", //
				" element newElement end", //
				" element child2 end // C2", //
				" // C3", //
				"end" //
		);
		String result = document.get();
		assertEquals(expected, result);
	}

	@Test
	public void testMoveElement() throws Exception {
		final String model = text( //
				"element Root", //
				" element child1 end // C1", //
				" element child2 end // C2", //
				" // C3", //
				"end" //
		);
		final IXtextDocument document = createDocument(model);
		document.modify(new IUnitOfWork.Void<XtextResource>() {
			@Override
			public void process(XtextResource state) throws Exception {
				EList<EObject> contents = state.getContents();
				FoldingModel foldingModel = (FoldingModel) contents.get(0);
				EList<Element> elements = foldingModel.getElements();
				Element rootElement = elements.get(0);
				EList<Element> subelements = rootElement.getSubelements();
				subelements.move(1, 0);
			}
		});
		final String expected = text( //
				"element Root", //
				" element child2 end // C2", //
				" element child1 end // C1", //
				" // C3", //
				"end" //
		);
		String result = document.get();
		assertEquals(expected, result);
	}

	private IXtextDocument createDocument(String model) throws Exception {
		resource = getResource(new StringInputStream(model));
		DocumentTokenSource tokenSource = new DocumentTokenSource();
		tokenSource.setLexer(new Provider<Lexer>() {
			@Override
			public Lexer get() {
				return new InternalXtextLexer();
			}
		});

		final XtextDocument document = new XtextDocument(tokenSource, get(ITextEditComposer.class), new OutdatedStateManager(),
				new OperationCanceledManager()) {
			@Override
			public <T> T internalModify(IUnitOfWork<T, XtextResource> work) {
				try {
					return work.exec((XtextResource) resource);
				} catch (Exception e) {
					throw new RuntimeException(e);
				}
			}
		};
		document.set(model);
		return document;
	}

	private String text(String... lines) {
		return Joiner.on(System.lineSeparator()).join(lines);
	}
}

It fails like this where a comment has been duplicated:

And like this where a comment has gone missing:

I've tried to investigate how to fix these problems but that's easier said that done.

Part of the problem (in my case) is in the org.eclipse.xtext.formatting2.regionaccess.internal.HiddenRegionPartAssociator.associate method where the partial serializer serializes content that later needs to be associated but I get the sense that this code assumes the that when region.getPreviousSemanticRegion() is null it should skip associating with the previous, but it's only looking at a snippet in the middle of other content. But the above test case does not reach this code...

Another part of the problem (in my case) is in org.eclipse.xtext.formatting2.regionaccess.internal.StringBasedTextRegionAccessDiffBuilder.create() where content appears to go missing. So I "fixed" it in my case with this hack:

				// EATM2
				appender.copyAndAppend(f, PREVIOUS);
				appender.copyAndAppend(f, NEXT);

But none of this seems to kick in for this test case, though those parts do kick in for the XtextDocumentModifyTest . I don't understand why.

I'd like to try to help fix this problem, but I would need some guidance...

Start by reproducing the failures in XtextDocumentModifyTest and the FoldingTestLanguageDocumentModifyTest examples, where comments are duplicated, lost, or produce invalid output. Then trace document.modify through HiddenRegionPartAssociator.associate and StringBasedTextRegionAccessDiffBuilder.create, comparing the affected regions and serialized result. Done means document modification preserves each comment exactly and produces syntactically valid output.