nextToken coming back null when limit exceeded

I had a pipeline that was behaving normal and returning a list of data points similar to this:
{
nextToken: String,
data: [DataPoint]
}
type DataPoint { value: Float, timeStamp: String }

The request was a simple Query request to a DynamoDB table (index).

I needed to expand this to return another value, so I updated DataPoint to return instead:
{ value: Float, rawData: Float, timeStamp: String }

As such, I'm returning a lot more data. In my frontend app, I was only receiving data for the first 10 or so days of my query after this change. I verified that nextToken is always coming back null now. I opened up the Query tab in AppSync console in AWS, and ran the query manually, and the same thing happened. I didn't get back all of my data, but nextToken is null, and there is no other indication that I hit any limits. By updating my resolver request template to add a limit of 1000, nextToken began to report a value back, and things are correctly paginated into my frontend. Setting the limit to 2000, I received more data but with large holes in the results, as if some of them returned nextToken and some didn't. I suspect maybe there's something strange going on here and just wanted to report it.

The work around is to use a low value for the limit parameter in the Query request, but it seems somewhat arbitrary picking one that works and doesn't return holes in the data. How do I know if I'm near some edge of a limit, which isn't being reported to me. I would like to keep the limit parameter as high as possible so the frontend loads quickly. Any suggestions?

Start by reproducing the DynamoDB index Query through the AppSync console with the described DataPoint fields and different limit values. Trace when nextToken becomes null and verify that pagination returns all data points without holes or silently truncated results.

Written by the indexing model from the issue text.