[Go] What is the best way to handle dictionary overflow?

Arrow allows declaring a string dictionary with different indice type (e.g. uint8, uint16, ...). Unfortunately, sometimes we don't know in advance the exact cardinality of a column, so deciding in advance the proper indice type is not always feasible and we have to rely on some kind of adaptive approach.

I'd like to automatically determine when a dictionary usage overflows (during insertion) and then fallback to a larger indice type or directly to a string when I've reached the maximum indice size I want for a specific application. So far, I haven't found a very effective way to detect this overflow. The `AppendString` method doesn't return an error in case of overflow. So I have to insert all my data, then visit all my dictionaries and call the `Dictionary().Len()` methods to check if the size is greater than the maximum value for the current indice type.

Is there a better approach to handling dictionary overflow with Go Arrow SDK?

Note 1: Not detecting dictionary overflow can lead to a whole bunch of problems that are currently silently ignored by the existing infrastructure. For example, the creation of a record with this type of overflow could be sent on an IPC stream without any problem. However, on reception the content of the reconstructed record will not be strictly equivalent because the indices of the dictionary have not magically exceeded their limit as defined in the schema.

Note 2: I didn't open a Jira issue as it's not really a bug per se but more a question.

@zeroshade I would like to draw your attention to this question as it is very close to the subject we discussed a few days ago.

Start with the Go Arrow SDK entry points named in the issue: AppendString and Dictionary().Len(), and trace how dictionary indices are inserted and validated. Compare the current behavior with IPC record serialization and define what observable overflow handling and equivalent reconstruction should look like.

Written by the indexing model from the issue text.