Error in "_merge_no_wait": The config isn't consistent between chunks. This shouldn't have happened.

Hello,

I am pretraining Tinyllama on Lightning AI studio on my custom dataset. I am using prepare_starcoder.py to convert the parquet files because my data has one folder of parquet files. After it writes .bin files it raises an error in the commented section below.

Error:

raise Exception("The config isn't consistent between chunks. This shouldn't have happened."

File location:

/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/lightning/data/streaming/writer.py

I could not resolve the problem, commented it out, and trained the model. But, I want to ensure it does not affect anything bad. I would appreciate it if you could address the issue.

    def _merge_no_wait(self, node_rank: Optional[int] = None) -> None:
        """Once all the workers have written their own index, the merge function is responsible to read and merge them
        into a single index."""
        files = os.listdir(self._cache_dir)
        index_files = [f for f in files if f.endswith(_INDEX_FILENAME)]

        chunks_info = []
        config = None
        for index_filename in sorted(index_files):
            chunk_path = os.path.join(self._cache_dir, index_filename)
            with open(chunk_path) as f:
                data = json.load(f)

                if config is None:
                    config = data["config"]

                #elif config != data["config"]:

                   # print(config)
                   # print("\n\n\n")
                   # print(data['config'])

                   # breakpoint()
                    #raise Exception("The config isn't consistent between chunks. This shouldn't have happened.")

                chunks_info.extend(data["chunks"])

            os.remove(chunk_path)

        if node_rank is None:
            with open(os.path.join(self._cache_dir, _INDEX_FILENAME), "w") as f:
                json.dump({"chunks": chunks_info, "config": config}, f, sort_keys=True)
        else:
            with open(os.path.join(self._cache_dir, f"{node_rank}-{_INDEX_FILENAME}"), "w") as f:
                json.dump({"chunks": chunks_info, "config": config}, f, sort_keys=True)

Start with prepare_starcoder.py and the reported lightning/data/streaming/writer.py location, then reproduce the parquet conversion with the same dataset and inspect the differing chunk configs. Done means identifying why the configs diverge, confirming whether the merged index remains valid, and verifying the conversion and subsequent training without commenting out the consistency check.