can we set fill values for compound types with size larger than 64 MB?

This is coming from a netCDF issue: https://github.com/Unidata/netcdf-c/issues/2738

Is it possible to set a fill value for a compound type, when the size of the type is >= 64 MB?

According to claude code:

> Why >64 MB compound fill values cause problems
> Fill values are stored in the object header. When you call H5Pset_fill_value and create a dataset, HDF5 serializes the fill value bytes directly into the dataset's object header in the HDF5 file. The object header is a fixed-size metadata structure.
>
> The 64 KB object header limit. HDF5 object headers have a default maximum size. In the HDF5 file format spec, a single object header message has a maximum data size of 64 KB (not 64 MB — the limit is actually much smaller than 64 MB). A fill value larger than this cannot fit in a single header message.
>
> HDF5 1.8+ uses "object header version 2" which supports continuation blocks and can store larger messages, but even then there are practical limits. The real-world problem reported in netCDF-4 issue #2738 is:
>
> For large compound types, H5Pset_fill_value or H5Dcreate may succeed (HDF5 accepts it)
> But the resulting object header becomes enormous — potentially hundreds of MB
> This causes HDF5 to emit errors like:
> H5Oedit.c: object header too large
> or silently produce a corrupt/unreadable file.
> Why compound types specifically? The fill value buffer must be exactly sizeof(compound_type) bytes. For a compound type with a 64 MB array field, the fill value is literally 64 MB of raw bytes that HDF5 must embed in the object header metadata of that dataset. This is fundamentally different from a VLEN fill value (which is just a hvl_t struct — a pointer and a length, 16 bytes) even though the data it points to may be large.

Here's a HDF5 program which seems to confirm that claude is correct and HDF5 will not let me set fill values for compound types larger than 64 MB in size:

```
/* This is part of the netCDF package. Copyright 2018 University
Corporation for Atmospheric Research/Unidata See COPYRIGHT file for
conditions of use.

This program tests HDF5 compound types of varying sizes, including
types whose total data size is less than, equal to, and greater than
64 MB. This is relevant to netCDF-4 issue #2738, which concerns HDF5
object header size limits when storing fill values for large compound
types.

Edward Hartnett, 4/24/26
*/

#include "h5_err_macros.h"
#include
#include
#include
#include

#define FILE_NAME "tst_h_large_compound.h5"
#define DIM1_LEN 3
#define DATASET_NAME "compound_var"
#define FIELD_NAME "data_field"
#define FILL_VAL -42

/* Number of ints in the array field for each test. */
#define SMALL_NINTS 1
#define EXACT_NINTS (64 * 1024 * 1024 / sizeof(int))
#define LARGE_NINTS (64 * 1024 * 1024 / sizeof(int) + 1)

/* Write one record of a compound type whose single field is an array
of nints integers into a DIM1_LEN-element dataset. The dataset is
created with a fill value whose every integer is fill_val. Only the
first element is written; the remaining elements are left to take
the fill value. The file is then reopened and all elements are read
back and verified. */
static int
test_compound(size_t nints, int fill_val)
{
hid_t fileid, spaceid, typeid, array_typeid, datasetid, dcplid, wspaceid;
hsize_t dims[1] = {DIM1_LEN};
hsize_t wdims[1] = {1};
hsize_t field_dims[1];
int *data_out = NULL;
int *data_in = NULL;
int *fill_buf = NULL;
size_t compound_size = nints * sizeof(int);
size_t i, j;

field_dims[0] = nints;

/* Allocate and initialise the single-record write buffer. */
if (!(data_out = malloc(compound_size))) ERR;
for (i = 0; i < nints; i++)
data_out[i] = (int)i;

/* Allocate and initialise the fill value buffer. */
if (!(fill_buf = malloc(compound_size))) ERR;
for (i = 0; i < nints; i++)
fill_buf[i] = fill_val;

/* Allocate the read buffer for all DIM1_LEN records. */
if (!(data_in = malloc(compound_size * DIM1_LEN))) ERR;
memset(data_in, 0, compound_size * DIM1_LEN);

/* Create file. */
if ((fileid = H5Fcreate(FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT,
H5P_DEFAULT)) < 0) ERR;

/* Create a 1-D array HDF5 type for the field. */
if ((array_typeid = H5Tarray_create2(H5T_NATIVE_INT, 1, field_dims)) < 0) ERR;

/* Create the compound type containing that array field. */
if ((typeid = H5Tcreate(H5T_COMPOUND, compound_size)) < 0) ERR;
if (H5Tinsert(typeid, FIELD_NAME, 0, array_typeid) < 0) ERR;

/* Create dataset creation property list and set fill value. */
if ((dcplid = H5Pcreate(H5P_DATASET_CREATE)) < 0) ERR;
if (H5Pset_fill_value(dcplid, typeid, fill_buf) < 0) ERR;

/* Create dataspace and dataset. */
if ((spaceid = H5Screate_simple(1, dims, NULL)) < 0) ERR;
if ((datasetid = H5Dcreate1(fileid, DATASET_NAME, typeid,
spaceid, dcplid)) < 0) ERR;

/* Write only the first element using a memory and file selection of size 1. */
if ((wspaceid = H5Screate_simple(1, wdims, NULL)) < 0) ERR;
if (H5Dwrite(datasetid, typeid, wspaceid, wspaceid,
H5P_DEFAULT, data_out) < 0) ERR;

if (H5Dclose(datasetid) < 0) ERR;
if (H5Sclose(spaceid) < 0) ERR;
if (H5Sclose(wspaceid) < 0) ERR;
if (H5Pclose(dcplid) < 0) ERR;
if (H5Tclose(array_typeid) < 0) ERR;
if (H5Tclose(typeid) < 0) ERR;
if (H5Fclose(fileid) < 0) ERR;

/* Reopen the file and read all DIM1_LEN records back. */
if ((fileid = H5Fopen(FILE_NAME, H5F_ACC_RDONLY, H5P_DEFAULT)) < 0) ERR;

if ((array_typeid = H5Tarray_create2(H5T_NATIVE_INT, 1, field_dims)) < 0) ERR;
if ((typeid = H5Tcreate(H5T_COMPOUND, compound_size)) < 0) ERR;
if (H5Tinsert(typeid, FIELD_NAME, 0, array_typeid) < 0) ERR;

if ((datasetid = H5Dopen1(fileid, DATASET_NAME)) < 0) ERR;
if (H5Dread(datasetid, typeid, H5S_ALL, H5S_ALL,
H5P_DEFAULT, data_in) < 0) ERR;

/* First record must match what was written. */
for (i = 0; i < nints; i++)
if (data_in[i] != data_out[i]) ERR;

/* Remaining records must contain the fill value. */
for (j = 1; j < DIM1_LEN; j++)
for (i = 0; i < nints; i++)
if (((int *)(data_in + j * nints))[i] != fill_val) ERR;

if (H5Dclose(datasetid) < 0) ERR;
if (H5Tclose(array_typeid) < 0) ERR;
if (H5Tclose(typeid) < 0) ERR;
if (H5Fclose(fileid) < 0) ERR;

free(data_out);
free(data_in);
free(fill_buf);
return 0;
}

int
main()
{
printf("\n*** Checking HDF5 large compound types.\n");

/* Test 1: Small compound type (SMALL_NINTS int fields) with fill value.
The compound record is well under 64 MB. A fill value of FILL_VAL
is set on the dataset. Only the first of DIM1_LEN elements is
written; the remaining elements must read back as FILL_VAL. */
printf("*** Checking HDF5 compound type smaller than 64 MB with fill value...");
if (test_compound(SMALL_NINTS, FILL_VAL)) ERR;
SUMMARIZE_ERR;

/* Test 2: Compound type exactly 64 MB in size with fill value.
The single array field contains EXACT_NINTS integers making the
compound record exactly 64 MB. The fill value buffer is likewise
64 MB. HDF5 should accept and store this fill value and return it
for unwritten elements on read. */
printf("*** Checking HDF5 compound type exactly 64 MB in size with fill value...");
if (test_compound(EXACT_NINTS, FILL_VAL)) ERR;
SUMMARIZE_ERR;

/* Test 3: Compound type larger than 64 MB with fill value.
The single array field contains LARGE_NINTS integers making the
compound record just over 64 MB. The fill value buffer is likewise
just over 64 MB. HDF5 should still accept and store this fill value
and return it for unwritten elements on read. */
printf("*** Checking HDF5 compound type larger than 64 MB with fill value...");
if (test_compound(LARGE_NINTS, FILL_VAL)) ERR;
SUMMARIZE_ERR;

FINAL_RESULTS;
}

```

Running this test produces the following output:

```
*** Checking HDF5 large compound types.
*** Checking HDF5 compound type smaller than 64 MB with fill value...ok.
HDF5-DIAG: Error detected in HDF5 (2.1.0):
#000: /home/ed/Downloads/hdf5-2.1.0/src/H5Ddeprec.c line 137 in H5Dcreate1(): unable to create dataset
major: Dataset
minor: Unable to initialize object
#001: /home/ed/Downloads/hdf5-2.1.0/src/H5VLcallback.c line 1982 in H5VL_dataset_create(): dataset create failed
major: Virtual Object Layer
minor: Unable to create file
#002: /home/ed/Downloads/hdf5-2.1.0/src/H5VLcallback.c line 1947 in H5VL__dataset_create(): dataset create failed
major: Virtual Object Layer
minor: Unable to create file
#003: /home/ed/Downloads/hdf5-2.1.0/src/H5VLnative_dataset.c line 282 in H5VL__native_dataset_create(): unable to create dataset
major: Dataset
minor: Unable to initialize object
#004: /home/ed/Downloads/hdf5-2.1.0/src/H5Dint.c line 398 in H5D__create_named(): unable to create and link to dataset
major: Dataset
minor: Unable to initialize object
#005: /home/ed/Downloads/hdf5-2.1.0/src/H5Lint.c line 522 in H5L_link_object(): unable to create new link to object
major: Links
minor: Unable to initialize object
#006: /home/ed/Downloads/hdf5-2.1.0/src/H5Lint.c line 768 in H5L__create_real(): can't insert link
major: Links
minor: Unable to insert object
#007: /home/ed/Downloads/hdf5-2.1.0/src/H5Gtraverse.c line 846 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#008: /home/ed/Downloads/hdf5-2.1.0/src/H5Gtraverse.c line 618 in H5G__traverse_real(): traversal operator failed
major: Symbol table
minor: Callback failed
#009: /home/ed/Downloads/hdf5-2.1.0/src/H5Lint.c line 567 in H5L__link_cb(): unable to create object
major: Links
minor: Unable to initialize object
#010: /home/ed/Downloads/hdf5-2.1.0/src/H5Oint.c line 2384 in H5O_obj_create(): unable to open object
major: Object header
minor: Can't open object
#011: /home/ed/Downloads/hdf5-2.1.0/src/H5Doh.c line 273 in H5O__dset_create(): unable to create dataset
major: Dataset
minor: Unable to initialize object
#012: /home/ed/Downloads/hdf5-2.1.0/src/H5Dint.c line 1357 in H5D__create(): can't update the metadata cache
major: Dataset
minor: Unable to initialize object
#013: /home/ed/Downloads/hdf5-2.1.0/src/H5Dint.c line 1019 in H5D__update_oh_info(): unable to update new fill value header message
major: Dataset
minor: Unable to initialize object
#014: /home/ed/Downloads/hdf5-2.1.0/src/H5Omessage.c line 157 in H5O_msg_append_oh(): unable to create new message in header
major: Attribute
minor: Unable to insert object
#015: /home/ed/Downloads/hdf5-2.1.0/src/H5Omessage.c line 192 in H5O__msg_append_real(): unable to create new message
major: Object header
minor: No space available for allocation
#016: /home/ed/Downloads/hdf5-2.1.0/src/H5Omessage.c line 1741 in H5O__msg_alloc(): unable to allocate space for message
major: Object header
minor: Unable to initialize object
#017: /home/ed/Downloads/hdf5-2.1.0/src/H5Oalloc.c line 1255 in H5O__alloc(): object header message is too large
major: Object header
minor: Unable to initialize object
*** Checking HDF5 compound type exactly 64 MB in size with fill value...Sorry! Unexpected result, tst_h_large_compound.c, line: 87
Sorry! Unexpected result, tst_h_large_compound.c, line: 152
```