Why Are Certain Layer Normalization Parameters Cast to trt_llm_config.dtype Instead of Using float32 for Better Precision?

In the following file:

tensorrt_llm/models/gemma/convert.py

Why, in the following section:

elif any(keyword in name for keyword in (
                "pre_attention_norm.scale",
                "pre_ffw_norm.scale",
                "final_norm.scale",
                "pre_attention_norm/vars/0",
                "pre_ffw_norm/vars/0",
                "rms_normalization/vars/0",
                "input_layernorm",
                "post_attention_layernorm",
                "pre_feedforward_layernorm",
                "post_feedforward_layernorm",
                "model.norm.weight",
                "q_norm.weight",
                "k_norm.weight",
        )):
            param = param + 1.0  # upcasted to float32 in case of bfloat16
            add_trt_llm_weight(weights, trt_llm_name, param,
                               trt_llm_config.dtype)

is there a strict insistence that these layers are served using trt_llm_config.dtype?

These layers have a very small number of parameters, so serving them in float32 would have a minimal impact on GPU memory usage. On the other hand, it would improve precision.