Using Separate Hypernetworks for Each Transformer Layer

Hello,

I noticed in the paper that a single hypernetwork is used to generate parameters for all layers. Have you tried using a separate hypernetwork for each Transformer layer? Would this lead to a larger performance improvement?

Thanks!