Enable PETSc for HIP machines

When building Serac TPLs for tioga/ rzvernal I encountered two problems related to building PETSc:

1. PETSc doesn't like when the HIP prefix is `/opt/rocm-5.7.1` in the `spack.yaml` (prefers `/opt/rocm-5.7.1/hip`, which messes up Axom, Tribol, Serac build systems).
2. PETSc had to be built with no optimization flags (Serac spec contained `^petsc cflags=-O0`)

The process of getting PETSc to build previously is outlined here: https://github.com/spack/spack/issues/46468

Ideally, we want to build PETSc while keeping the HIP prefix the same and with full optimization flags. This will likely involve changes in PETSc's Spack package and/ or a patch, as well as some help from PETSc developers.