Why looping through hdus at pos=2 in fits files is much slower than looping through hdus at pos=1?

### Description
I am looping through the hdus of astro fits files I am analyzing. When I loop through the hdus at position 1 (`hdu=1`) with `astropy.table.read` the time for that is reasonably short. But if I do the same for the hdus at position 2 (`hdu=2`) the time is 10 times slower, despite that the size of hdu=2 is smaller than for hdu=1. The tables/hdus at position 2 are smaller (~27 KB) than the tables at position 1 (~130 kB). Further, tables at position 1 have shape 2833 rows * 8 columns and tables at position 2 have shape 1 row * 126 columns. The time matters to me, because I need to loop through tens of thousands of files eventually, so in the magnitude of hours.

Example file can be this one: `https://data.sdss.org/sas/dr16/eboss/spectro/redux/v5_13_0/spectra/lite/3699/spec-3699-55517-0420.fits`

### Expected behavior
Looping should be the same or maybe faster for hdu=2.

### Actual behavior
Looping through hdu=2 is much slower (~10 times).

### Steps to Reproduce
To reproduce you can do

```
from astropy.table import Table

for file_name in fits_files_list:
table1 = Table.read(file_name, hdu=1)

# fits_files_list is a list of containing the file paths to fits files. If you want to reproduce my code you can download an example from here:
# https://data.sdss.org/sas/dr16/eboss/spectro/redux/v5_13_0/spectra/lite/3699/spec-3699-55517-0420.fits
# this is the sort of files I am working with now, they all have the same structure
```
Change then `hdu=1` to `hdu=2`, keep track of the execution time and note that the second time it is ~10 times slower.

### System Details

The details to my system/environment are:
Linux-5.11.0-40-generic-x86_64-with-glibc2.10
Python 3.8.5 (default, Sep 4 2020, 07:30:14)
[GCC 7.3.0]
Numpy 1.19.1
astropy 3.2.3

I have to erfa or scipy installed.

## How can I speed up the looping through hdu=2? Why is it 10 times slower? Are there any work around? Tnx

Reproduce the timing difference with astropy.table.Table.read using hdu=1 and hdu=2 on the linked SDSS FITS file, under the reported Python, NumPy, and Astropy versions. Compare the work performed for both HDUs and establish whether the slowdown is expected or a bug; done means identifying the cause and documenting a verified workaround or fix direction.

Written by the indexing model from the issue text.