Scrolling happens when user returns to a page with hash using browser back button

Bug report

Describe the bug

When user returns to a page which have url with hash (e.g. http://localhost:3000/test#anchor) from another page, unexpected scrolling to the anchor element happen.
It seems componentDidUpdate of Container triggers scrollToHash even when one returns by clicking browser back button.
(https://github.com/vercel/next.js/blob/v9.4.4/packages/next/client/index.js#L135-L137)

To Reproduce

Example repo: https://github.com/monae/nextjs-scrolltohash-bug

1. git clone git@github.com:monae/nextjs-scrolltohash-bug.git and npm run dev.
2. Click on 'Open the test page without hash'. You are on top of the page.
3. Scroll down and click on 'Open another page'.
4. Click on browser back button, then you return to the preserved scroll position (Chrome, Safari) or top of the page (Firefox).
5. Back to the first page and click on 'Open the test page with hash'. You are on the anchor element.
6. Scroll down and click on 'Open another page'.
7. Click on browser back button, then you return to the anchor element.

Expected behavior

The results of 4. & 7. should be same (although the behavior of Firefox is not desirable).

Screenshots

System information

• OS: macOS
• Browser (if applies) Chrome, Safari, Firefox
• Version of Next.js: 9.4.4
• Version of Node.js: 14.3.0

Additional context

Reproduce the behavior with the linked monae/nextjs-scrolltohash-bug example, then inspect packages/next/client/index.js around lines 135-137, where Container componentDidUpdate triggers scrollToHash. Compare browser-back behavior for URLs with and without hashes across the listed browsers. Done means returning with a hash preserves the same scroll behavior as returning without one.