Major Performance issue even with new caching?

Hi @cebe

Almost 2 years ago we had a rather big issue with the performance for parsing an API-spec, where all components were store in another (the same) file. It took a couple of minutes to process. My collegue then proposed this PR #65

We ended up using this fork for the last two years now. This week I'm updating dependencies, and having seen that you merged #67 made me realize, that I can finally get rid of our fork.

However, running our test-suite after the update was rather painful: Instead of ~1s to parse our specification, like with our fork, it now took ~30s. I spent a couple of hours analyzing, but I couldn't find the issue. It looks like resolving references is painfully slow, it seems to take 100-300ms for each reference. Clearly, this was much faster in version 1.4, where our fork originates.

---

Therefore, I hope you can find a minute to look into it as well. I prepared a minimal test setup:

```console
$ git clone https://github.com/lextira/openapi-parse-test
$ composer update

# now we have a composer install with our fork, see https://github.com/lextira/openapi-parse-test/blob/master/composer.json
$ php test.php
Start Reading...
Read took 1248 miliseconds!
Parsed file is 791837 characters long.

$ git checkout with-openapi-1.6
$ composer update

# new we are at the latest release of the official repository
$ php test.php
Start Reading...
Read took 37302 miliseconds!
Parsed File is 791837 characters long.
```

So with v1.6, reading the exact same file from the exact same URL takes almost 30 times longer than with v1.4 (and our custom cache implementation, which is very similar to your implemented cache now). For context, the specification used for the test is our official API documentation and can be found here: https://docs.lexgate.ch/api/

Commencez par la reproduction minimale sur github.com/lextira/openapi-parse-test et exécutez test.php sur le fork basé sur v1.4 et la release officielle v1.6. Examinez le chemin de résolution des références et le comportement du cache, en utilisant la spécification OpenAPI liée comme charge de travail. Le travail est considéré comme terminé lorsque le chemin v1.6 n'affiche plus la régression de parsing signalée par rapport au fork.

Rédigé par le modèle d'indexation à partir du texte de l'issue.