lighthouse downloads chunk marked as prefetch and displays warning while the network tab doesn't show the chunk as loaded

I've a fairly big app ( site + shop ) written in vue and split in several chunks using webpack.

Now when auditing the site I only get a 82 in performance and get the warning to Avoid eourmous network payloads.

However, I split the code and components that way that the admin stuff shouldn't load, and I think it isn't loaded because when I navigate to the network tab this particular chunk doesn't load, but when I run lighthouse it loads multiple chunks that don't show up on the network panel and are marked as prefetch in my html markup.

Any Idea why that is? I really would like to get that 100 lighthouse score.. :>