Caching of babel.config.js doesn't seem to be working properly when using api.caller for different webpack targets

**I'm submitting a bug report**

**Webpack Version:**
4.29.6

**Babel Core Version**:
7.10.4

**Babel Loader Version**:
8.0.0

**Please tell us about your environment:**
Windows 10, WSL, Ubuntu

**Current behavior:**
Using one babel.config.js file, trying to change this config based on two targets, web and node, using api.caller. I want tree shaking just for web, and not for node environment. When I have my webpack config with target set to 'web' it works correctly, with treeshaking. But if I add another webpack config to my array of webpack exports and that config has the target set to 'node', then my treeshaking no longer works for the web bundle. I believe babel caches the config function, when using api.caller, but I would think that when the target changes it will invalidate the cache. This seems to be the recommended style of config syntax, am I missing something?

My babel.config.js looks like this:
```

const basePlugins = [
'babel-plugin-styled-components',
'@babel/transform-async-to-generator',
'@babel/plugin-syntax-dynamic-import',
'react-loadable/babel',
'babel-plugin-root-import',
'@babel/plugin-proposal-class-properties',
];

module.exports = (api) => {
// Flag when the target is for the web so we will do treeshaking
// Otherwise, assume it is a node environment(eg. tests or node running server side)
const isWebackTargetWeb = api.caller(caller => caller && caller.target === 'web');

const configObj = {
sourceType: (isWebackTargetWeb ? 'unambiguous' : 'module'),
presets: [
'@babel/preset-react',
'@babel/typescript',
['@babel/preset-env', {
targets: '> 0.3%, not IE 11, not op_mini all',
useBuiltIns: 'usage',
corejs: { version: '3.8', proposals: false },
...(isWebackTargetWeb && { modules: false }),
}],
],
env: {
development: {
sourceMaps: true,
plugins: ['source-map-support'],
},
},
plugins: basePlugins,
ignore: ['node_modules/is_js'],
};

if (!isWebackTargetWeb) {
configObj.plugins.push('@babel/plugin-transform-modules-commonjs');
}

return configObj;
};

```

**Expected/desired behavior:**
When having two webpack configs exported from my webpack config file, one having target set to 'web', and one target set to 'node', tree shaking should still work correctly.

* **If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem along with a gist/jsbin of your webpack configuration.**
Steps:
Use the babel config I provided above. And then export two webpack configs from one webpack.config.js file. Run in production mode with NODE_ENV=production. Use webpack-bundle-analyzer plugin to see the output is not correct

* **What is the motivation / use case for changing the behavior?**
Need to be able to change babel.config.js based on webpack target

Start by reproducing the issue with the provided babel.config.js and two target configurations exported from webpack.config.js, using production mode and webpack-bundle-analyzer. Inspect how the loader caches configuration between the web and node targets. Done means the web bundle retains tree shaking while the node bundle uses its separate configuration.

Written by the indexing model from the issue text.