HTTP Task failure: Unable to connect to Postgres whilst looking up IP Blacklist on GCP using private networking

Description
HTTP Task failure: Unable to connect to Postgres whilst looking up IP Blacklist on GCP using private networking

When executing a HTTP Task as part of a standard "Get > Uint256" job, the task fails complaining about a "disallowed IP".

The full error in the CL Node GUI is:
connections to local resources are disabled by default, if you are sure this is safe, you can enable on a per-task basis by setting allowUnrestrictedNetworkAccess="true" in the pipeline task spec, e.g. fetch [type="http" method=GET url="$(decode_cbor.url)" allowUnrestrictedNetworkAccess="true"]: error making http request: Get "https://min-api.cryptocompare.com/data/pricemultifull?fsyms=ETH&tsyms=USD": disallowed IP 40.115.22.134. Connections to local/private and multicast networks are disabled by default for security reasons: disallowed IP

The job and included HTTP task used to run successfully under smartcontract/chainlink:1.4.1. It now fails under smartcontract/chainlink:1.7.1 and 1.8.0.

The bug is actually caused by a failure of the node to lookup the list of blacklisted IP addresses. The node cannot establish a DB connection to retrieve the blacklist and in failing to do so blacklists any IP address by default. The blacklist feature was introduced in 1.5.0.

The establishment of a DB connection to fetch the blacklist seems to be different than in other parts of the node software. The node has no issues booting up and loading its configuration including lists of jobs and tasks, and the GUI is operating as expected.

Further investigation shows that the code to fetch the blacklist performs a check on the database's IP address by calling net.LookupIP(dbURL.String())
See https://github.com/smartcontractkit/chainlink/blob/v1.5.0/core/utils/http/http_allowed_ips.go#L73
The full error log is attached below.
This net.LookupIP() call only exists in this part of the node software and isn't used by any parts of the node software during the establishment of a DB connection.

I think this check is failing because we're running Postgres via GCP's hosted CloudSQL using private IP networking. The node software is running under Kubernetes and can establish DB connection using the private IP address of the CloudSQL instance. The node software is also able to resolve the private IP address of the CloudSQL instance using the Kubernetes DNS service. However, the net.LookupIP() call is failing to resolve the private IP address of the CloudSQL instance.
We've also tested that we can connect from inside the K8S cluster to the Postgres DB using Google's cloud_sql_proxy tool using the private IP address of the CloudSQL instance. However we cannot ping the private IP address of the CloudSQL instance from the node container.

Basic Information

• Network: Rinkeby (although issue isn't EVM network specific)
• Operating System: Container-Optimized OS from Google
• Kubernetes Version: 1.22.10-gke.600
• Commit: [log INFO line when starting node]
• Hosting Provider: GCP
• Startup Command: Kubernetes pod runnning smartcontract/chainlink:1.8.0-root ["local", "n", "-p", "/chainlink/.password", "-a", "/chainlink/.api"]

Environment Variables
ADVISORY_LOCK_CHECK_INTERVAL 1s
ADVISORY_LOCK_ID 1027321974924625846
ALLOW_ORIGINS *
BLOCK_BACKFILL_DEPTH 10
BLOCK_HISTORY_ESTIMATOR_BLOCK_DELAY 0
BLOCK_HISTORY_ESTIMATOR_BLOCK_HISTORY_SIZE 0
BLOCK_HISTORY_ESTIMATOR_TRANSACTION_PERCENTILE 0
BRIDGE_RESPONSE_URL
CHAINLINK_DEV false
CHAINLINK_PORT 6688
CHAINLINK_TLS_HOST
CHAINLINK_TLS_PORT 0
CHAINLINK_TLS_REDIRECT false
CHAIN_TYPE
DATABASE_BACKUP_FREQUENCY 1h0m0s
DATABASE_BACKUP_MODE none
DATABASE_BACKUP_ON_VERSION_UPGRADE true
DATABASE_LOCKING_MODE lease
DEFAULT_HTTP_LIMIT 32768
DEFAULT_HTTP_TIMEOUT 15s
ETH_CHAIN_ID 4
ETH_HTTP_URL
ETH_SECONDARY_URLS [https://rinkeby.infura.io/v3/xxx]
ETH_URL wss://rinkeby.infura.io/ws/v3/xxx
EVM_RPC_ENABLED true
EXPLORER_URL
FEATURE_EXTERNAL_INITIATORS false
FEATURE_OFFCHAIN_REPORTING false
FLAGS_CONTRACT_ADDRESS
FM_DEFAULT_TRANSACTION_QUEUE_DEPTH 1
GAS_ESTIMATOR_MODE
INSECURE_FAST_SCRYPT false
JOB_PIPELINE_REAPER_INTERVAL 1h0m0s
JOB_PIPELINE_REAPER_THRESHOLD 24h0m0s
JSON_CONSOLE true
KEEPER_BASE_FEE_BUFFER_PERCENT 20
KEEPER_CHECK_UPKEEP_GAS_PRICE_FEATURE_ENABLED false
KEEPER_DEFAULT_TRANSACTION_QUEUE_DEPTH 1
KEEPER_GAS_PRICE_BUFFER_PERCENT 20
KEEPER_GAS_TIP_CAP_BUFFER_PERCENT 20
KEEPER_MAXIMUM_GRACE_PERIOD 100
KEEPER_REGISTRY_CHECK_GAS_OVERHEAD 200000
KEEPER_REGISTRY_PERFORM_GAS_OVERHEAD 150000
KEEPER_REGISTRY_SYNC_INTERVAL 30m0s
KEEPER_REGISTRY_SYNC_UPKEEP_QUEUE_SIZE 10
KEEPER_TURN_FLAG_ENABLED false
KEEPER_TURN_LOOK_BACK 1000
LEASE_LOCK_DURATION 10s
LEASE_LOCK_REFRESH_INTERVAL 1s
LINK_CONTRACT_ADDRESS
LOG_FILE_DIR /chainlink
LOG_FILE_MAX_AGE 0
LOG_FILE_MAX_BACKUPS 1
LOG_FILE_MAX_SIZE 1.02gb
LOG_LEVEL debug
LOG_SQL false
OCR_DEFAULT_TRANSACTION_QUEUE_DEPTH 1
OCR_TRACE_LOGGING false
P2PV2_ANNOUNCE_ADDRESSES []
P2PV2_BOOTSTRAPPERS []
P2PV2_DELTA_DIAL 15s
P2PV2_DELTA_RECONCILE 1m0s
P2PV2_LISTEN_ADDRESSES []
P2P_BOOTSTRAP_CHECK_INTERVAL 20s
P2P_BOOTSTRAP_PEERS []
P2P_DHT_LOOKUP_INTERVAL 10
P2P_INCOMING_MESSAGE_BUFFER_SIZE 10
P2P_LISTEN_IP 0.0.0.0
P2P_LISTEN_PORT
P2P_NETWORKING_STACK V1
P2P_NEW_STREAM_TIMEOUT 10s
P2P_OUTGOING_MESSAGE_BUFFER_SIZE 10
P2P_PEER_ID
REAPER_EXPIRATION 240h0m0s
ROOT /chainlink
SECURE_COOKIES false
SESSION_TIMEOUT 15m0s
SHUTDOWN_GRACE_PERIOD 5s
TELEMETRY_INGRESS_LOGGING false
TELEMETRY_INGRESS_SERVER_PUB_KEY
TELEMETRY_INGRESS_URL
TRIGGER_FALLBACK_DB_POLL_INTERVAL 30s

Node Logs:
{"caller":"http/http_allowed_ips.go:62", "err":"failed to lookup IP for DB URL: lookup postgresql://chainlink:xxx@10.84.48.7:5432/chainlink-rinkeby?application_name=Chainlink+1.8.0+%7CDefault: no such host", "errVerbose":"lookup postgresql://chainlink:xxx@10.84.48.7:5432/chainlink-rinkeby?application_name=Chainlink+1.8.0+%7CDefault: no such host
failed to lookup IP for DB URL
github.com/smartcontractkit/chainlink/core/utils/http.isBlacklistedIP
/chainlink/core/utils/http/http_allowed_ips.go:76
github.com/smartcontractkit/chainlink/core/utils/http.isRestrictedIP
/chainlink/core/utils/http/http_allowed_ips.go:60
github.com/smartcontractkit/chainlink/core/utils/http.makeRestrictedDialContext.func1
/chainlink/core/utils/http/http_allowed_ips.go:105
net/http.(*Transport).dial
/usr/local/go/src/net/http/transport.go:1169
net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1607
net/http.(*Transport).dialConnFor
/usr/local/go/src/net/http/transport.go:1449
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1571", "ip":"40.115.22.134", "level":"error", "logger":"1.8.0@bb6e95a", "msg":"Failed to check IP blacklist status, this IP will be blocked", "sentryEventID":null, "stacktrace":"github.com/smartcontractkit/chainlink/core/utils/http.isRestrictedIP
/chainlink/core/utils/http/http_allowed_ips.go:62
github.com/smartcontractkit/chainlink/core/utils/http.makeRestrictedDialContext.func1
/chainlink/core/utils/http/http_allowed_ips.go:105
net/http.(*Transport).dial
/usr/local/go/src/net/http/transport.go:1169
net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1607
net/http.(*Transport).dialConnFor
/usr/local/go/src/net/http/transport.go:1449", "ts":"2022-09-03T04:35:58.438Z"}

Steps to Reproduce
Deploy a chainlink node to GKE using the smartcontract/chainlink:1.8.0-root image. Configure the node to use a CloudSQL instance as the database. Attempt to connect to the CloudSQL instance using the private IP address.

Deploy a standard "Get > Uint256" job to the node. Observe that the job fails to run the httpGet task.

Additional Information
The workaround is to allowUnrestrictedNetworkAccess="true" on the httpGet task so that the blacklist is bypassed. This is not ideal as it raises security concerns.

Start in core/utils/http/http_allowed_ips.go, especially isBlacklistedIP, isRestrictedIP, and the net.LookupIP call described in the report. Reproduce the failure with a GKE node using a private CloudSQL address and a standard HTTP task; done means the blacklist check no longer blocks the HTTP request because the database lookup fails.