cisagov / cisagov/pshtt

Problems with hstspreload.org

Open
#241 1 comment 0 reactions 1 assignee Claimed by @jsf9k View on GitHub
bug
Dominant language
Python
Stars
693
Forks
91
No merged PRs in 30d

Description

# 🐛 Summary #

The `hstspreload.org` dependency is resulting in the inability to actually scan anything without any real errors or notices. My experience for the past hour has been the inability to do anything with PSHTT due to the dependecy on hstspreload, which is currently exhibiting odd connection responses. This can also happen in isolated environments where one would like to scan without the ability to connect out to the internet.

## To reproduce ##

Steps to reproduce the behavior:

1. Connect to an isolated network or run PSHTT during very high load times
2. Run PSHTT with debug flag
3. Observe indefinite hang at hstspreload connection like so:

```
pshtt myhosts.csv -j -d --timeout=5
-------------------------

Fetching Chrome preload list from source...
Starting new HTTPS connection (1): chromium.googlesource.com:443
https://chromium.googlesource.com:443 "GET /chromium/src/+/main/net/http/transport_security_state_static.json?format=TEXT HTTP/1.1" 200 None

-------------------------

Fetching hstspreload.org pending list...
Starting new HTTPS connection (1): hstspreload.org:443
```

## Expected behavior ##

There should be a way to either forgo this mandatory connection, or the connection should respect the timeout flag even though the server hasn't presented flags that it's hanging on. This can be seen where CURLing the domain during this time will hang indefinitely as well, but if you specify a timeout with CURL, you'll get the homepage.

## Any helpful log output or screenshots ##

Expected cURL result that only happens when using timeout flag around these times:

```console
[root@host user]# curl -vm 3 https://hstspreload.org
* About to connect() to hstspreload.org port 443 (#0)
* Trying 216.239.32.21...
* After 1498ms connect time, move on!
* Trying 216.239.34.21...
* After 748ms connect time, move on!
* Trying 216.239.38.21...
* Connected to hstspreload.org (216.239.38.21) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=hstspreload.org
* start date: Apr 12 15:03:56 2023 GMT
* expire date: Jul 11 15:55:28 2023 GMT
* common name: hstspreload.org
* issuer: CN=GTS CA 1D4,O=Google Trust Services LLC,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: hstspreload.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Fri, 12 May 2023 22:58:33 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 13379
< Vary: Accept-Encoding
< Accept-Ranges: bytes
< Last-Modified: Wed, 03 May 2023 16:27:17 GMT
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Via: 1.1 google
<



HSTS Preload List Submission









window.addEventListener('load', function() {
new Form(PreloadController);
});

On GitHub




Enter a domain:





document.getElementById("domain-form").classList.remove("hidden");
Submitting entries to the HSTS preload list via this site requires JavaScript.



Submit




I am the site owner of example.com or have their permission to preload HSTS.


(If this is not the case, example.com may be sending the HSTS preload directive by accident. Please contact hstspreload@chromium.org to let us know.)





I understand that preloading example.com through this form will prevent all subdomains and nested subdomains from being accessed without a valid HTTPS certificate:
*.example.com
*.*.example.com
...










Information



This form is used to submit domains for inclusion in Chrome's HTTP Strict Transport Security (HSTS) preload list.
This is a list of sites that are hardcoded into Chrome as being HTTPS only.



Most major browsers (Chrome, Firefox, Opera, Safari, IE 11 and Edge) also have HSTS preload lists based on the Chrome list. (See the HSTS compatibility matrix.)



Submission Requirements


If a site sends the preload directive in an HSTS header, it is considered to be requesting inclusion in the preload list and may be submitted via the form on this site.


In order to be accepted to the HSTS preload list through this form, your site must satisfy the following set of requirements:



  1. Serve a valid certificate.

  2. Redirect from HTTP to HTTPS on the same host, if you are listening on port 80.

  3. Serve all subdomains over HTTPS.

    • In particular, you must support HTTPS for the www subdomain if a DNS record for that subdomain exists.



  4. Serve an HSTS header on the base domain for HTTPS requests:

    • The max-age must be at least 31536000 seconds (1 year).

    • The includeSubDomains directive must be specified.

    • The preload directive must be specified.

    • If you are serving an additional redirect from your HTTPS site, that redirect must still have the HSTS header (rather than the page it redirects to).





For more details on HSTS, please see RFC 6797.

Here is an example of a valid HSTS header:



Strict-Transport-Security: max-age=63072000; includeSubDomains; preload


You can check the status of your request by entering the domain name again in the form above, or consult the current Chrome preload list by visiting chrome://net-internals/#hsts in your browser.
Note that new entries are hardcoded into the Chrome source code and can take several months before they reach the stable version.



Continued Requirements



You must make sure your site continues to satisfy the submission requirements at all times. Note that removing the preload directive from your header will make your site immediately eligible for the removal form, and that sites may be removed at any time for failing to keep up the requirements.



In particular, the requirements above apply to all domains submitted through hstspreload.org on or after October 11, 2017 (i.e. preloaded after Chrome 63)



The same requirements apply to earlier domains submitted on or after February 29, 2016 (i.e. preloaded after Chrome 50), except that the required max-age for those domains is only 10886400 seconds.



Deployment Recommendations


If your site is committed to HTTPS and you want to preload HSTS, we suggest the following steps:


  1. Examine all subdomains (and nested subdomains) of your site and make sure that they work properly over HTTPS.

  2. Add the Strict-Transport-Security header to all HTTPS responses and ramp up the max-age in stages, using the following header values:


    • 5 minutes:

      max-age=300; includeSubDomains


    • 1 week:

      max-age=604800; includeSubDomains


    • 1 month:

      max-age=2592000; includeSubDomains


    During each stage, check for broken pages and monitor your site's metrics (e.g. traffic, revenue). Fix any problems that come up and then wait the full max-age of the stage before you move on. For example, wait a month in the last stage.
  3. Once you're confident that there will be no more issues, increase the max-age to 2 years and submit your site to the preload list:


    • 2 years, requesting to be preloaded:

      max-age=63072000; includeSubDomains; preload




If you have a group of employees or users who can beta test the deployment, consider trying the first few ramp-up stages on those users. Then make sure to go through all stages for all users, starting over from the beginning.


Consult the Mozilla Web Security guidelines and the Google Web Fundamentals pages on security for more concrete advice about HTTPS deployment.


Preloading Should Be Opt-In



If you maintain a project that provides HTTPS configuration advice or provides an option to enable HSTS, do not include the preload directive by default. We get regular emails from site operators who tried out HSTS this way, only to find themselves on the preload list by the time they find they need to remove HSTS to access certain subdomains. Removal tends to be slow and painful for those sites.



It's great to support HSTS preloading as a best practice, and for projects to provide a simple option to enable it. However, site operators who enable HSTS should know about the long-term consequences of preloading before they turn it on for a given domain. They should also be informed that they need to meet additional requirements and submit their site to hstspreload.org to ensure that it is successfully preloaded (i.e. to get the full protection of the intended configuration).



Removal



Be aware that inclusion in the preload list cannot easily be undone.
Domains can be removed, but it takes months for a change to reach users with a Chrome update and we cannot make guarantees about other browsers.
Don't request inclusion unless you're sure that you can support HTTPS for your entire site and all its subdomains in the long term.


However, we will generally honor requests to be removed from Chrome's preload list if you find that you have a subdomain that you cannot serve over HTTPS for strong technical or cost reasons.
To request removal, please visit the removal form.



TLD Preloading



Owners of gTLDs, ccTLDs, or any other public suffix domains are welcome to preload HSTS across all their registerable domains. This ensures robust security for the whole TLD, and is much simpler than preloading each individual domain. Please contact us if you're interested, or would like to learn more.



Contact



Want to remove your domain? Please visit the removal form.



Else, if you have questions or requests that are not covered by this site, email us here using an appropriate subject line and one of the preload list maintainers will be in contact soon.



* Connection #0 to host hstspreload.org left intact
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.