Exclude directories

Can someone add a function where you can also exclude specific directories? Like, don't crawl example.com/products/ (and all products deeper inside the path?

Start at the crawler entry point and trace how URLs are selected and expanded from the starting site. Define done as excluding the requested directory path and deeper paths while continuing to crawl other URLs, then add or run coverage for that behavior if the project provides crawler tests.

Written by the indexing model from the issue text.