Adopt WebDriverAbstract as a solution for active (JavaScript) websites

Hello everyone,

A few weeks ago I came across the well-known problem that rss-bridge doesn't work for some websites. This is always the case when the website loads some content via XMLHttpRequest (XHR) and / or changes it using JavaScript.
After some experimentation and thinking about the problem, I came to the conclusion that it can only be solved using a web browser. Coincidentally, I already had a lot of experience with Selenium and the WebDriver and decided to come up with a solution.

I now suggest that the WebDriverAbstract class be imported into the project, temporarily in its own branch. As examples I have written two bridges that use these.

The following problems still need to be solved:

• I didn't commit my composer.lock because the changes look random and I didn't want to invest time (yet) to understand composer.
• I have added the php-webdriver dependencies to bootstrap.php, but they are not complete. They look messy and unsorted because of the dependencies between them. I suggest to use the autoloader.
• All bridges that depend on WebDriverAbstract would create high(er) load on a multi-user (open) server. Maybe there should be an option to disable the class instead of each bridge using it.
• Because loading web pages through a browser is such a heavier load, it would make sense to stop scraping as soon as an element is in a cache (assuming the elements are sorted chronologically). Is there already a cache for elements that have already been scraped? How can I use it? I looked around the code but didn't find any examples of this or documentation.
• What do you miss in the documentation for WebDriverAbstract?

Kind regards,

Holger

Start by reviewing the proposed WebDriverAbstract class, its two example bridges, bootstrap.php, and composer.lock. Trace the php-webdriver dependencies and existing cache or scraping documentation before deciding how adoption should work. Done means the dependencies and autoloading are complete, server load can be controlled, scraping can use the relevant cache behavior, and WebDriverAbstract documentation gaps are addressed.