Parsely / Parsely/wp-parsely

Add Support to Trigger Article Crawls & Hook into Appropriate Actions

Open
#265 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Enhancement wp-parsely
Dominant language
PHP
Stars
66
Forks
36
Avg merge
2d 2h
Merged PRs (30d)
16

Description

There's an API endpoint to instruct the service to re-crawl an Article:

Normally, articles are crawled during the first 24 hours after they are published when the metadata is most likely to change. It is possible to notify our servers when posts get updated. To do that, submit a POST request to the following address, appending the URL of the post to the end of it. The <API_KEY> should be replaced with your respective Site ID (apikey), and <API_SECRET> with the token accessible to the account administrators via API Settings.

https://dash.parsely.com/<API_KEY>/ping_crawl?secret=<API_SECRET>&url=<URL>

If an article is changed after that period, it's not apparent how long it would be before it gets recrawled.

WordPress does not have built-in Webhook support (though there exist plugins to add support). However, it does have built in support for running a function when various actions occur. Should we consider adding such support?

A potential (untested, somewhat pseudo-coded) approach to this might look like:

function wp_parsely_trigger_page_crawl( $post_id ) {
  if ( $POST_IS_NEWER_THAN_THRESHOLD ) {
     return;
  }
  $permalink = get_post_permalink( $post );
  wp_remote_post( get_ping_crawl_url( $permalink ), [ 'blocking' => false ] );
}
add_action( 'post_updated', 'wp_parsely_trigger_page_crawl' );

We should probably add a filter that lets users at least opt out of this behavior (if not opt in)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with WordPress's post_updated action and the Parse.ly trigger-crawl API details linked in the issue. Determine the appropriate update-age threshold, permalink handling, non-blocking request behavior, and an opt-out filter; done means supported post actions trigger the crawl request without forcing it on users who disable the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, wordpress
Domain
analytics, api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.