adobe / adobe/spacecat-api-service
API For Web Scraping / Processing
- Dominant language
- JavaScript
- Stars
- 10
- Forks
- 15
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 150
Description
In order to integrate the PoC-style [Content Scraper](https://github.com/adobe/spacecat-content-scraper) and [Content Processor](https://github.com/adobe/spacecat-content-processor) an HTTP API is needed providing the following features:
- trigger an async scraping -> processing task, which will have the content-scraper scrape content off the input URL, store the results and forward the task to the content-processor
- check the status of a triggered task and eventually get the results of the processor stages/handlers
Here is a proposal for amending the HTTP API spec:
```yaml
openapi: 3.0.0
info:
title: Web Scraping and Processing API
version: 1.0.0
paths:
/scrape:
post:
summary: Initiates a web scraping job.
description: Triggers a new scraping job for the given URL and returns a task ID for status polling.
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
url:
type: string
format: uri
description: The URL to be scraped.
required:
- url
examples:
example-1:
value: { "url": "https://example.com" }
responses:
202:
description: Accepted. The scraping job is initiated, and a task ID is returned.
content:
application/json:
schema:
type: object
properties:
taskId:
type: string
description: The unique identifier for the scraping task.
examples:
example-1:
value: { "taskId": "12345" }
400:
description: Bad Request. The URL is invalid or missing.
429:
description: Too Many Requests. Rate limit exceeded.
500:
description: Internal Server Error.
/scrape/{taskId}:
get:
summary: Polls the status and results of a scraping job.
description: Retrieves the status and, if available, the results of a scraping job by task ID.
parameters:
- in: path
name: taskId
required: true
schema:
type: string
description: The unique identifier for the scraping task.
responses:
200:
description: OK. Returns the status of the scraping job and results if completed.
content:
application/json:
schema:
type: object
properties:
status:
type: string
description: The current status of the job ('pending', 'in_progress', 'completed', 'failed').
results:
type: object
properties:
translation:
type: string
description: URL or location of the translation result.
seoKeywords:
type: string
description: URL or location of the SEO keyword extraction result.
sentimentAnalysis:
type: string
description: URL or location of the sentiment analysis result.
required: []
examples:
pending:
value:
status: "pending"
completed:
value:
status: "completed"
results:
translation: "https://results.example.com/translation/12345"
seoKeywords: "https://results.example.com/seo/12345"
sentimentAnalysis: "https://results.example.com/sentiment/12345"
404:
description: Not Found. The task ID does not exist.
429:
description: Too Many Requests. Rate limit exceeded.
500:
description: Internal Server Error.
```
Contributor guide
Assessment
This issue has not been assessed yet.