trekhleb / trekhleb/trekhleb.github.io

Deep-dive resource proposal for the web crawler sketch

Open Beginner friendly
#103 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
239
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Hi Oleksii - I reviewed the System Design Sketches page and the source repository before proposing a change. The crawler sketch is compact but covers the right backbone: requirements and non-functional constraints, URL frontier, DNS and fetching, extraction, storage, deduplication, and politeness.

Would a crawler-specific interview deep dive be useful as optional further reading beneath that sketch?

PracHub's Web Crawler System Design Interview guide covers frontier scheduling, per-host politeness, robots.txt, duplicate URL and content detection, retries, failure isolation, and the trade-offs candidates should explain aloud.
https://prachub.com/resources/web-crawler-system-design

The sketch would remain the fast visual overview; the proposed link would only serve readers who want to practice interviewer follow-ups. I would make the smallest possible edit in the existing format and am checking scope before opening a PR. No reciprocal link is expected.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the System Design Sketches page and its source repository, locate the web crawler sketch, and inspect how optional further-reading links are formatted. Add the PracHub web crawler guide beneath that sketch using the existing format, then verify that the link appears correctly and the sketch remains a concise visual overview.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.