packagecontrol / packagecontrol/thecrawl

Track file extensions for syntax definitions

Open
#40 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
19
Forks
7
Avg merge
4h 53m
Merged PRs (30d)
14

Description

The rewrite opens up the possibility of a feature I'd been kicking around in my head for a couple years, though it might be too ambitious: I think it would be interesting and useful to index the file extensions that a package claims to provide syntax for.

Benefits
  • PC can suggest packages from inside ST for un-highlighted (or wrongly, badly, etc. highlighted) files.
  • It's another view into the package database, potentially exposing some additional utility for packages.
  • The first time we do this, we may discover a bunch of packages that should have the language syntax tag, but don't.
Drawbacks
  • Requires downloading, parsing, and saving results for packages. This can use a lot of API rate-limit.
    • Mitigation: Maybe running only on language syntax packages with new updates will drop the API / CPU requirements sufficiently. A one-time run of every package to fill in missing tags is probably still needed.
Prior Art
  • This feature does not exist in the current ST PC.
  • VS Code's marketplace does track file extensions that could be highlighted. They don't need to do all the parsing, though, because the manifest file for a package tracks the extensions that it has syntax for. (Then Microsoft somewhat aggressively pushes those packages on people, which we would not want to do.)
Discussion

Taken from Discord

@kaste: I never had problems just searching for the actual "language" and then each of them should have the language syntax label. I often rather know that name (e.g. Clojure) before its file extension.

@michaelblyons:

  • I don't actually trust the language syntax tag. Some packages have syntax definitions but not the tag.
    • For some, we could one-time scan and add the tag to the appropriate Channel repo.
    • But for others, arguably the language is kind of incidental to something else that the package is doing. I don't have strong opinions that we should add a tag if that is the case.
  • There's also the case where it's not immediately apparent what the file extension is for. Granted, you could $favorite_search_engine it and probably comes up with one of those ad-stuffed file-extensions.info type sites that gives you a good guess.
  • And lastly, there's the case where the name of the package might not signal well enough how much it does.
    1. PackageDev: If you find a *.tmLanguage file, you don't know to search for a Sublime configuration editing package because the internet tells you it's a TextMate file.
    2. SSH Config has minimal syntaxes thrown in for known hosts, authorized keys, and a couple other things that don't suggest "Go look for SSH packages" to a user.
Disclaimer

I'm not going to be upset if the amount of work outweighs the benefit.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue proposes indexing file extensions by downloading and parsing package syntax definitions, but it does not name implementation files, tests, or entry points. First map the crawler's package scanning and result-storage flow, then define the package sources, parsing scope, rate-limit strategy, and a way to verify stored extension data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.