tree-sitter / tree-sitter/java-tree-sitter

Add access to the source bytes of the Tree

Open
#180 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
141
Forks
34
Avg merge
14d 2h
Merged PRs (30d)
2

Description

Currently when you parse some source code, you can only provide it as a String. The Parser#parse function then converts it to a byte[] before it is passed to tree-sitter. Also, the Tree class only exposes the underlying source code as a String. It stores the content as a byte[]. But every call to Tree#getText converts it to a String.

Those conversion create computational overhead as Java stores its string in UTF-16 while the input encoding for the source code in the byte array can be UTF-8.

In my use case, I parse some text and then want to edit it. I check for some nodes and remove their contents from the text (e.g. by replacing them with whitespace). To avoid those conversions, I currently need to store the input text two times (once in the Tree instance and once separately for the byte level editing). If one could access the byte[] of the tree this could be avoided. I understand, that we do not want the internal byte[] to be edited. Thus one could also expose it as a read only ByteBuffer using ByteBuffer#asReadOnlyBuffer

I proposed a similar change already in https://github.com/tree-sitter/java-tree-sitter/pull/65 but it was closed. In fact, I've been maintaining a private fork of this repo since then, to have the functionality available. I'd really like to have this possibility in the official bindings as I think it could also benefit others.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading Parser#parse and Tree#getText to trace how source bytes are converted and retained, then review the approach in the previously closed pull request #65. Define an access path for the stored bytes that avoids copying or String conversion while preventing edits to internal data; done means callers can use the tree’s source bytes safely without duplicate storage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.