apache / apache/seatunnel

[Feature][transform-v2] Vector Functions Support in SQL Transform

Open
#9,782 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 17h
Merged PRs (30d)
210

Description

### Search before asking

- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.

### Description

Add comprehensive vector processing capabilities to SeaTunnel's SQL Transform engine, enabling users to perform vector operations directly within SQL queries for machine learning and AI workloads.

### Core Vector Functions

1. **VECTOR_REDUCE** - Vector dimensionality reduction with multiple algorithms
2. **VECTOR_NORMALIZE** - Normalize vector to unit length

### Supported Reduction Methods

- **TRUNCATE** - Simple truncation: keep first N dimensions
- **RANDOM_PROJECTION** - Gaussian random projection
- **SPARSE_RANDOM_PROJECTION** - Sparse random projection for memory efficiency

### Usage Scenario

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating SeaTunnel's SQL Transform engine and its existing function registration and vector-related handling. Review how transform functions are defined and tested, then define support for VECTOR_REDUCE with the listed reduction methods and VECTOR_NORMALIZE; done means these operations work directly in SQL queries with coverage for each method.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
ai, data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.