[codegen] Add fluss-codegen module for runtime code generation
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
Runtime code generation is essential for achieving high performance in data processing systems. Instead of using reflection or generic implementations, generated code can:
1. **Eliminate virtual method dispatch** - Direct field access and type-specific operations
2. **Enable JIT optimization** - Generated code can be better optimized by the JVM
3. **Reduce boxing/unboxing overhead** - Type-specific code avoids primitive boxing
4. **Support complex type comparisons** - Nested types (arrays, maps, rows) require specialized comparison logic
The initial use case is `RecordEqualiser` for comparing `InternalRow` instances, which is needed for:
- Change data capture (CDC) deduplication
- Aggregation state management
- Primary key table updates
### Solution
### Core Framework
| Component | Description |
|-----------|-------------|
| `CodeGeneratorContext` | Manages reusable code fragments, member variables, and class-level declarations |
| `JavaCodeBuilder` | Type-safe builder for constructing Java source code with fluent API |
| `CompileUtils` | Compiles generated source code using Janino with LRU caching |
| `GeneratedClass` | Wrapper holding generated source code and compiled class |
| `CodeGenException` | Exception type for code generation failures |
### Type-Safe API
- `Modifier` enum: PUBLIC, PRIVATE, PROTECTED, STATIC, FINAL, ABSTRACT, SYNCHRONIZED, VOLATILE, TRANSIENT
- `PrimitiveType` enum: BOOLEAN, BYTE, CHAR, SHORT, INT, LONG, FLOAT, DOUBLE, VOID
- `Param` class: Type-safe method parameter representation
- Helper methods: `mods()`, `params()`, `typeOf()`, `arrayOf()`
### Code Generators
| Generator | Output Interface | Description |
|-----------|------------------|-------------|
| `EqualiserCodeGenerator` | `RecordEqualiser` | Generates code for comparing two `InternalRow` instances |
### Supported Data Types
The `EqualiserCodeGenerator` supports all Fluss data types:
- **Primitive types**: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE
- **String types**: CHAR, VARCHAR, STRING
- **Binary types**: BINARY, VARBINARY, BYTES
- **Temporal types**: DATE, TIME, TIMESTAMP, TIMESTAMP_LTZ
- **Numeric types**: DECIMAL (with precision/scale)
- **Complex types**: ARRAY, MAP, ROW (nested)
### Features
- Field projection support for partial row comparison
- Compiled class caching with configurable cache size
- Janino dependency shaded to `org.apache.fluss.shaded.org.codehaus.janino` to avoid classpath conflicts
- Comprehensive Javadoc and package-info documentation
### Anything else?
_No response_
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the proposed CodeGeneratorContext, JavaCodeBuilder, CompileUtils, GeneratedClass, and EqualiserCodeGenerator components and their RecordEqualiser use case. Done means the fluss-codegen module provides the listed runtime-generation framework, supports the stated data types and field projection, includes Janino caching and shading, and has the described documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, performance, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100