apache / apache/hop

[Feature Request]: Request for Built-in CPython Support to Enhance External Library Compatibility in Apache Hop

Open
#3,660 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage Other P3
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 33m
Merged PRs (30d)
204

Description

### What would you like to happen?

## Request for Built-in CPython Support to Enhance External Library Compatibility in Apache Hop

### Summary
The current external Python plugin (Script) in Apache Hop faces challenges, particularly in efficiently utilizing external libraries (does not work with it). Integrating built-in CPython support would resolve these issues, leveraging CPython's robust handling of external libraries for advanced data processing capabilities. Pentaho, e.g, has a built in CPython.

### Issue Description
The external Python plugin available in Apache Hop does not seamlessly support the use of external libraries, which is crucial for complex data transformations and analysis. This limitation hampers the execution of advanced Python scripts within Apache Hop workflows, affecting performance and utility.

### Proposal
I propose the integration of built-in CPython support in Apache Hop. This would enable direct execution of Python scripts, including those requiring external libraries, within Apache Hop's environment. The main advantages include:

- Improved performance and reliability.
- Expanded data processing and analysis capabilities through direct access to Python's extensive library ecosystem.
- A streamlined workflow for Python-based data manipulation directly within Apache Hop.

### Expected Outcome
By addressing the current limitations with a built-in CPython solution, Apache Hop can enhance its attractiveness to a broader audience, including data scientists and engineers who rely heavily on Python's ecosystem for data processing tasks.

### Conclusion
Incorporating built-in CPython support will significantly enhance Apache Hop's functionality and user experience, particularly for advanced data processing and analysis tasks. I look forward to the community's input and the potential development of this feature.

### Issue Priority

Priority: 3

### Issue Component

Component: Other

Contributor guide

Open the contributing guide

Research direction

Start by locating Apache Hop's current external Python plugin (Script) and reviewing how it launches Python and handles external libraries. Define the CPython integration scope and verify that scripts requiring external libraries execute reliably within Hop workflows, with performance and compatibility meeting the proposal's expected outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.