hackforla / hackforla/311-data

ER: Create data model from Huggingface without loading data

Open
#1,928 1 comment 0 reactions 0 assignees View on GitHub
Complexity: Small Emergent Request Feature: Code Health Role: Frontend size: 0.25pt
Dominant language
JavaScript
Stars
73
Forks
74
PR merge metrics
No merged PRs in 30d

Description

### Emergent Requirement - Problem Description
- We want to create the data model from Hugging Face without loading data by querying the parquet file. Currently, we can only obtain the data model while loading the entire parquet file, which is inflexible.

### Relevant Issue(s)
- #1891

### Date discovered
- 2025-02-14

### Did this require a temporary workaround? If yes, what was it?
- N

### Who was involved
- @DrAcula27 @ryanfchase

### What happens if this is not addressed
- This is another tool available to us that we are exploring.

### Resources
- DuckDB Docs → https://duckdb.org/docs/
- DESCRIBE statement → https://duckdb.org/docs/sql/statements/describe.html
- Parquet Format Docs → https://parquet.apache.org/
- DuckDB GitHub Discussions (for real-world usage examples) → https://github.com/duckdb/duckdb/discussions

### Recommended Action Items

### Potential Solutions

```javascript
/**
* What if we want to dynamically define the table structure without loading the data?
* We can modify the try-catch to read the schema from the parquet file to create the table:
*/
try {
// Get schema without loading data
const schemaQuery = `DESCRIBE "${datasetFileName}"`;
const schemaResult = await conn.query(schemaQuery);
const schemaLines = schemaResult.toString().split('\n');

// Generate CREATE TABLE statement based on schema
const columnDefinitions = schemaLines
.map(line => line.trim().split(/\s+/))
.filter(parts => parts.length >= 2) // Ensure valid schema lines
.map(parts => `${parts[0]} ${parts[1]}`) // Convert to SQL column format
.join(', ');

const createSQL = `CREATE TABLE IF NOT EXISTS requests (${columnDefinitions})`;

await conn.query(createSQL);
const endTime = performance.now();
console.log(`Table creation time: ${Math.floor(endTime - startTime)} ms.`);
} catch (error) {
console.error("Error creating table:", error);
} finally {
setState({ isTableLoading: false });
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.