hackforla / hackforla/311-data
ER: Create data model from Huggingface without loading data
- Dominant language
- JavaScript
- Stars
- 73
- Forks
- 74
- PR merge metrics
- No merged PRs in 30d
Description
### Emergent Requirement - Problem Description
- We want to create the data model from Hugging Face without loading data by querying the parquet file. Currently, we can only obtain the data model while loading the entire parquet file, which is inflexible.
### Relevant Issue(s)
- #1891
### Date discovered
- 2025-02-14
### Did this require a temporary workaround? If yes, what was it?
- N
### Who was involved
- @DrAcula27 @ryanfchase
### What happens if this is not addressed
- This is another tool available to us that we are exploring.
### Resources
- DuckDB Docs → https://duckdb.org/docs/
- DESCRIBE statement → https://duckdb.org/docs/sql/statements/describe.html
- Parquet Format Docs → https://parquet.apache.org/
- DuckDB GitHub Discussions (for real-world usage examples) → https://github.com/duckdb/duckdb/discussions
### Recommended Action Items
### Potential Solutions
```javascript
/**
* What if we want to dynamically define the table structure without loading the data?
* We can modify the try-catch to read the schema from the parquet file to create the table:
*/
try {
// Get schema without loading data
const schemaQuery = `DESCRIBE "${datasetFileName}"`;
const schemaResult = await conn.query(schemaQuery);
const schemaLines = schemaResult.toString().split('\n');
// Generate CREATE TABLE statement based on schema
const columnDefinitions = schemaLines
.map(line => line.trim().split(/\s+/))
.filter(parts => parts.length >= 2) // Ensure valid schema lines
.map(parts => `${parts[0]} ${parts[1]}`) // Convert to SQL column format
.join(', ');
const createSQL = `CREATE TABLE IF NOT EXISTS requests (${columnDefinitions})`;
await conn.query(createSQL);
const endTime = performance.now();
console.log(`Table creation time: ${Math.floor(endTime - startTime)} ms.`);
} catch (error) {
console.error("Error creating table:", error);
} finally {
setState({ isTableLoading: false });
}
```
Contributor guide
Assessment
This issue has not been assessed yet.