IQSS / IQSS/dataverse

Can't convert large .SAV file to .TAB file.

Open
#11,361 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Bug
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

What steps does it take to reproduce the issue?
Uploading an SAV file.

  • Which page(s) does it occurs on?
    List of files

  • What happens?
    On Ingest: tabular ingest was unsuccessful. ingest produced tabular data but failed to save it in the database. Transaction aborted. No further inofmation is available.

  • To whom does it occur (all users, curators, superusers)?
    all users

  • What did you expect to happen?
    Converting the .SAV to .TAB file

In [31]: # Print relevant metadata
...: print("File Encoding:", meta.file_encoding)
...: print("Creation Time:", meta.creation_time)
...: print("Modification Time:", meta.modification_time)
...: print("File Format:", meta.file_format)
...: print("Number of Rows:", meta.number_rows)
...: print("Number of Columns:", meta.number_columns)
...: print("Notes:", meta.notes)

File Encoding: WINDOWS-1252
Creation Time: 2024-06-14 11:03:04
Modification Time: 2024-06-14 11:03:04
File Format: sav/zsav
Number of Rows: 108252
Number of Columns: 691
Notes: []
File Size (bytes): 673496320
Bytes per Cell: 9.003704867663398

Total cells: 108252 rows * 691 columns = 74,802,132.
Bytes per cell: 673,496,320 / 74,802,132 ≈ 9.00.

With 691 columns and 9 bytes/cell, the file might include extensive metadata (e.g., variable labels, value labels) or string-heavy data, potentially overwhelming Dataverse’s ingest process or hitting a database constraint.

Which version of Dataverse are you using?
https://github.com/scholarsportal/dataverse latest

Any related open or closed issues to this bug report?
didn't find

Screenshots:

Image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the List of files upload flow and the tabular ingest/database transaction logs, reproducing the failure with the reported 108252×691 SAV file. Compare the result with a smaller SAV file and trace where the transaction aborts. Done means the file converts and saves successfully, or the limiting condition is identified and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.