IQSS / IQSS/dataverse

text/tsv is not a valid media type (text/tab-separated-values is)

Open
#11,505 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: Bug
Dominant language
Java
Stars
1.1k
Forks
564
Avg merge
2d 2h
Merged PRs (30d)
29

Description

Before #4854 we only had text/tab-separated-values and in retrospect, I don't think we ever should have added text/tsv since it's not listed https://www.iana.org/assignments/media-types/media-types.xhtml

Here's where text/tsv is used in the code base, as of d8a55a9

src/test/java/edu/harvard/iq/dataverse/api/FileTypeDetectionIT.java
106:                .body("data.files[0].dataFile.contentType", equalTo("text/tsv"))

src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java
141:        STATISTICAL_FILE_EXTENSION.put("tsv", "text/tsv");
154:    public static final String MIME_TYPE_TSV     = "text/tsv";
843:        } else if (fileType.equalsIgnoreCase("text/tsv") || fileType.equalsIgnoreCase("text/tab-separated-values")) {

src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java
496:                        // "text/tsv" should be used instead: 

src/main/java/edu/harvard/iq/dataverse/dataaccess/StoredOriginalFile.java
112:        } else if (fileType.equalsIgnoreCase("text/tsv") || fileType.equalsIgnoreCase("text/tab-separated-values")) {

src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java
87:    private static final String MIME_TYPE_TSV   = "text/tsv";

src/main/java/META-INF/mime.types
10:text/tsv tab TAB tsv TSV

src/main/java/propertyFiles/MimeTypeDisplay.properties
87:text/tsv=Tab-Separated Values

src/main/java/propertyFiles/MimeTypeFacets.properties
85:text/tsv=Data

In particular, we should look at IngestServiceBean.java. Here's a bit more of the code added in #6517:

} else if (FileUtil.MIME_TYPE_INGESTED_FILE.equals(dataFile.getContentType())) {
    // Make sure no *uningested* tab-delimited files are saved with the type "text/tab-separated-values"!
    // "text/tsv" should be used instead: 
    dataFile.setContentType(FileUtil.MIME_TYPE_TSV);
}

Also related:

  • #4866
  • #4867

Some context for this issue is that I was considering making a pull request to add text/tsv as an alternative to text/tab-separated-values at https://github.com/mlcommons/croissant/blob/v1.0.17/python/mlcroissant/mlcroissant/_src/operation_graph/operations/read.py#L126 but again, since text/tsv doesn't seem to be a legit format, I don't think I should.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the listed usages in FileUtil.java, IngestServiceBean.java, StoredOriginalFile.java, DataFileServiceBean.java, META-INF/mime.types, and the MIME type properties files. Run FileTypeDetectionIT.java and inspect the related references #4866, #4867, and #6517. Done means the TSV media type handling and test expectations consistently use the valid type without breaking ingestion behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.