Hope to Support Multi-Task / Multi-Dataset Training for Embedding Models with Dataset-Specific Negative Sampling

Checklist / 检查清单

• I have searched existing issues, and this is a new feature request. / 我已经搜索过现有的 issues，确认这是一个新的 Feature Request。

Feature Request Description / Feature Request 描述

🚀 Overview

We would like to request support for multi-task / multi-dataset joint training for embedding models, with proper handling of dataset-specific negative sampling strategies.

---

🎯 Motivation

In real-world embedding training scenarios, it is common to:

• Train on multiple datasets simultaneously
• Mix tasks such as retrieval, classification, etc.
• Assign different sampling and negative strategies per dataset

However, the current training logic assumes:

• A single dataset per run, or
• Globally shared negative sampling configuration (e.g., via environment variables INFONCE_USE_BATCH, INFONCE_HARD_NEGATIVES, ...)

This makes it difficult to perform principled multi-task training where each dataset may require different training semantics.

---

🔍 Core Requirements

1️⃣ Dataset-Consistent Global Batch

For embedding models trained with in-batch negatives, it is crucial that:

Each global batch contains samples from only one sub-dataset.

Reasoning:

• In-batch negatives assume semantic comparability within a batch.
• Mixing heterogeneous datasets in one batch may introduce invalid or noisy negatives.
• Many retrieval-style objectives rely on same-task negative semantics.

Therefore, batch construction must guarantee dataset-level isolation.

---

2️⃣ Dataset-Specific Negative Sampling Configuration

Different datasets may require different Negatives Sampling configurations, for example:

Dataset	In-Batch Negatives	Expanded Negatives	#Expanded
Dataset A	Enabled	Disabled	0
Dataset B	Disabled	Enabled	8
Dataset C	Enabled	Enabled	16

Currently, negative strategy configuration is controlled globally (e.g., via environment variables INFONCE_USE_BATCH, INFONCE_HARD_NEGATIVES,), which prevents flexible multi-task training.

We propose:

• Allow per-dataset configuration of:
  • use_in_batch_negatives
  • use_expanded_negatives
  • num_expanded_negatives
• These configurations should be attached to the dataset definition rather than globally shared.

Pull Request / Pull Request 信息

No response

No files or tests are named. Start by tracing the current single-dataset training path, dataset definitions, batch construction, and the global INFONCE_* configuration; then determine where per-dataset settings can be represented. Done means multi-task batches remain dataset-consistent and each dataset applies its own negative-sampling configuration.