qdrant / qdrant/qdrant-js

UND_ERR_CONNECT_TIMEOUT after multiple upsert requests, no error messages

Open
#92 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
464
Forks
46
PR merge metrics
No merged PRs in 30d

Description

I'm working on a webserver that let's users upload large numbers of documents and I've run into a bottleneck, the js server creates async requests to the qdrant docker container, most of them work, but it seems to get overwhelmed when the client continues to send file upload requests. I've also noticed that this seems to lock up other system resources like mongodb connections. Our server is able to process and store the files just fine, the source of the error is strictly the qdrant .upsert(...) call:

TypeError: fetch failed
    at node:internal/deps/undici/undici:12500:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async fetchJson (/home/torrin/Repos/Equator/ai-app/node_modules/@qdrant/openapi-typescript-fetch/dist/cjs/fetcher.js:135:22)
    at async /home/torrin/Repos/Equator/ai-app/node_modules/@qdrant/js-client-rest/dist/cjs/api-client.js:46:26
    at async handler (/home/torrin/Repos/Equator/ai-app/node_modules/@qdrant/openapi-typescript-fetch/dist/cjs/fetcher.js:156:16)
    at async /home/torrin/Repos/Equator/ai-app/node_modules/@qdrant/js-client-rest/dist/cjs/api-client.js:32:24
    at async handler (/home/torrin/Repos/Equator/ai-app/node_modules/@qdrant/openapi-typescript-fetch/dist/cjs/fetcher.js:156:16)
    at async fetchUrl (/home/torrin/Repos/Equator/ai-app/node_modules/@qdrant/openapi-typescript-fetch/dist/cjs/fetcher.js:162:22)
    at async Object.fun [as upsertPoints] (/home/torrin/Repos/Equator/ai-app/node_modules/@qdrant/openapi-typescript-fetch/dist/cjs/fetcher.js:168:20)
    at async QdrantClient.upsert (/home/torrin/Repos/Equator/ai-app/node_modules/@qdrant/js-client-rest/dist/cjs/qdrant-client.js:553:26)
    at (backend/module/ai/Vector.js:132:22)
    at (backend/module/ai/Vector.js:178:12)
    at (backend/module/util/Upload.js:44:21)
    at (backend/module/account/Login.js:75:13) {
  [cause]: ConnectTimeoutError: Connect Timeout Error
      at onConnectTimeout (/home/torrin/Repos/Equator/ai-app/node_modules/undici/lib/core/connect.js:186:24)
      at /home/torrin/Repos/Equator/ai-app/node_modules/undici/lib/core/connect.js:133:46
      at Immediate._onImmediate (/home/torrin/Repos/Equator/ai-app/node_modules/undici/lib/core/connect.js:174:9)
      at process.processImmediate (node:internal/timers:478:21) {
    code: 'UND_ERR_CONNECT_TIMEOUT'
  }
}

The code I'm using looks something like this:

	const create = async ({ text, fileID, user }) => {
		text = text.replace(/\s+/g, ' ').trim();
		if (!text) throw new Error('File does not contain valid text.');

		const { embeddings, chunks } = await getTextEmbeddings({ text, user });

		const points = embeddings.map((embedding, i) => ({
			id: uuidv4(),
			vector: embedding,
			payload: {
				file_id: fileID,
				chunk_id: i,
				text: chunks[i],
				creator: user
			}
		}));

		const batchPoints = (points, maxSize) => {
			const batches = [];
			let currentBatch = [];
			let currentSize = 0;

			for (const point of points) {
				const pointSize = Buffer.byteLength(JSON.stringify(point), 'utf8'); // Estimate size
				if (currentSize + pointSize > maxSize && currentBatch.length > 0) {
					batches.push(currentBatch);
					currentBatch = [];
					currentSize = 0;
				}
				currentBatch.push(point);
				currentSize += pointSize;
			}

			if (currentBatch.length > 0) batches.push(currentBatch);
			return batches;
		};

		const batches = batchPoints(points, MAX_PAYLOAD_SIZE);

		try {
			for (const batch of batches) {
				await qdrantClient.upsert(COLLECTION_NAME, { points: batch  });
			}
		} catch (e) {
			console.error(e);
		}
	};

Everything up until the upsert call works as expected, chunking and embedding the incoming text. The error happens a large number of requests take place to qdrant. Which seems odd since it's not a crazy number of requests imo: 38 files with a combined size of 300mb.

If this is a fundamental limit with qdrant then I'm a bit concerned, and might have to resort to a queue system, my hope is there is something very wrong with my setup.

Here is the config file, copied it mostly from the docs:

log_level: INFO

storage:
  # Path to store all the data
  storage_path: /qdrant/storage

  # Where to store snapshots
  snapshots_path: /qdrant/storage/snapshots

  snapshots_config:
    # "local" or "s3" - where to store snapshots
    snapshots_storage: local

  # Where to store temporary files
  # If null, temporary snapshot are stored in: storage/snapshots_temp/
  temp_path: null

  # If true - point's payload will not be stored in memory.
  # It will be read from the disk every time it is requested.
  on_disk_payload: true

  # Maximum number of concurrent updates to shard replicas
  update_concurrency: null

  # Write-ahead-log related configuration
  wal:
    wal_capacity_mb: 32
    wal_segments_ahead: 0

  node_type: "Normal"

  performance:
    max_search_threads: 0
    max_optimization_threads: 0
    optimizer_cpu_budget: 0
    update_rate_limit: null

  optimizers:
    deleted_threshold: 0.2
    vacuum_min_vector_number: 1000
    default_segment_number: 0
    max_segment_size_kb: null
    memmap_threshold_kb: 1000
    indexing_threshold_kb: 20000
    flush_interval_sec: 5
    max_optimization_threads: null

  hnsw_index:
    m: 16
    ef_construct: 100
    full_scan_threshold_kb: 10000
    max_indexing_threads: 0
    on_disk: true

service:
  max_request_size_mb: 32
  max_workers: 0
  host: 0.0.0.0
  http_port: 6333
  grpc_port: 6334
  enable_cors: true
  enable_tls: false

telemetry_disabled: false

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the qdrantClient.upsert call in backend/module/ai/Vector.js and trace how it is invoked from Upload.js and Login.js. Reproduce the repeated requests using the shown Docker YAML and inspect the UND_ERR_CONNECT_TIMEOUT stack trace; done means identifying whether the SDK usage or configuration causes the timeout and documenting a confirmed resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, javascript, node.js, typescript
Domain
api, backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.