taosdata / taosdata/TDengine

It is not possible to set 'smlAutoChildTableNameDelimiter' nor 'smlChildTableName' when trying to insert schemaless OpenTSDBJsonFormatProtocol using the nodeJs connector

Open
#28,657 1 comment 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C
Stars
25.1k
Forks
5k
Avg merge
4d 59m
Merged PRs (30d)
7

Description

Bug Description
When setting smlAutoChildTableNameDelimiter to any value and then trying to insert schemaless data using the InfluxDB Line Protocol via the @tdengine/websocket node package, it results in an error. Using smlChildTableName has no effect, and the table names are the same as when omitting to set this value.

All of the above works when using InfluxDB Line Protocol instead of OpenTSDB JSON protocol but i actually would prefer using the OpenTSDB JSON protocol

I'm using the kubernetes Deployment from https://docs.tdengine.com/operation/deployment/#kubernetes. This has a few problems to start when using as described with tdengine 3.3.3.0 which i could resolve. I had to replace the taoskeeper.toml and set the TAOS_ADAPTER_NODE_EXPORTER_URLS env variable as both had at buildkitsandbox somewhere instead of 127.0.0.1.

To Reproduce
Steps to reproduce the behavior:

  1. set smlAutoChildTableNameDelimiter to any value i tried it with '_'
  2. run the example https://docs.tdengine.com/reference/connectors/node/#schemaless-writing removing/commenting out the telnet and influxparts
  3. The creation of the db works but writing the data fails

  1. set smlChildTableName to 'tname'
  2. run the example https://docs.tdengine.com/reference/connectors/node/#schemaless-writing removing/commenting out the telnet and influxparts and edit the influxdata var to
    let jsonData = ["{\"metric\": \"metric_json\",\"tname\":\"some_tname\",\"timestamp\": 1626846400,\"value\": 10.3, \"tags\": {\"groupid\": 2, \"location\": \"California.SanFrancisco\", \"id\": \"d1001\"}}"];
  3. the generated table does not have the name some_tname as defined above but the autogenerated name

Expected Behavior
It should work as described in the documentation https://docs.tdengine.com/reference/schemaless/#main-processing-logic-for-schemaless-writing, which is the case with the InfluxDB Line Protocol

Screenshots
Error when using smlAutoChildTableNameDelimiter

schemalessInsertError

Environment
statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: tdengine
  namespace: test
  labels:
    app: taosd
spec:
  replicas: 1
  selector:
    matchLabels:
      app: taosd
  template:
    metadata:
      labels:
        app: taosd
    spec:
      containers:
        - name: tdengine
          image: tdengine/tdengine:3.3.3.0
          ports:
            - name: tcp0
              containerPort: 6030
              protocol: TCP
            - name: tcp1
              containerPort: 6041
              hostPort: 36041
              protocol: TCP
          envFrom:
            - configMapRef:
                name: tdengine-config
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: SERVICE_NAME
              value: tdengine
            - name: STS_NAME
              value: tdengine
            - name: STS_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: TAOS_SERVER_PORT
              value: '6030'
            - name: TAOS_FIRST_EP
              value: $(STS_NAME)-0.$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local:$(TAOS_SERVER_PORT)
            - name: TAOS_FQDN
              value: $(POD_NAME).$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local
            - name: TAOS_MONITOR_FQDN
              value: 'NONE'
            - name: TAOS_SML_AUTO_CHILD_TABLE_NAME_DELIMITER
              value: '_'
            #            - name: TAOS_SML_CHILD_TABLE_NAME
            #              value: 'tname'
            - name: TAOS_ADAPTER_NODE_EXPORTER_URLS
              value: '["http://localhost:9100"]'
          resources: { }
          volumeMounts:
            - name: tdengine-taosdata
              mountPath: /var/lib/taos
            - name: tdengine-taoslog
              mountPath: /var/log/taos
            - name: taos-keeper-toml
              mountPath: /etc/taos/taoskeeper.toml
              subPath: taoskeeper.toml
              readOnly: true
          livenessProbe:
            exec:
              command:
                - taos-check
            initialDelaySeconds: 15
            timeoutSeconds: 1
            periodSeconds: 20
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            exec:
              command:
                - taos-check
            initialDelaySeconds: 5
            timeoutSeconds: 5000
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      volumes:
        - name: taos-keeper-toml
          configMap:
            name: taos-keeper-toml-cm
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: tdengine-taosdata
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        volumeMode: Filesystem
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: tdengine-taoslog
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
  serviceName: tdengine
  updateStrategy:
    type: RollingUpdate
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: tdengine
  namespace: test
spec:
  ports:
    - name: tcp0
      protocol: TCP
      port: 6030
    - name: tcp1
      protocol: TCP
      port: 36041
      targetPort: 6041

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: tdengine-config
  namespace: test
data:
  CLUSTER: '1'
  TAOS_REPLICA: '1'

taos-keeper-toml-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: taos-keeper-toml-cm
  namespace: test
data:
  taoskeeper.toml: |
    instanceId = 64
    
    # Listen port, default is 6043
    port = 6043
    
    # go pool size
    gopoolsize = 50000
    
    # interval for metrics
    RotationInterval = "15s"
    
    [tdengine]
    host = "127.0.0.1"
    port = 6041
    username = "root"
    password = "taosdata"
    usessl = false
    
    [metrics]
    # metrics prefix in metrics names.
    prefix = "taos"
    
    # export some tables that are not super table
    tables = []
    
    # database for storing metrics data
    [metrics.database]
    name = "log"
    # database options for db storing metrics data
    [metrics.database.options]
    vgroups = 1
    buffer = 64
    keep = 90
    cachemodel = "both"
    
    [environment]
    # Whether running in cgroup.
    incgroup = false
    
    [log]
    # The directory where log files are stored.
    # path = "/var/log/taos"
    level = "info"
    # Number of log file rotations before deletion.
    rotationCount = 30
    # The number of days to retain log files.
    keepDays = 30
    # The maximum size of a log file before rotation.
    rotationSize = "1GB"
    # If set to true, log files will be compressed.
    compress = false
    # Minimum disk space to reserve. Log files will not be written if disk space falls below this limit.
    reservedDiskSize = "1GB"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.