newrelic / newrelic/newrelic-java-agent

Agent don't work with reactor-kafka (automatic or manual instrumentation)

Open
#1,090 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

8 feature request large
Dominant language
Java
Stars
240
Forks
170
Avg merge
3d 2h
Merged PRs (30d)
21

Description

Description

I have app that works with spring boot and webflux and the kafka consumers using reactor-kafka, but any trace, transaction have sending to newrelic. I tried dowgrade all versions (spring boot, agent, webflux...) but it didn't work. All web transactions works fine (or maybe, http integrations or reactive mongodb is not tracing too) but the custom have no trace there,

Follow versions that I using:
webflux: 2.2.0.RELEASE
agent: 7.11.1
reactor-kafka: 1.3.14
reactor-netty: 0.9.25.RELEASE

For understand the problem, follow de code that configure my consumer, considering that my consumer implement a interface KafkaReactiveConsumer:

@Configuration
class ReactorKafkaConsumer(
    private val receiver: ReceiverOptions<String, Any>,
    private val tracingKafkaConsumerFactory: TracingKafkaConsumerFactory,
    private val consumers: List<KafkaReactiveConsumer>
) {

    private val log = KotlinLogging.logger { }
    private val disposables = Disposables.composite()

    @PostConstruct
    fun connect() {
        consumers.map { consumer ->
            disposables.add(
                consumes(consumer).subscribe()
            )
        }
    }

    @PreDestroy
    fun disconnect() {
        this.disposables.dispose()
    }

    fun consumes(consumer: KafkaReactiveConsumer): Flux<Void> {
        val receiverSubscribed = receiver.subscription(consumer.getTopics())
            .addAssignListener { partitions -> log.debug("on partitions assigned {}", partitions) }
            .addRevokeListener { partitions -> log.debug("on partitions revoked {}", partitions) }

        return KafkaReceiver.create(tracingKafkaConsumerFactory, receiverSubscribed).receive()
            .flatMap {
                consumer.consumer(it)
            }
    }
}

Sample of my consumer...


    @Trace(dispatcher = true, metricName = "consumer-document-reference-capture")
    override fun consumer(receive: ReceiverRecord<String, Any>): Mono<Void> {
        return receive.toMono()
            .doOnNext { log.info { "Received: ${it.value()}" } }
            .flatMap { message ->
                val offset = message.receiverOffset()
                message.toMono()
                    .map {
                        objectMapper.readValue<Object>(it.value() as String)
                    }
                    .filterWhen { event -> callService(event) } }
                    .zipWhen {
                        callOtherService(it)
                    }
                    .map {
                        Tuples.of(callOneOtherServiceit.t1.payload), it.t2)
                    }
                    .map { callServiceFromThisClass(it) }
                    .flatMap {
                        callMoreOneService(it)
                            .thenReturn(message)
                    }
                    .flatMap {
                        val event = objectMapper.readValue<Object>(it.value() as String).event
                        callOneLastService.create(event.uuid)
                    }
                    .doOnSuccess {
                        log.debug { "Success to consuming message with id ${receive.key()} ack on message" }
                        offset.acknowledge()
                    }
                    .onErrorResume { t ->
                        log.error(t) { "Consuming messages failed" }
                        offset.acknowledge()
                        Mono.empty()
                    }
            }
    }

And I've got the from agent:

2022-11-22T22:35:10,036-0300 [99365 85] com.newrelic FINEST: Setting transaction name to "consumer-document-reference-capture" for transaction com.newrelic.agent.Transaction@37ff828f using LEGACY scheme
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Added log attribute "timestamp": 1.669.167.310.035
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Added log attribute "logger.fqcn": mu.internal.LocationAwareKLogger
2022-11-22T22:35:10,036-0300 [99365 85] com.newrelic FINEST: tracerFinished: com.newrelic.agent.tracers.OtherRootTracer@17ad6e12 opcode: 176 in transactionActivity com.newrelic.agent.TransactionActivity@0
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Added log attribute "logger.name": com.unico.auto.order.application.usecase.SaveDocumentReference
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Added log attribute "thread.name": kafka-producer-network-thread | auto-order-local
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Added log attribute "level": INFO
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINEST: Added event of type LogEvent
2022-11-22T22:35:10,036-0300 [99365 85] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@37ff828f: Activity com.newrelic.agent.TransactionActivity@0 finished with opcode 176
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,036-0300 [99365 85] com.newrelic FINEST: Tracer Attributes for com.newrelic.agent.tracers.OtherRootTracer@17ad6e12 are {async_context=reactive-kafka-auto-order-local-3, code.namespace=com.samples.test.MyConsumer, code.function=consumer}
2022-11-22T22:35:10,036-0300 [99365 85] com.newrelic FINEST: tx response time set: 308.443
2022-11-22T22:35:10,036-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,036-0300 [99365 85] com.newrelic FINER: Transaction OtherTransaction/Custom/consumer-document-reference-capture for request: consumer-document-reference-capture finished 0ms com.newrelic.agent.Transaction@37ff828f
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,037-0300 [99365 85] com.newrelic FINEST: SlowQueryAggregator: addSlowQueriesFromTransaction: no listener
2022-11-22T22:35:10,037-0300 [99365 209] com.newrelic FINER: Transaction com.newrelic.agent.Transaction@6676b6b1: ignoring link call because there is no started txa to link to: null.
2022-11-22T22:35:10,038-0300 [99365 209] com.newrelic FINEST: Expiring token com.newrelic.agent.TokenImpl@32531767 on transaction com.newrelic.agent.Transaction@6676b6b1
2022-11-22T22:35:10,038-0300 [99365 209] com.newrelic FINEST: Transaction com.newrelic.agent.Transaction@6676b6b1: expired token com.newrelic.agent.TokenImpl@32531767
2022-11-22T22:35:10,038-0300 [99365 85] com.newrelic FINER: Added log attribute "thread.id": 85
2022-11-22T22:35:10,038-0300 [99365 209] com.newrelic FINER: Token has already been expired com.newrelic.agent.TokenImpl@32531767.
2022-11-22T22:35:10,038-0300 [99365 85] com.newrelic FINER: Added log attribute "timestamp": 1.669.167.310.038
2022-11-22T22:35:10,039-0300 [99365 85] com.newrelic FINER: Added log attribute "logger.fqcn": mu.internal.LocationAwareKLogger
2022-11-22T22:35:10,039-0300 [99365 85] com.newrelic FINER: Added log attribute "logger.name": com.unico.auto.order.adapter.kafka.in.DocumentReferenceConsumer
2022-11-22T22:35:10,039-0300 [99365 85] com.newrelic FINER: Added log attribute "thread.name": reactive-kafka-auto-order-local-3
2022-11-22T22:35:10,039-0300 [99365 85] com.newrelic FINER: Added log attribute "level": INFO
2022-11-22T22:35:10,039-0300 [99365 85] com.newrelic FINEST: Added event of type LogEvent
2022-11-22T22:35:10,039-0300 [99365 209] com.newrelic FINER: Transaction null: ignoring link call because transaction already on thread.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository files or tests are identified. Start by reproducing the supplied Reactor Kafka consumer with the listed Spring Boot, agent, Reactor Kafka, and Reactor Netty versions, then compare the agent logs with automatic and manual instrumentation. Done means the consumer's trace and transaction are reported to New Relic.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kotlin, spring-boot
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.