apache / apache/logging-flume

Accessing Flume logs on the Windows 11 platform

Open
#422 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.6k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
7

Description

My log configuration file:

a1.sources = r1
a1.channels = c1
a1.sinks = k1
# Describe/configure the source
a1.sources.r1.type = TAILDIR
#断点续传
#a1.sources.r1.positionFile = D:\\flumedev\\tail_dir.json
# 多目录监控
a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = D:\\flumedev\\source\\test2\\asdfile.log
a1.sources.r1.filegroups.f2 = D:\\flumedev\\source\\test1\\test.log

# channal
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Describe the sink
a1.sinks.k1.type = logger

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

My startup Flume command:
D:\apache-flume-1.11.0-bin>bin\flume-ng agent --conf conf --conf-file D:\\flumedev\\flume-taildir-hdfs.conf --name a1

The error logs I received:
17 七月 2024 00:00:01,709 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:01,709 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:01,712 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:01,712 ERROR [lifecycleSupervisor-1-4] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:04,724 ERROR [lifecycleSupervisor-1-3] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:07,728 ERROR [lifecycleSupervisor-1-8] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]

Contributor guide

Open the contributing guide

Research direction

Start with org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode at line 284 and reproduce the failure using the provided Windows 11 configuration and startup command. Check the Taildir source startup path and verify completion when the configured log files can be monitored without the “View 'unix' not available” exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.