Accessing Flume logs on the Windows 11 platform
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 7
Description
My log configuration file:
a1.sources = r1
a1.channels = c1
a1.sinks = k1
# Describe/configure the source
a1.sources.r1.type = TAILDIR
#断点续传
#a1.sources.r1.positionFile = D:\\flumedev\\tail_dir.json
# 多目录监控
a1.sources.r1.filegroups = f1 f2
a1.sources.r1.filegroups.f1 = D:\\flumedev\\source\\test2\\asdfile.log
a1.sources.r1.filegroups.f2 = D:\\flumedev\\source\\test1\\test.log
# channal
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Describe the sink
a1.sinks.k1.type = logger
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
My startup Flume command:
D:\apache-flume-1.11.0-bin>bin\flume-ng agent --conf conf --conf-file D:\\flumedev\\flume-taildir-hdfs.conf --name a1
The error logs I received:
17 七月 2024 00:00:01,709 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:01,709 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:01,712 INFO [lifecycleSupervisor-1-4] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:01,712 ERROR [lifecycleSupervisor-1-4] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:04,724 INFO [lifecycleSupervisor-1-3] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:04,724 ERROR [lifecycleSupervisor-1-3] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.TaildirSource.start:94) - r1 TaildirSource source starting with directory: {f1=D:\flumedev\source\test2\asdfile.log, f2=D:\flumedev\source\test1\test.log}
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:84) - taildirCache: [{filegroup='f1', filePattern='D:\flumedev\source\test2\asdfile.log', cached=true}, {filegroup='f2', filePattern='D:\flumedev\source\test1\test.log', cached=true}]
17 七月 2024 00:00:07,728 INFO [lifecycleSupervisor-1-8] (org.apache.flume.source.taildir.ReliableTaildirEventReader.:85) - headerTable: {}
17 七月 2024 00:00:07,728 ERROR [lifecycleSupervisor-1-8] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner: { source:Taildir source: { positionFile: C:/Users/liu970/.flume/taildir_position.json, skipToEnd: false, byteOffsetHeader: false, idleTimeout: 120000, writePosInterval: 3000 } counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.UnsupportedOperationException: View 'unix' not available
at sun.nio.fs.AbstractFileSystemProvider.readAttributes(AbstractFileSystemProvider.java:91) ~[?:1.8.0_43]
at java.nio.file.Files.readAttributes(Files.java:1964) ~[?:1.8.0_43]
at java.nio.file.Files.getAttribute(Files.java:1869) ~[?:1.8.0_43]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode(ReliableTaildirEventReader.java:284) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.updateTailFiles(ReliableTaildirEventReader.java:248) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:93) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader.(ReliableTaildirEventReader.java:49) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.ReliableTaildirEventReader$Builder.build(ReliableTaildirEventReader.java:355) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.taildir.TaildirSource.start(TaildirSource.java:105) ~[flume-taildir-source-1.11.0.jar:1.11.0]
at org.apache.flume.source.PollableSourceRunner.start(PollableSourceRunner.java:71) ~[flume-ng-core-1.11.0.jar:1.11.0]
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:249) ~[flume-ng-core-1.11.0.jar:1.11.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_43]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_43]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_43]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_43]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_43]
Contributor guide
Research direction
Start with org.apache.flume.source.taildir.ReliableTaildirEventReader.getInode at line 284 and reproduce the failure using the provided Windows 11 configuration and startup command. Check the Taildir source startup path and verify completion when the configured log files can be monitored without the “View 'unix' not available” exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100