internetarchive / internetarchive/heritrix3

Intermittent problems with log rotation

Open
#426 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.3k
Forks
793
Avg merge
1d 8h
Merged PRs (30d)
8

Description

In production, outputting log files to Gluster, we occasionally see problems with log file rotation that lead to checkpoint failure (as mentioned in #392).

```
SEVERE: org.archive.crawler.framework.CheckpointService checkpointFailed Checkpoint failed [Fri May 28 10:28:03 GMT 2021]
java.io.IOException: Unable to move /heritrix/output/frequent-npld/20210519154706/logs/crawl.log to /heritrix/output/frequent-npld/20210519154706/logs/crawl.log.cp00032-20210528102802
at org.archive.io.GenerationFileHandler.rotate(GenerationFileHandler.java:127)
at org.archive.crawler.reporting.BufferedCrawlerLoggerModule.rotateLogFiles(BufferedCrawlerLoggerModule.java:331)
at org.archive.crawler.reporting.BufferedCrawlerLoggerModule.doCheckpoint(BufferedCrawlerLoggerModule.java:393)
at org.archive.crawler.framework.CheckpointService.requestCrawlCheckpoint(CheckpointService.java:285)
...
```

Prior to the checkpoint failure, there are missing log files, e.g. no `crawl.log` / `alerts.log`. Instead, usually, the checkpoint-version of the file is still being written to, e.g. `crawl.log.cp00011-xxxxx`. In rare cases, the underlying Java logger FileHandler rotation appears to have kicked in, because we found a `crawl.log.1` file that was being written to.

This is presumably some kind of threading/race-condition in GenerationalFileHandler, perhaps brought on by Gluster occasionally blocking when performing file operations. Unfortunately, this is just a guess, and I don't know how to reproduce this error.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.