Scheduler ignores per-feed update intervals and error backoff

The scheduler fetches every enabled feed on every tick, so the per-feed update interval set in the UI and the exponential backoff applied to failing feeds never affect what gets fetched. Line references are against 35fcc394ad4e79b5302e4729f634052fd34a131a.

Mechanism

UpdateAllFeeds selects all enabled feeds rather than the ones that are due:

• pkg/scheduler/feed_processor.go:287-301: GetFeeds(ctx, true), then UpdateFeed for every result.
• pkg/scheduler/feed_processor.go:382-385: a successful fetch computes nextFetch = time.Now().Add(f.FetchInterval) and stores it through UpdateFeedFetched.
• pkg/repository/feed.go:154-165: a failed fetch pushes next_fetch out with exponential backoff, from 10 minutes up to 24 hours.
• pkg/repository/feed.go:99-117: GetFeedsToFetch filters on next_fetch IS NULL OR next_fetch <= datetime('now'), and nothing calls it outside pkg/repository/feed_test.go.

next_fetch is written on both paths and shown to the user on the feed card (server/templates/feed-card.html:27-29), but the scheduler never consults it when deciding what to fetch. The FeedManager interface at pkg/scheduler/scheduler.go:53-58 does not carry GetFeedsToFetch, so the scheduler has no way to ask for the due set.

Reproduction

Drop this in pkg/scheduler/repro_test.go and run go test ./pkg/scheduler/ -run TestRepro_FeedIntervalsIgnored:

package scheduler

import (
	"context"
	"sync"
	"testing"
	"time"

	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"

	"github.com/umputun/newscope/pkg/domain"
	"github.com/umputun/newscope/pkg/repository"
	"github.com/umputun/newscope/pkg/scheduler/mocks"
)

func TestRepro_FeedIntervalsIgnored(t *testing.T) {
	ctx := context.Background()
	repos, err := repository.NewRepositories(ctx, repository.Config{
		DSN: ":memory:", MaxOpenConns: 1, MaxIdleConns: 1, ConnMaxLifetime: 30 * time.Second,
	})
	require.NoError(t, err)
	defer repos.Close()

	due := &domain.Feed{URL: "https://example.com/due.xml", Title: "due", FetchInterval: 30 * time.Minute, Enabled: true}
	waiting := &domain.Feed{URL: "https://example.com/waiting.xml", Title: "waiting", FetchInterval: 30 * time.Minute, Enabled: true}
	failing := &domain.Feed{URL: "https://example.com/failing.xml", Title: "failing", FetchInterval: 30 * time.Minute, Enabled: true}
	for _, f := range []*domain.Feed{due, waiting, failing} {
		require.NoError(t, repos.Feed.CreateFeed(ctx, f))
	}

	// waiting feed was just fetched, next fetch is 29 minutes out
	require.NoError(t, repos.Feed.UpdateFeedFetched(ctx, waiting.ID, time.Now().Add(29*time.Minute)))
	// failing feed hit an error, backoff pushes next fetch 10 minutes out
	require.NoError(t, repos.Feed.UpdateFeedError(ctx, failing.ID, "connection refused"))

	toFetch, err := repos.Feed.GetFeedsToFetch(ctx, 100)
	require.NoError(t, err)
	require.Len(t, toFetch, 1)
	require.Equal(t, "due", toFetch[0].Title)

	var mu sync.Mutex
	var parsed []string
	parser := &mocks.ParserMock{ParseFunc: func(ctx context.Context, url string) (*domain.ParsedFeed, error) {
		mu.Lock()
		defer mu.Unlock()
		parsed = append(parsed, url)
		return &domain.ParsedFeed{Title: "empty"}, nil
	}}

	fp := NewFeedProcessor(FeedProcessorConfig{
		FeedManager:           repos.Feed,
		ItemManager:           &mocks.ItemManagerMock{},
		ClassificationManager: &mocks.ClassificationManagerMock{},
		SettingManager:        &mocks.SettingManagerMock{},
		Parser:                parser,
		Extractor:             &mocks.ExtractorMock{},
		Classifier:            &mocks.ClassifierMock{},
		MaxWorkers:            1,
		RetryFunc:             func(ctx context.Context, op func() error) error { return op() },
	})

	fp.UpdateAllFeeds(ctx, make(chan domain.Item, 10))

	mu.Lock()
	defer mu.Unlock()
	assert.Equal(t, []string{"https://example.com/due.xml"}, parsed,
		"only the feed whose next_fetch has passed should be fetched")
}

Relevant excerpt of the failure on master:

--- FAIL: TestRepro_FeedIntervalsIgnored (0.31s)
        Error:      	Not equal:
                    	expected: []string{"https://example.com/due.xml"}
                    	actual  : []string{"https://example.com/due.xml", "https://example.com/failing.xml", "https://example.com/waiting.xml"}
        Messages:   	only the feed whose next_fetch has passed should be fetched

GetFeedsToFetch agrees that only due is due, and UpdateAllFeeds fetches all three anyway, including the one under error backoff.

Impact

Every enabled feed is fetched at the scheduler tick, so the tick is the real interval for all of them and the per-feed value has no scheduling effect, it only sets the next-fetch timestamp that gets stored and displayed. With the loader and README default of schedule.update_interval: 1m, a feed left at the form default of 30 minutes is fetched 30 times too often and one set to the form maximum of 1440 minutes is fetched 1440 times too often; with the 30-minute tick in the checked-in config.yml the 30-minute feed happens to line up while shorter and longer intervals do not. A feed that keeps failing is retried at every tick as well, even after the backoff has moved its next_fetch up to 24 hours out, which increases the risk of being rate limited or blocked by that source. The card meanwhile shows a "Next fetch" time that nothing acts on.

Options

1. Use GetFeedsToFetch in UpdateAllFeeds. Add the method to the FeedManager interface, regenerate pkg/scheduler/mocks/feed_manager.go, change UpdateAllFeeds and the scheduler tests that stub GetFeedsFunc, and pass an effectively unbounded limit, since maxWorkers already bounds concurrency. FeedRepository implements it already. Three things worth deciding along with it: the tick stays the scheduling resolution, so a feed cannot run more often than schedule.update_interval whatever its own interval says; a restart no longer refetches everything, since feeds under an interval or a backoff stay skipped until due; and editing an interval (pkg/repository/feed.go:188-190) or re-enabling a feed (pkg/repository/feed.go:178-184) leaves the stored next_fetch alone, so a feed switched from 24 hours to 5 minutes waits out the old deadline unless those two also reset it.
2. Same as 1, but force a full pass on startup, honouring next_fetch only on subsequent ticks, so "restart to refresh everything" keeps working. It costs a flag on the first call plus the tests to cover both paths.
3. Keep fetching everything each tick and drop the per-feed interval from the feed form, the feed card and UpdateFeed, since it is a stored value with no effect. This leaves the error backoff still ignored, so it only addresses half of the problem.

I would go with option 1 plus resetting next_fetch when the interval changes or a feed is re-enabled, and option 2 on top if you want the restart behaviour preserved. Whichever shape you pick wants integration tests covering a due feed, one waiting on its interval, one under error backoff, and an interval edit.

Start in pkg/scheduler/feed_processor.go at UpdateAllFeeds and inspect the FeedManager interface in pkg/scheduler/scheduler.go alongside GetFeedsToFetch in pkg/repository/feed.go. Read the scheduler and repository tests, including the reproduction described in the issue. Done means due feeds are fetched while interval-waiting and backoff feeds are skipped, with tests covering the chosen startup and interval-edit behavior.