apache / apache/stormcrawler

Response headers can populate the reserved verbatim request and response metadata keys

Open Beginner friendly
#2,091 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
995
Forks
292
Avg merge
2d 49m
Merged PRs (30d)
62

Description

## What happens
`HttpProtocol.getProtocolOutput()` copies every response header into the protocol metadata. When a header name equals `_request.headers_` or `_response.headers_`, the value is base64 decoded first and then stored under exactly the key the WARC writer treats as the crawler's own capture record. `_response.ip_` and `_request.time_` are copied in the same loop without any name check. The interceptor that legitimately sets these keys is only installed when `http.store.headers` is true, which is not the default, so with the default configuration nothing overwrites a value that came from the wire.

## Where
`core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java:434-446` on main; consumers in `external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java:349` and `:372-379` and `WARCRequestRecordFormat.java:54`. Config key: `http.store.headers`.

```java
if (key.equals(ProtocolResponse.REQUEST_HEADERS_KEY)
|| key.equals(ProtocolResponse.RESPONSE_HEADERS_KEY)) {
value = new String(Base64.getDecoder().decode(value), StandardCharsets.ISO_8859_1);
}
responsemetadata.addValue(key.toLowerCase(Locale.ROOT), value);
```

## Why it matters
`WARCRecordFormat.format` promotes a record from `resource` to `response` whenever `_response.headers_` is non-blank and embeds the decoded block verbatim, and `WARCRequestRecordFormat` writes the `_request.headers_` block as what the crawler supposedly sent. A fetched server can therefore choose the contents of records that are read as the crawler's own capture, including raw CRLF inside them, and can set `WARC-IP-Address` and `WARC-Date`. The archive consumer, not the crawler, carries the consequence, and `external/warc/README.md:150` tells WARC users to set `http.store.headers: true`, which makes the interceptor overwrite the forged values. Deployments that leave it false are the affected ones. Separately, the decode is unguarded, so a value that is not valid base64 throws `IllegalArgumentException` out of `getProtocolOutput` and fails that fetch.

## Reproduction

Save as `core/src/test/java/org/apache/stormcrawler/protocol/OkHttpReservedHeaderKeysTest.java`:

```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.stormcrawler.protocol;

import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
import org.apache.storm.Config;
import org.apache.stormcrawler.Metadata;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.handler.AbstractHandler;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

/** Checks that reserved protocol metadata keys cannot be set by a fetched server. */
class OkHttpReservedHeaderKeysTest extends AbstractProtocolTest {

private static String b64(String s) {
return Base64.getEncoder().encodeToString(s.getBytes(StandardCharsets.ISO_8859_1));
}

@Override
protected Handler[] getHandlers() {
return new Handler[] {
new AbstractHandler() {
@Override
public void handle(
String target,
Request baseRequest,
jakarta.servlet.http.HttpServletRequest request,
HttpServletResponse response)
throws IOException {
baseRequest.setHandled(true);
response.setStatus(HttpServletResponse.SC_OK);
response.setContentType("text/html");
response.setHeader(
ProtocolResponse.RESPONSE_HEADERS_KEY,
b64("HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\n\r\n"));
response.setHeader(
ProtocolResponse.REQUEST_HEADERS_KEY,
b64("GET /elsewhere HTTP/1.1\r\nHost: somewhere.example\r\n\r\n"));
response.setHeader(ProtocolResponse.RESPONSE_IP_KEY, "10.0.0.5");
response.setHeader(ProtocolResponse.REQUEST_TIME_KEY, "1");
final byte[] content = "hello".getBytes(StandardCharsets.UTF_8);
response.setContentLength(content.length);
try (OutputStream out = response.getOutputStream()) {
out.write(content);
}
}
}
};
}

@Test
void responseHeadersCannotSetReservedKeys() throws Exception {
Config conf = new Config();
conf.put("http.agent.name", "this_is_only_a_test");
org.apache.stormcrawler.protocol.okhttp.HttpProtocol protocol =
new org.apache.stormcrawler.protocol.okhttp.HttpProtocol();
protocol.configure(conf);
ProtocolResponse response =
protocol.getProtocolOutput("http://localhost:" + HTTP_PORT + "/", new Metadata());
Metadata md = response.getMetadata();
Assertions.assertNull(
md.getFirstValue(ProtocolResponse.RESPONSE_HEADERS_KEY),
"a response header must not populate the verbatim response record");
Assertions.assertNull(
md.getFirstValue(ProtocolResponse.REQUEST_HEADERS_KEY),
"a response header must not populate the verbatim request record");
Assertions.assertNull(
md.getFirstValue(ProtocolResponse.RESPONSE_IP_KEY),
"a response header must not populate the recorded IP address");
protocol.cleanup();
}
}
```

Run it:

```
mvn -pl core test -Dtest=OkHttpReservedHeaderKeysTest
```

A local Jetty server returns the reserved header names; the test asserts the intended behaviour and fails on main.

```
[ERROR] OkHttpReservedHeaderKeysTest.responseHeadersCannotSetReservedKeys:82 a response header must not populate the verbatim response record ==> expected: but was:
```

## Suggested fix
In `getProtocolOutput`, drop incoming response headers whose lowercased name matches one of the reserved keys (`REQUEST_HEADERS_KEY`, `RESPONSE_HEADERS_KEY`, `RESPONSE_IP_KEY`, `REQUEST_TIME_KEY`, `PROTOCOL_VERSIONS_KEY`, `TRIMMED_RESPONSE_KEY`, `TRIMMED_RESPONSE_REASON_KEY`), or store them under a quarantined prefix. Remove the base64 decoding of server-supplied values entirely, since only `HTTPHeadersInterceptor` should ever write these keys and it runs after this loop. Nothing legitimate sends header names of that shape, so no compatibility impact is expected.

Contributor guide

Open the contributing guide

Research direction

Start in core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java:434-446 and review how response headers enter protocol metadata, then compare the reserved keys consumed by the WARC format classes. Run core/src/test/java/org/apache/stormcrawler/protocol/OkHttpReservedHeaderKeysTest.java with mvn -pl core test -Dtest=OkHttpReservedHeaderKeysTest. Done means the test passes and server-supplied reserved metadata cannot populate crawler-owned values.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
networking, security
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
86/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.