InetIntel / InetIntel/grip-api-v2

Unable to get all events at a given point in time

Open
#2 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

When using the GRIP API (https://api.grip.inetintel.cc.gatech.edu/json/events), I encountered the issue that if many events occur simultaneously, it is not (easily) possible to fetch all events due to the limit of returning at most 10000 results for a given query.
To solve this issue, I initially tried to split my query into smaller queries using the available search filters described in (https://github.com/InetIntel/grip-api-legacy/blob/master/api-spec.md). In particular, I tried to narrow down the time range (`ts_start` and `ts_end`), the event duration (`min_duration` and `max_duration`), the event type (`event_type`), and the suspicion level (`min_susp` and `max_susp`). However even when narrowing all of these filters down to a single value, I still hit the limit of 10000 events.

- Concrete example of the issue:
The following query which fetches `moas` events for a specific duration (300s) and suspicion level (80) at time 2024-11-04T11:30:00 returns 10000 events (indicating that more events exist at this point in time): https://api.grip.inetintel.cc.gatech.edu/json/events?event_type=moas&start=0&full=true&min_duration=300&max_duration=300&min_susp=80&max_susp=80&length=1&ts_start=2024-11-04T11:30:00&ts_end=2024-11-04T11:30:00

- Proposed solution:
Keep the record limit at 10000 but allow the user to fetch events beyond the record limit through the use of the elasticsearch `search_after` feature (https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after). If the other issue regarding the nondeterministic sort order (#1) is addressed, this could be implemented as follows:

Allow the user to specify two optional GET parameters `search_after_view_ts` and `search_after_id`, and then add the following logic in line https://github.com/InetIntel/grip-api-v2/blob/main/app/elastic.py#L340:

```
search_after_view_ts = queryparams.get("search_after_view_ts", type=int)
search_after_id = queryparams.get("search_after_id", type=str)
if search_after_view_ts is not None and search_after_id is not None:
kwargs["search_after"] = [search_after_view_ts, search_after_id]
```

Unfortunately, since I do not have access to the database and the elasticsearch backend, I cannot test whether this change will solve the issue or not.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in app/elastic.py around line 340 and review the GRIP API event-query handling, then read the Elasticsearch search_after documentation and issue #1 about deterministic sorting. The change is complete when the optional search_after_view_ts and search_after_id parameters allow clients to retrieve events beyond 10,000 while preserving the existing query behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
api, databases, search
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.