Duplicated Fitbit data on output storage for certain topics
@mpgxvii is already working on this.
Since Jan 15, 2025.
Assessment
This issue has not been assessed yet.
Description
Problem
For certain Fitbit topics on a client deployment, selected topics started showing duplicated data rows in output storage analogous to:
| key.subject | key.source | key.project | value.time | value.timeRecieved | value.value |
|---|---|---|---|---|---|
| my-subject-id | my-source-id | my-project-id | 000000001 | 000000005 | 20 |
| my-subject-id | my-source-id | my-project-id | 000000001 | 0000000010 | 20 |
Note that Fitbit data recorded at the same time appears as a separate line; the only difference is the value.timeReceived.
This was observed for the following Fitbit topics:
connect_fitbit_breathing_rate
connect_fitbit_intraday_heart_rate_variability
connect_fitbit_sleep_stages
connect_fitbit_activity_log
connect_fitbit_skin_temperature
connect_fitbit_sleep_classic
it was noted to NOT occur for the following Fitbit topics:
connect_fitbit_intraday_calories
connect_fitbit_intraday_steps
connect_fitbit_intraday_heart_rate
Some observations
- Data deduplication by output-restructure normally takes all columns into account. For this reason the
value.timeReceivedis causing the same piece of data to be added as a separate line. - I suspect that previous from older versions of Fitbit connector, the current connector repeatedly downloads the same date range for the problematic topics. The inclusion of the
value.timeReceivedcolumn during de-deduplication causes the data to appear multiple times in the output files. - Many of the problematic topics were updated in PR #118
- The topics that work well all have a
1dtime period hard-coded in the request URL whereas others do not:
FitbitIntradayCaloriesRoute.java: return baseUrl + "/1/user/%s/activities/calories/date/%s/1d/1min/time/%s/%s.json?timezone=UTC";
FitbitRestingHeartRateRoute.java: return baseUrl + "/1/user/%s/activities/heart/date/%s/1d.json?timezone=UTC";
FitbitIntradayStepsRoute.java: return baseUrl + "/1/user/%s/activities/steps/date/%s/1d/1min/time/%s/%s.json?timezone=UTC";
FitbitIntradayHeartRateRoute.java: return baseUrl + "/1/user/%s/activities/heart/date/%s/1d/1sec/time/%s/%s.json?timezone=UTC";
FitbitSkinTemperatureRoute.java: return baseUrl + "/1/user/%s/temp/skin/date/%s/%s.json";
FitbitBreathingRateRoute.java: return baseUrl + "/1/user/%s/br/date/%s/%s/all.json";
FitbitIntradayHeartRateVariabilityRoute.java: return baseUrl + "/1/user/%s/hrv/date/%s/%s/all.json";
FitbitIntradaySpo2Route.java: return baseUrl + "/1/user/%s/spo2/date/%s/%s/all.json";
FitbitTimeZoneRoute.java: return baseUrl + "/1/user/%s/profile.json";
FitbitSleepRoute.java: return baseUrl + "/1.2/user/%s/sleep/list.json?sort=asc&afterDate=%s&limit=100&offset=0";
FitbitActivityLogRoute.java: return baseUrl + "/1/user/%s/activities/list.json?sort=asc&afterDate=%s&limit=20&offset=0";
- Dominant language
- Java
- Stars
- 5
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from RADAR-base/RADAR-REST-Connector
-
bug
RADAR-base/RADAR-REST-Connector#212 · 1 assignee ·
-
Difficulty 5/5 Over a week Newbie friendliness 42/100
-
RADAR-base/RADAR-REST-Connector#193 · 1 reaction · 1 assignee ·
-
RADAR-base/RADAR-REST-Connector#190 · 1 comment · 1 assignee ·
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 35/100
All issues in RADAR-base/RADAR-REST-Connector
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
bug needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 1/5 Under an hour Newbie friendliness 94/100
objectionary/hone-maven-plugin#1061 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
spring-projects/spring-modulith#1895 ·