Duplicated Fitbit data on output storage for certain topics

Open
#154 0 comments 0 reactions 1 assignee View on GitHub

@mpgxvii is already working on this.

Since Jan 15, 2025.

Assessment

This issue has not been assessed yet.

Description

Problem

For certain Fitbit topics on a client deployment, selected topics started showing duplicated data rows in output storage analogous to:

key.subject key.source key.project value.time value.timeRecieved value.value
my-subject-id my-source-id my-project-id 000000001 000000005 20
my-subject-id my-source-id my-project-id 000000001 0000000010 20

Note that Fitbit data recorded at the same time appears as a separate line; the only difference is the value.timeReceived.

This was observed for the following Fitbit topics:

connect_fitbit_breathing_rate
connect_fitbit_intraday_heart_rate_variability
connect_fitbit_sleep_stages
connect_fitbit_activity_log
connect_fitbit_skin_temperature
connect_fitbit_sleep_classic

it was noted to NOT occur for the following Fitbit topics:

connect_fitbit_intraday_calories
connect_fitbit_intraday_steps
connect_fitbit_intraday_heart_rate

Some observations

  • Data deduplication by output-restructure normally takes all columns into account. For this reason the value.timeReceived is causing the same piece of data to be added as a separate line.
  • I suspect that previous from older versions of Fitbit connector, the current connector repeatedly downloads the same date range for the problematic topics. The inclusion of the value.timeReceived column during de-deduplication causes the data to appear multiple times in the output files.
  • Many of the problematic topics were updated in PR #118
  • The topics that work well all have a 1d time period hard-coded in the request URL whereas others do not:
FitbitIntradayCaloriesRoute.java:    return baseUrl + "/1/user/%s/activities/calories/date/%s/1d/1min/time/%s/%s.json?timezone=UTC";
FitbitRestingHeartRateRoute.java:    return baseUrl + "/1/user/%s/activities/heart/date/%s/1d.json?timezone=UTC";
FitbitIntradayStepsRoute.java:    return baseUrl + "/1/user/%s/activities/steps/date/%s/1d/1min/time/%s/%s.json?timezone=UTC";
FitbitIntradayHeartRateRoute.java:    return baseUrl + "/1/user/%s/activities/heart/date/%s/1d/1sec/time/%s/%s.json?timezone=UTC";
FitbitSkinTemperatureRoute.java:    return baseUrl + "/1/user/%s/temp/skin/date/%s/%s.json";
FitbitBreathingRateRoute.java:    return baseUrl + "/1/user/%s/br/date/%s/%s/all.json";
FitbitIntradayHeartRateVariabilityRoute.java:    return baseUrl + "/1/user/%s/hrv/date/%s/%s/all.json";
FitbitIntradaySpo2Route.java:    return baseUrl + "/1/user/%s/spo2/date/%s/%s/all.json";
FitbitTimeZoneRoute.java:    return baseUrl + "/1/user/%s/profile.json";
FitbitSleepRoute.java:    return baseUrl + "/1.2/user/%s/sleep/list.json?sort=asc&afterDate=%s&limit=100&offset=0";
FitbitActivityLogRoute.java:    return baseUrl + "/1/user/%s/activities/list.json?sort=asc&afterDate=%s&limit=20&offset=0";
Dominant language
Java
Stars
5
Forks
6
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from RADAR-base/RADAR-REST-Connector

All issues in RADAR-base/RADAR-REST-Connector

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.