googlesamples / googlesamples/mlkit

[Bug report] Why can the values of x and y be negative when the score is > 0.5?

Open
#732 0 comments 0 reactions 1 assignee Claimed by @lipenny00 View on GitHub
Dominant language
Kotlin
Stars
4.3k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
Currently, I am developing a React Native application using a bridge to support pose detection based on MLKit, following the instructions provided here: https://medium.com/dogtronic/real-time-pose-detection-in-react-native-using-mlkit-e1819847c340

However, I am facing an issue with iPhone 14, where the values of position.x and position.y can be negative. It seems like the landmarks are skewed to the left and top, based on the negative values I am getting.

Has anyone encountered this issue, and does anyone have a solution?

**To Reproduce**
My code here:
`
#import
#import "PoseDetection.h"
#import
#import
#import
#import
#import

@implementation PoseDetection : NSObject

+ (NSArray *)findPose:(Frame *)frame {
CMSampleBufferRef buffer = frame.buffer;
UIImageOrientation orientation = frame.orientation;

MLKPoseDetectorOptions *options = [[MLKPoseDetectorOptions alloc] init];
options.detectorMode = MLKPoseDetectorModeSingleImage;

MLKPoseDetector *poseDetector =
[MLKPoseDetector poseDetectorWithOptions:options];

MLKVisionImage *image = [[MLKVisionImage alloc] initWithBuffer:buffer];
image.orientation = orientation;

NSError *error;
NSArray *poses = [poseDetector resultsInImage:image error:&error];

if (error != nil) {
// Error.
return @[];
}

if (poses.count == 0) {
// No pose detected.
return @[];
}

for (MLKPose *pose in poses) {
return @[
// More landmark here....
[self getLandmarkPosition:[pose landmarkOfType:MLKPoseLandmarkTypeLeftPinkyFinger] nameLandmark:@"left_pinky"],
[self getLandmarkPosition:[pose landmarkOfType:MLKPoseLandmarkTypeRightPinkyFinger] nameLandmark:@"right_pinky"],
// .....
];
}

return @[];
}

+ (NSDictionary *)getLandmarkPosition:(MLKPoseLandmark *)landmark nameLandmark:(NSString *)name {
MLKVision3DPoint *position = landmark.position;

// Check if position.x or position.y is less than 0
if ((position.x < 0 || position.y < 0) && landmark.inFrameLikelihood > 0.6) {
// Log the parameter values
NSLog(@"Landmark Name: %@", name);
NSLog(@"X: %f", position.x);
NSLog(@"Y: %f", position.y);
NSLog(@"Score: %f", landmark.inFrameLikelihood);
}

// Create the dictionary
NSDictionary *landmarkDict = @{
@"name": name,
@"x": [NSNumber numberWithDouble:position.x],
@"y": [NSNumber numberWithDouble:position.y],
@"score": [NSNumber numberWithDouble:landmark.inFrameLikelihood]
};

return landmarkDict;
}

@end

`

When I try to run, I get some value for position.x < 0 || position.y < 0, when score > 0.6
`
2023-09-18 11:32:50.105459+0700 FlexifymeApp[1014:336125] Landmark Name: right_pinky

2023-09-18 11:32:50.105516+0700 FlexifymeApp[1014:336125] X: -64.024948

2023-09-18 11:32:50.105533+0700 FlexifymeApp[1014:336125] Y: 311.074951

2023-09-18 11:32:50.105542+0700 FlexifymeApp[1014:336125] Score: 0.723728

2023-09-18 11:32:50.105555+0700 FlexifymeApp[1014:336125] Landmark Name: right_index

2023-09-18 11:32:50.105564+0700 FlexifymeApp[1014:336125] X: -32.133579

2023-09-18 11:32:50.105573+0700 FlexifymeApp[1014:336125] Y: 302.895447

2023-09-18 11:32:50.105593+0700 FlexifymeApp[1014:336125] Score: 0.793484
`

**Expected behavior**
A clear and concise description of what you expected to happen.

**SDK Info:**
- pod 'GoogleMLKit/PoseDetection', '3.2.0'

**Smartphone:**
- Device/Simulator: iPhone14

**Development Environment:**
*(For Android issue feel free to skip this section)*
- IDE Eversion: Xcode 144.3.1
- Laptop/Desktop:Macbook M2 pro 13inch

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.