googlesamples / googlesamples/mlkit
[Bug report] Why can the values of x and y be negative when the score is > 0.5?
- Dominant language
- Kotlin
- Stars
- 4.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Currently, I am developing a React Native application using a bridge to support pose detection based on MLKit, following the instructions provided here: https://medium.com/dogtronic/real-time-pose-detection-in-react-native-using-mlkit-e1819847c340
However, I am facing an issue with iPhone 14, where the values of position.x and position.y can be negative. It seems like the landmarks are skewed to the left and top, based on the negative values I am getting.
Has anyone encountered this issue, and does anyone have a solution?
**To Reproduce**
My code here:
`
#import
#import "PoseDetection.h"
#import
#import
#import
#import
#import
@implementation PoseDetection : NSObject
+ (NSArray *)findPose:(Frame *)frame {
CMSampleBufferRef buffer = frame.buffer;
UIImageOrientation orientation = frame.orientation;
MLKPoseDetectorOptions *options = [[MLKPoseDetectorOptions alloc] init];
options.detectorMode = MLKPoseDetectorModeSingleImage;
MLKPoseDetector *poseDetector =
[MLKPoseDetector poseDetectorWithOptions:options];
MLKVisionImage *image = [[MLKVisionImage alloc] initWithBuffer:buffer];
image.orientation = orientation;
NSError *error;
NSArray *poses = [poseDetector resultsInImage:image error:&error];
if (error != nil) {
// Error.
return @[];
}
if (poses.count == 0) {
// No pose detected.
return @[];
}
for (MLKPose *pose in poses) {
return @[
// More landmark here....
[self getLandmarkPosition:[pose landmarkOfType:MLKPoseLandmarkTypeLeftPinkyFinger] nameLandmark:@"left_pinky"],
[self getLandmarkPosition:[pose landmarkOfType:MLKPoseLandmarkTypeRightPinkyFinger] nameLandmark:@"right_pinky"],
// .....
];
}
return @[];
}
+ (NSDictionary *)getLandmarkPosition:(MLKPoseLandmark *)landmark nameLandmark:(NSString *)name {
MLKVision3DPoint *position = landmark.position;
// Check if position.x or position.y is less than 0
if ((position.x < 0 || position.y < 0) && landmark.inFrameLikelihood > 0.6) {
// Log the parameter values
NSLog(@"Landmark Name: %@", name);
NSLog(@"X: %f", position.x);
NSLog(@"Y: %f", position.y);
NSLog(@"Score: %f", landmark.inFrameLikelihood);
}
// Create the dictionary
NSDictionary *landmarkDict = @{
@"name": name,
@"x": [NSNumber numberWithDouble:position.x],
@"y": [NSNumber numberWithDouble:position.y],
@"score": [NSNumber numberWithDouble:landmark.inFrameLikelihood]
};
return landmarkDict;
}
@end
`
When I try to run, I get some value for position.x < 0 || position.y < 0, when score > 0.6
`
2023-09-18 11:32:50.105459+0700 FlexifymeApp[1014:336125] Landmark Name: right_pinky
2023-09-18 11:32:50.105516+0700 FlexifymeApp[1014:336125] X: -64.024948
2023-09-18 11:32:50.105533+0700 FlexifymeApp[1014:336125] Y: 311.074951
2023-09-18 11:32:50.105542+0700 FlexifymeApp[1014:336125] Score: 0.723728
2023-09-18 11:32:50.105555+0700 FlexifymeApp[1014:336125] Landmark Name: right_index
2023-09-18 11:32:50.105564+0700 FlexifymeApp[1014:336125] X: -32.133579
2023-09-18 11:32:50.105573+0700 FlexifymeApp[1014:336125] Y: 302.895447
2023-09-18 11:32:50.105593+0700 FlexifymeApp[1014:336125] Score: 0.793484
`
**Expected behavior**
A clear and concise description of what you expected to happen.
**SDK Info:**
- pod 'GoogleMLKit/PoseDetection', '3.2.0'
**Smartphone:**
- Device/Simulator: iPhone14
**Development Environment:**
*(For Android issue feel free to skip this section)*
- IDE Eversion: Xcode 144.3.1
- Laptop/Desktop:Macbook M2 pro 13inch
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.