Behavior of rotationX/Y/Z and accelerationX/Y/Z on mobile browsers

Hi, I would like to report some inconsistencies/failures of rotationX/Y/Z and accelerationX/Y/Z on different mobile browsers, and propose some fixes.

• A demo comparing current behavior and my patches can be found at https://p5-accelerometer-test.glitch.me/
• The full code of the above can be found at https://glitch.com/edit/#!/p5-accelerometer-test
• A video recording of the demo featuring me and 2 phones, illustrating the problem and the fix

I've tested on iOS 13.4.1 (iPhone and iPad) and Android 10 (Pixel 4) because these are the only devices I have access to.

p5.js version: 1.1.9 (latest at time of writing)

I'm not familiar with implementation details of p5, so please correct me if I've made any mistakes :)

I: iOS 13 sensor access permission

On iOS 13 webpages need to pop up a dialog to beg for the user's permission to access sensors. The dialog can only be triggered when the user has interacted with the page (ontouchstart doesn't seem to count, while ontouchend does). p5.js doesn't seem to implement this.

Proposed fix

let hasSensorPermission = !(DeviceOrientationEvent.requestPermission || DeviceMotionEvent.requestPermission);

function begPermission(){
  if (DeviceOrientationEvent.requestPermission){
    DeviceOrientationEvent.requestPermission()
    .then(response => {
      if (response == 'granted') {
        if (DeviceMotionEvent.requestPermission){
          DeviceMotionEvent.requestPermission()
          .then(response => {
            if (response == 'granted') {
              hasSensorPermission = true;
            }
          })
          .catch(alert)
        }
      }
    })
    .catch(alert)
  }
}

function touchEnded() {
  if (!hasSensorPermission){
    begPermission();
  }
}

II: accelerationX/Y/Z does not work on iOS

It seems that iOS devices (or at least all the ones I have access to) doesn't have DeviceMotionEvent.acceleration. They only have what's called DeviceMotionEvent.accelerationIncludingGravity. p5 seems to only listen to the former, and as such, the variables accelerationX/Y/Z are constantly 0.

Moreover, as its name suggests, accelerationIncludingGravity takes the gravitational acceleration into account, producing inconsistent behavior compared to devices that supports the plain acceleration.

Proposed fix

The fix involves calculating components of g=9.80665ms^-2 on each axis based on the current rotation and cancelling them out from the accelerometer reading. (Which won't work accurately in outer space or some really high parts of the Earth. navigator.geolocation.getCurrentPosition might be further used to deduce the correct g but that's probably too crazy :P).

// fixed values
let accX;
let accY;
let accZ;

// 3d transformation helpers
let ROTX = a=> [1,0,0,0, 0,cos(a),-sin(a),0, 0,sin(a),cos(a),0, 0,0,0,1]
let ROTY = a=> [cos(a),0,sin(a),0, 0,1,0,0, -sin(a),0,cos(a),0, 0,0,0,1]
let MULT = (A,B)=> [(A)[0]*(B)[0]+(A)[1]*(B)[4]+(A)[2]*(B)[8]+(A)[3]*(B)[12],(A)[0]*(B)[1]+(A)[1]*(B)[5]+(A)[2]*(B)[9]+(A)[3]*(B)[13],(A)[0]*(B)[2]+(A)[1]*(B)[6]+(A)[2]*(B)[10]+(A)[3]*(B)[14],(A)[0]*(B)[3]+(A)[1]*(B)[7]+(A)[2]*(B)[11]+(A)[3]*(B)[15],(A)[4]*(B)[0]+(A)[5]*(B)[4]+(A)[6]*(B)[8]+(A)[7]*(B)[12],(A)[4]*(B)[1]+(A)[5]*(B)[5]+(A)[6]*(B)[9]+(A)[7]*(B)[13],(A)[4]*(B)[2]+(A)[5]*(B)[6]+(A)[6]*(B)[10]+(A)[7]*(B)[14],(A)[4]*(B)[3]+(A)[5]*(B)[7]+(A)[6]*(B)[11]+(A)[7]*(B)[15],(A)[8]*(B)[0]+(A)[9]*(B)[4]+(A)[10]*(B)[8]+(A)[11]*(B)[12],(A)[8]*(B)[1]+(A)[9]*(B)[5]+(A)[10]*(B)[9]+(A)[11]*(B)[13],(A)[8]*(B)[2]+(A)[9]*(B)[6]+(A)[10]*(B)[10]+(A)[11]*(B)[14],(A)[8]*(B)[3]+(A)[9]*(B)[7]+(A)[10]*(B)[11]+(A)[11]*(B)[15],(A)[12]*(B)[0]+(A)[13]*(B)[4]+(A)[14]*(B)[8]+(A)[15]*(B)[12],(A)[12]*(B)[1]+(A)[13]*(B)[5]+(A)[14]*(B)[9]+(A)[15]*(B)[13],(A)[12]*(B)[2]+(A)[13]*(B)[6]+(A)[14]*(B)[10]+(A)[15]*(B)[14],(A)[12]*(B)[3]+(A)[13]*(B)[7]+(A)[14]*(B)[11]+(A)[15]*(B)[15]]
let TRFM = (A,v)=> [((A)[0]*(v)[0]+(A)[1]*(v)[1]+(A)[2]*(v)[2]+(A)[3])/((A)[12]*(v)[0]+(A)[13]*(v)[1]+(A)[14]*(v)[2]+(A)[15]),((A)[4]*(v)[0]+(A)[5]*(v)[1]+(A)[6]*(v)[2]+(A)[7])/((A)[12]*(v)[0]+(A)[13]*(v)[1]+(A)[14]*(v)[2]+(A)[15]),((A)[8]*(v)[0]+(A)[9]*(v)[1]+(A)[10]*(v)[2]+(A)[11])/((A)[12]*(v)[0]+(A)[13]*(v)[1]+(A)[14]*(v)[2]+(A)[15])]

window.ondevicemotion = function(event) {
  if (!event.acceleration){ // devices that don't support plain acceleration
    
    // compute gravitational acceleration's component on X Y Z axes based on gyroscope
    // g = ~ 9.80665
    let grav = TRFM(MULT(
      ROTY(radians(rotationY)),
      ROTX(radians(rotationX))
    ),[0,0,-9.80665]);
    
    accX =  (event.accelerationIncludingGravity.x+grav[0]);
    accY =  (event.accelerationIncludingGravity.y+grav[1]);
    accZ =  (event.accelerationIncludingGravity.z-grav[2]);
    
    // p5 appears to be doubling the acceleration for reasons that aren’t explained:
    // https://github.com/processing/p5.js/blob/main/src/events/acceleration.js#L647
    accX *= 2;
    accY *= 2;
    accZ *= 2;
  }
  
}

III: rotationX/Y does not take device orientation into account

When user is in landscape/portrait, rotationX and rotation Y's correspondence to rotateX() and rotateY() are swapped.

I know this is inherited from HTML's API, and the users of p5 library can always figure out the orientation of the device first and swap these variables themselves. But I think the fact that the graphics (which is forcibly rotated by the OS) is no longer in sync with the sensor data can be anti-intuitive. And as a beginner friendly library I think there's an opportunity for p5 to wrap this a bit differently (or perhaps provide it as an option for the user.)

Proposed fix

This fix involves reading window.orientation which is theoretically one of (0,90,-90,180). My phones (and probably most phones in general) don't allow people to hold them vertically upside-down (flipped portrait), so I didn't include the case.

rotX = radians([-rotationY,-rotationX,rotationY][~~(window.orientation/90)+1]);
rotY = radians([-rotationX, rotationY,rotationX][~~(window.orientation/90)+1]);
rotZ = radians(rotationZ);

IV: Documentation error on https://p5js.org/reference/#/p5/rotationX

The text says the order should be Z-X-Y (which is correct), but the example code on the same page apparently applies them in Y-X-Z order (which is incorrect).

I believe in p5.js, a series of transformations is equivalent to left matrix multiplication in reverse order, so

rotateZ(z);
rotateX(x);
rotateY(y);

is

Rz * (Rx * (Ry * v)))

Which is in fact what people call Y-X-Z order.

The correct Z-X-Y order should be written as:

rotateY(y);
rotateX(x);
rotateZ(z);

The same issue also appears in https://p5js.org/reference/#/p5/rotationY and https://p5js.org/reference/#/p5/rotationZ

Start by reproducing the cases in the linked p5-accelerometer-test demo on the stated iOS and Android devices, then inspect src/events/acceleration.js and the rotationX, rotationY, and rotationZ reference pages. The issue covers sensor permission, acceleration values, device orientation, and documentation examples; done requires an agreed scope and verified behavior or documentation for each affected case.