Pose Animator WIP

12 Oct 2020 - Jake Sherwood

Pose Animator w/ Friends

Pose Animator WIP

Determined to get the Pose Animator lib to work, this week I dug deeper into the code base.

It is using a number of includes and a bundler named Parcel.js to bundle everything and build the app with all dependencies. It is also using Yarn package manager instead of NPM.

With the help from my professor I was able to get it running as an additional example in the WebRTC repo. I run the express server from there as normal and use yarn run watch from within the pose-animator dir to run the animator / parcel server. The pose-animator code is set to include webrtc_peer_client.js in the build.

import {
    initSocketClient,
    initPeerClient,
    sendData,
    sendVideo,
    getData,
    getVideo,
    terminateSession,
    handleClose,
    addStream,
} from "./utils/webrtc_peer_client.js";

This allows me to run multiple peers on localhost:1234

Once I had that sorted I dove deeper into the code to figure out what made it tick.

First things first I needed to send my poseNet data to my peer. Lisa had already given me the basics of “getting data” from the peer. So I just needed to modify that to save to an additional peerPoses array and then add it to an additional canvas element for the peer.

if (newData !== null) {

	let peerPoses = newData.data.skeleton;
	let peerFace = newData.data.face;
	
	...
	
	// don't think I really need to declare them as new vars here
	posesPeer = peerPoses;
	faceDetectionPeer = peerFace;

	...

 	drawPoseSkeletons(keypointCtxPeer, posesPeer, faceDetectionPeer, "pink", "orange");
	

I wrote a drawPoseSkeletons func and modified the original drawKeypoints & drawSkeleton funcs in /utils/demoUtils.js, to allow me to pass in pose arrays and colors.

function drawPoseSkeletons(canvas, pose, facePose, myColor, faceColor) {
	 canvas.clearRect(0, 0, videoWidth, videoHeight);

	 pose.forEach(({
		 score,
		 keypoints,

	 }) => {
		 if (score >= minPoseConfidence) {
			 drawKeypointsPose(keypoints, minPartConfidence, canvas, myColor, pose);
			 drawSkeletonPose(keypoints, minPartConfidence, canvas, myColor, pose);
		 }
	 });

	...

	drawPoseSkeletons(keypointCtx, poses, faceDetection, "aqua", "red");
	

Once I had that I spent some time trying to get the peer video streamed but was not able to solve it. I decided I needed to move on as its not necessary.

// trying to get Video data but seems like it is still the posenet data?
        let newVideoStream = getVideo();

        //console.log("newVideo = " + newVideoStream.id);
        //console.log("newVideo = " + JSON.stringify(newVideoStream, null, 2));
        // console.log(Object.entries(newVideoStream));

        newVideo = document.getElementById('remoteVideo');

        newVideo.srcObject = newVideoStream;
        

This just kept playing the same local stream and not the peer stream.

I had my poseNet data but I couldn’t figure out how to get a second SVG added to the main canvas. There are so many files. I pretty much went line by line through everything.

These seemed like the most relevant:

-/illustrationGen/illustration.js
-/illustrationGen/skeleton.js
-/utils/colorUtils.js
-/utils/demoUtils.js
-/utils/fileUtils.js
-/utils/mathUtils.js
-/utils/svgUtils.js

I just couldn’t find anything, but then on a hale mary I found the solution.

In the initial async bindPage() func there was this line.

await parseSVG(Object.values(avatarSvgs)[1]);  // sets default SVG

Commenting that out made the main illustration go away.

So I updated to add an additional parsePeerSVG call and it worked!

await parseSVG(Object.values(avatarSvgs)[1]);  // sets default SVG
await parsePeerSVG(Object.values(avatarSvgs)[0]); // sets peer SVG  adding this seemed to make it

Now I had two peers and two different illustrations dancing together. Yay!

Things still to do
1) Figure out how to run note on localhost only. I’m working on Lisa’s suggestion of using glitch.me
2) Update drag and drop to be able to change both illustrations by dragging onto poseNet windows.
3) Show/hide poseNet windows
4) Add additional interaction affects?
5) Additional simplified illustrations
6) Refactor duplicate code.

Work in progress code here.