Pose Animator p2p

20 Oct 2020 - Jake Sherwood

Pose Animator w/ Friends

Pose Animator - Dancing Together

For my final for The Body Everywhere and Here I worked further on the peer 2 peer pose animator.

Using PoseNet and the Pose Animator lib I was able to create a fun collaborative space where you can bring illustrations to life and dance together.

The things I still needed to work on from last week were:
1) Figure out how to run note on localhost only. I’m working on Lisa’s suggestion of using glitch.me
2) Update drag and drop to be able to change both illustrations by dragging onto poseNet windows.
3) Show/hide poseNet windows
4) Add additional interaction affects?
5) Additional simplified illustrations
6) Refactor duplicate code.

I accomplished most of those but still need to clean up the code more and sort out a few last bugs.

As in the last post the biggest hurdle here was just finding out what did what in the massive Pose Animator library. It was definitely daunting but I learned a lot in the process.

To get it up and running with a remote client, I tried a number of things. I tried glitch, I tried running it on my on server, setting up a proxy and I tried just exposing the ngrok address.

All of which had their own problems.

Eventually with a little help, I was able to get it working on my own server.

I ended up having to build all the pose animator files on the server. Then I set up a simple expressjs app to run a server with the built files, which were pointed to my ngrok address exposing my local peer.

// We need the file system for getting to the cert
var fs = require('fs');
                                
// Express is a node module for building HTTP servers
var express = require('express');
var app = express();

// Tell Express to look in the "public" folder for any files first
app.use(express.static('dist'));

// If the user just goes to the "route" / then run this function
app.get('/', function (req, res) {
  res.send('Hello World!')
});

// Here is the actual HTTP server 
// In this case, HTTPS (secure) server
var https = require('https');

// Security options - key and certificate
var options = {
        key: fs.readFileSync('INSERT_CERT_KEY'),
        cert: fs.readFileSync('INSERT_CERT_FILE')
};

// We pass in the Express object and the options object
var httpServer = https.createServer(options, app);

// Default HTTPS port
httpServer.listen(443);

Once I had the peer connection sorted, this took a few days, I was back to developing code.

The original drag and drop functionality allowed for dragging SVG files to the canvas window to change the animation svg. But I wanted to be able to control/change both local and peer illustrations.

I didn’t get all the way through the code clean up so there is some dupe code here, but to make it work I ended up making a couple new functions to drag and drop and then parse local and peer SVGs. That allowed me to drag to my local poseNet canvas window or the peer window.

FileUtils.setDragDropHandlerLocal((result) => {
    parseSVG(result);
    uploadedSvg = result;
});
FileUtils.setDragDropHandlerPeers((result) => {
    parsePeerSVG(result);
    uploadedPeerSvg = result;
});

async function parseSVG(target) {
    let svgScope = await SVGUtils.importSVG(target /* SVG string or file path */ );
    let skeleton = new Skeleton(svgScope);
    illustration = new PoseIllustration(canvasScope);
    illustration.bindSkeleton(skeleton, svgScope);
}

async function parsePeerSVG(target) {
    let svgScope = await SVGUtils.importSVG(target /* SVG string or file path */ );
    let skeleton = new Skeleton(svgScope);
    illustrationPeer = new PoseIllustration(canvasScope);
    illustrationPeer.bindSkeleton(skeleton, svgScope);
}

drag and drop local and peer

Next was hiding the poseNet data windows. This was relatively easy but I probably spent more time than I should have on the button. Clicking the button just shows and hides the poseNet data windows container element.

poseBtn.addEventListener('click', function () {
	console.log('toggle pose windows');
	let poseWindowDisplay = poseWindow.style.display;
	if (poseWindowDisplay == 'block') {
		poseWindow.style.display = 'none';
		eyeShow.classList.remove('icon-eye');
		eyeShow.classList.add('eye-hide');
		eyeHide.classList.remove('icon-eye');
		eyeHide.classList.add('eye-hide');
		eyeLable.classList.add('eye-hide');
	} else {
		poseWindow.style.display = 'block';
		eyeShow.classList.remove('eye-hide');
		eyeShow.classList.add('icon-eye');
		eyeHide.classList.remove('eye-hide');
		eyeHide.classList.add('icon-eye');
		eyeLable.classList.remove('eye-hide');
	}
});

show poses button

I also wanted to add some more playful interaction between the the 2 users. I ended up with a somewhat buggy way of changing the SVG by both users touching right hands (wrists). It was working when I ran it locally but with a remote peer it seemed much more buggy. But when it worked, it would cycle through an array of available SVGs.

I also worked with interactions of just swapping the 2 user’s SVGs. But allowing them to try out multiple illustrations if they didn’t have illustrations of their own seemed fun.

I just reverted back to the pose wrists touching interaction. This way the remote peer can touch wrists and change the illustrations for the other user.

For some reason this work more reliably.

// are peer wrists touching?  - works more reliably for some reason
if (touched(peerWristR, peerWristL)) {
	console.log('peer wrist touching from touched func');
	swapSvgs();

} else {}

// are local and remote wrists touching? - interaction hard to control
// if (touched(posesWristR, peerWristR)) {
//     console.log('pose wristr & peer wristr touching from touched func');
//     swapSvgs();

// } else {}

I’d like to add functionality to send the SVG used to the peer, but thought it might be problematic if either peer was using custom SVGs that weren’t in the repo.

I did get a couple of new simplified illustrations done, but they still had strange stretching problems.

new illustrations

These worked better than my more complicated previous illustrations, but some more fine tuning is required to get them to look good when animated.

I also cleaned up the code a bit but have a bunch more to do.

I was still getting some strange peer pose data but it only appeared to really happen over a remote connection. Not when I was connected locally.

remote peer data bug & local peer w/o data bug

I also want to create a good README.md file for the GitHub repo incase others want to try it out.

All in all I’m happy with the result. I do plan to put a little more work into it. As Fernando mentioned I can see this being a really great storytelling tool and I want to explore that direction.

Github here.