MemePeeps hi5.

27 Sep 2020 - Jake Sherwood

MemePeeps hi5

MemePeeps hi5

Assigned to build on our p2p work, this week we add ml5 and posenet to create a shared experience with the body.

Inspired by some classmate work in another class, my concept was to take a set of memes and have them build out “meme people” aka meme peeps. If you hi5 with a partner user you build out the skeleton keypoints with memes.

I also had a refresher and accessing API data in p5 as I wanted to pull the data from the imgflip.com API

Lisa helped me get my code sorted out to properly iterate and create variables for each img and push the image url into an array.

let url = "https://api.imgflip.com/get_memes";
// array for memes 
let imgs = [];

function gotData(data) {
  // get data 
  let memes = data.data.memes;

  // load each image 
  for (let j = 0; j < memes.length; j++) {
    imgs[j] = loadImage(memes[j].url);
  }
}

function preload() {
  loadJSON(url, gotData);
}

This was working great for most of my testing but as I write this I’m not sure if they’re throttling my API calls but my app is not working very well, with huge latency right now. Looking at the network monitor it is taking 4+ minutes to load the page and then extra time past that to load the model.

I added some functions to get the left and the right wrist.

Originally I had it just looking for right wrist but then when the app loads it almost immediately builds out the meme person due to the the wrist being in similar areas.

Actually turns out same thing happens when I changed the logic to have right wrist and left wrist touch.

memePeeps mirrored interaction

Oddly it does’t have this problem when running in mirror mode.

I also experimented with various apps to use my iPhone as a webcam. OPCcam and Had medium to bad results.

Initially I really wanted to use SVG animation library Lisa linked in the syllabus. But it gave me a number of problems.

I put a bit of work into this as well. I did some illustrations. Got the repo cloned and played around with it.

My main issue was taking the code from the repo and refactoring it into our p5 sketch setup. There a lot of additional dependencies. It was using node includes and babel to compile them. I wasn’t sure how to work through all that and add it to the p5 sketch.js we’ve been working with.

It was also creating some strange stretching of my illustrations that I couldn’t get sorted even with adjusting the SVG skeleton.

Hopefully I can get a better idea of how to do something like that in office hours with my professor.

Here are some of the illustrations and the skewness.

SVG pirates

Overall I am pleased with the outcome. Mostly just the process of working through the issue.

I didn’t get it to the exact experience I wanted but solved a bunch of issues and learned a lot along the way.

The interaction is a little confusing and the memes jump around a lot so it’s a bit hard to tell what is going on.

Also now trying to just run the examples I’m having issues. Wondering if it’s something with my connection or the ml5 unpkg link.

Things I’d like to still figure out
1) Why does it go so crazy when not in mirror mode?
2) How / Is there a way to stablize the poseNet elements (keypoints and skeleton) a bit?
3) Why can’t I load the model right now even with the examples? This connection process seems fragile, having to refresh both clients etc. Is there a way to make it more reliable?
4) In general what is the best way to merge code into this scaffold we’ve been using? I also had this mostly worked out here. But again had problems trying to add the code written from the posenet walkthrough video.

Work in progress code here.

p5 sketch