tvLand

06 Oct 2020 - Jake Sherwood

tvLand

tvLand

Phew… this project turned out to be a doozy.

Tasked to explore the getUserMedia method, I came up with tvLand. A place for a group “video” experience. Eventually I’d like to add audio and or chat options to this. But it’s already doing ALOT.

Each participant get’s they’re own TV. Each local user will be in the black TV on their screen. Clicking on that TV allows you to manipulate your image and / or change the audio track.

I spent way too much time looking at myself in little TV screens for this project. So I’m looking forward to class and sharing it with others. Hopefully it doesn’t crash…

New Things
1) I used the Underscore util lib for easier array handling
2) I used getImageData to get each pixel of the canvas and manipulate it. There are 4 manipulation modes that you choose by button and enact by clicking on the black TV.
3) I had to use mouse events to get my mouse X and Y so I could map it to a value for manipulation.
4) Manipulating canvas outside of p5.
5) Getting to know toDataURL… kinda.
6) Meeting the oom_reaper :(

I wanted to have every one in the class in their own TV. To start I found and image with a few TVs and added a few more in Photoshop. I cut out the screens so I could place each user’s image behind.

I also thought it would be cool to have the SMPTE TV color bars show if there was not a user assigned to the TV.

This presented a number of problems:
1) How do I keep each user in their own TV?
2) How do I reset things if a user leaves?
3) How do I update the toDataURL for each unique user?

Answer: Arrays, arrays, and more arrays.

To get the data in a way that I could easily traverse it, I put each user’s / TV’s data into an array as and object. To make working with the data in that format a bit easier I used the Underscore lib. It offers a bunch of functionality but I was mostly using it for array handling. From reading about it is seems pretty cool because it ads a bunch of functionality but doesn’t mess with any existing built in objects.

I mostly used the _.findLastIndex & _.contains methods. Both of which allow me to easily work with nested object arrays.

I use it on the client and also include it in my server code.

I’m adding both the socket.id and toDataURL for each user. I think check each time there is a new user to see if I already have them in my array. If not add them. Similarly I check for new .toDataURL and update accordingly.

function receiveData(data) {
    // check if id already in array
    if (myFriendTvsIndex < 0 && tv.id != myId) {
        console.log("friendsTvs did not include " + tv.id + "so adding");
        friendTvs.push({
            'id': tv.id,
            'toDataURL': tv.toDataURL
        });
    } 

I then update every user when the local user sends out a new set of data.

//on each image emit event check friendsTvs array for existing user id and set accordingly.
     socket.on('image', function (data, tvsid) {
            for (let i = 0; i < friendTvs.length; i++) {
                if (_.contains(friendTvs[i], tvsid.id)) {
                    friendTvs[i].toDataURL = tvsid.toDataURL;
                } //else console.log("socketID doesn't exist");
            }
	 })

Then I have an additional loadDataImg() func that is running in my draw() func on a SetTimeOut.

function draw() {
		if (!ctx1Clicked) ctx1.drawImage(myVideo, 0, 0, 174, 140);
		else {
			if (isManipulated) isManipulated = false;
			window.setTimeout(drawDataToCtx1, 1);
		}
		socket.emit('image', canvas1.toDataURL('image/webp', 1.0))
		loadDataImg();
		window.setTimeout(draw, 500);

	   }

        ...

// add toDataURL images to tvs
function loadDataImg() {
		for (let i = 1; i < friendTvs.length; i++) {
			let tempDiv = document.getElementById('image' + i);
			tempDiv.style.backgroundImage = "url('" + friendTvs[i].toDataURL + "')";
			if (tempDiv.classList.contains("bars")) {
				tempDiv.classList.remove("bars");
				tempDiv.classList.add("tvSize");
				tempDiv.classList.add(friendTvs[i].id);
			}

		}
	  }

Issues
1) One of the biggest issues I had was my dataToURL images coming in as 1/4 the size of my container. I tried numerous things to get it sorted, but was unsuccessful. I ended up with someone of a hack. I changed all the TV containers to divs so I could place the toDataURL images as background-images and adjust the background-size accordingly to make the images fit the TVs. This ended up working pretty well but still feels a bit hacky and I had to hard code a bunch of CSS.

As you can see from the code snippet above, I also added some CSS class handling to add or remove the TV bars gif. I also set the socket.id as a class which let me easily iterate through my elements and find a specific one.

2) I came face to face with the oom_reaper. Not surprising… look at all this madness…

tvland too may tvs

tvland full set

I never sorted out what was happening to cause the OOM. I did up the Droplet size which helped but it still happens. I was adding the dataToURL to an array on the server, but realized I didn’t need to but the problem persisted.

I was trying this which seemed to help a little but it still crashes sometimes.

--max-old-space-size=8192

I also added a number of errer catching functions on the server before I realized that it was an Out of Memory issue. I might schedule office hours to try to figure it out.

There are four manipulation modes: invert, weird, tint, and threshold. The latter three all react to mouse position. You select your mode by clicking the button and then clicking he black TV. Double clicking will set the manipulation and allow you to move the mouse without further changing it.

Getting my canvas manipulations to send updated images took some work. At first it would only take my last canvas image. Which ended up being the main problem. I was only sending the canvas the image once then trying to manipulate it. But I need to send it more often. This is the function I end ed up with. I basically get the image data and save it to a variable then I check for manipulation and set a boolean to prevent my manipulation from flipping back to the non-manipulated image. I then clear the canvas and update to my imageData variable. That is then used in my image emit. This function is called in the draw() at a fast rate.

function drawDataToCtx1() {
                // get canvas image again
                ctx1.drawImage(myVideo, 0, 0, 174, 140);
                // get canvas data
                let imageData = ctx1.getImageData(0, 0, canvas1.width, canvas1.height);
                // !isManipulated run selected filter
                if (!isManipulated) filters(filterMode, imageData.data);
                // set isManipulated = true so it doesn't keep flipping
                isManipulated = true;
                // clear previous canvas image
                ctx1.clearRect(0, 0, canvas1.width, canvas1.height);
                // write new imageData
                ctx1.putImageData(imageData, 0, 0);

            }

Also every time a new user joins or you click on the black TV and random channel surfing audio track is played. It would be nice to also send this across so we are all listening to the same thing… but this beast already has a lot of features and code…

tvLand - 4 participants.. all me

I pretty happy with how this turned out. It was a lot more work than I originally thought it would be. I can see this being extended to be a different way of doing video meetings.

There could be wa way to change the frame we’re all in but that would require me to fix my hack for placement.

All in all I’m please with tvLand.

Github code here.