tvLand2

14 Oct 2020 - Jake Sherwood

tvLand2 - just some tvs goats and a rainbow

tvLand2 - just some tvs goats and a rainbow

Cy and I worked together this week to build on my tvland(1) project from last week. Incorporating the simplepeer.js lib to move the UX closer to a video chat experience than the previous version was.

To make tvLand2 less ridged we decide to let the users arrange their tvs as desired.

Using this w3c example we were able to tie the draggable functionality into the peer creation code to allow the local user to move the tvs as desired.

tvland2 user test

We also took the pixel manipulation functions from tvland1 and updated them to work with this new setup.

In order to stream the manipulated video we initially needed to split the stream and the recombine the audio only stream with our manipulated canvas prior to sending to the peer.

abbreviated code:

// set up some initial Media Stream vars - these are important to happen before we getUserMedia
   canvasStream = canvas.captureStream(25); // 25 FPS
   myCstream = new MediaStream;

   // simplified canvas maniuplation func from http://html5doctor.com/video-canvas-magic/
   // modified to have mulitple filters

   function draw(v, c, bc, w, h) {
	   //manipulate canvas 
	   ....
	   // apply filters based on filterMode set with button
	   filters(filterMode, data);
	   ....
	   //draw back to canvas
	   c.putImageData(idata, 0, 0);
	   // Start over!
	   setTimeout(function () {
		   draw(v, c, bc, w, h);
	   }, 0);
   }

  ...
  
   // Prompt the user for permission, get the stream
   navigator.mediaDevices.getUserMedia(constraints).then(function (stream) {
		  
		  ...
		  
		   // separate audio and video so we can add audio to canvas prior to streaming to peers    
		   audioStream = new MediaStream(stream.getAudioTracks());
		   videoStream = new MediaStream(stream.getVideoTracks());

		 ...
		   checkMyVidLoaded();
		   // once stream has loaded add canvasStream and audioStream to myCstream (my Combinded stream)
		   myCstream.addTrack(canvasStream.getTracks()[0]);
		   myCstream.addTrack(audioStream.getTracks()[0]);
  
  ...
  )};
  
  ...
  
  // stream to peers on list 
  let simplepeer = new SimplePeerWrapper(
	  true, data[i], socket, myCstream // send combined stream
  );

On each new SimplePeerWrapper we’re creating a container div, video element, and tv frame image. The placement and size dimensions are random and the user can adjust placement after.

There are a few bugs and additional features we could get sorted.
1) We initially wanted to allow the local user to resize the tvs. But the dragging functionality plus the resizing was not working.
2) A bug when leaving (closing the window or refreshing), sometimes the tv of the peer does not get removed and we get a RTCError.

Uncaught RTCError: Transport channel closed

3) Adding a way for a user to mute themselves was also something we wanted to add but didn’t get to it.

For my midterm I hope to continue on this adding / fixing the things above as well as some of these new features:
1) Allow user to rotate tvs.
2) Change frame of peer tvs.
3) Drag and drop in new background image.
4) Add text chat functionality.
5) Fix random location issues.
6) Attempt to handle bandwidth issues.
7) Add chromakey effect.
9) Add particle effect.
10) Some sort of game aspect?
11) Better UI to tell the user how to interact.

Github code here.