The browser as a video rendering engine: Deep dive into time management

YouTube generated these captions and sometimes gets names, punctuation, and code wrong. Check the recording if something reads oddly.

0:01 — [Music] [Music] hey everyone today we're going to take a look at how to build a deterministic time api for playable content in the browser my name is jacques and i work for a company called mulc video we help sales people and marketers turn webinars sales demos and other long-form content into short shareable clips

0:32 — here you can see how our platform works and on our platform users can also add designs to their clips kind of like canva and they can add animated captions all from it within their browser this means that we needed a consistent way to sync up all of the different elements on the canvas when users previewed the clip in the browser as well as for when we rendered their projects to mp4 files because we also used the browser for rendering so first let's take a look at how we used to do this so here's a quick diagram showing how our state used to be

1:02 — set up we had two pieces of core state the one was just a boolean which represented whether or not the project was playing and the other was a time stamp representing when the project started playing and some of the components such as the time code and captions component would run a loop to calculate the current time based on when playback started and whether the project is playing and then in that loop it would take the current time calculate some other piece of state such as which word was active and react accordingly and for

1:34 — video elements we would just play or pause them depending on whether there is playing boolean was true or false and finally for elements like the export button that displayed the project's total time we would use the project duration property that was stored in the project state and this property was updated whenever we inserted or removed items so whenever we inserted an item for example in that insert item function we would also recalculate whatever the longest item was on the canvas and then

2:04 — set the project's duration to that value same for when we removed items so we had a completely disconnected mix of ways of handling our state we had some state that was kind of reliably and reactively derived from the core playback state such as the captions or the time code we had some imperative calls based on changes to that state such as the video elements and in the case of the project duration we had what i refer to as synced state where you

2:34 — have a function that makes the main update so for example adding an item but then also has a function call within that to update some sort of dependent state in a very sort of imperative direct way but this model started to become a problem when we started building a new feature called highlight reels and this feature would let users combine multiple clips into a single video so here you can see i have my uh single scene video video

3:04 — but i actually want to display multiple clips so let me quickly show you how this works so here you can see i have my project and i have a single clip with a specific title and design but i actually want to display another clip as well so i'm going to add another scene and drag in my other clip and change the title so the first clip is going to play it's going to show that design with that title and then later on once that clip's finished playing it's going to move on to the next one so basically just like

3:34 — multiple chapters or multiple projects almost that are strung together and our first attempt at this implementation was to keep going with the synced state route so we had a bunch of different scenes and each item on the canvas would belong to a specific scene and the length of the scene was stored as its own value meaning that whenever we added or removed an item from a scene we had to recalculate the scene's duration and update it and in simple cases that work just fine but what if we added another action to say paste a clip

4:07 — well that would also probably change the scene's duration right but because we didn't have that recalculation in that paste action we wouldn't actually update the scene duration so there could be inconsistencies in this state and not only did we store each scene's duration in its own state we also stored the active scene in its own state so when the user pressed play we had to compute what the active scene was meaning which items should display on the canvas but what if playback was

4:38 — paused and someone deleted a scene for example well then we'd have to add logic there too to also recalculate the uh the active scene or what if someone deleted an item from a scene which changed its duration which based on the current play time then changed the active scene there are just too many events and orders of events that could affect these computed sync state values that it was impossible to keep up and this implementation is just asking for bugs to appear as you change or add things and as we worked on the implementation

5:08 — that's exactly what happened it became unmanageable the problem with synced state the fundamental problem is that you have to consider every single action that might affect your state and make sure you compute and update the state correctly so we tried a different approach our goal was to build a single source of truth for time and for any value ui or side effect that could possibly be derived from that single time state and it had to work while playing back in real time and with any other kind of

5:39 — arbitrary change in time such as when someone seeks or scrubs so this is how we set up our new time api we had this global context provider or rather the context provider wrapped any components that needed access to time and there were two core pieces of state the one was a timestamp for when playback was started or it would be null if it's not playing so if it's paused or if it stopped and then the second value would be the play offset meaning the last time code that the project was seeked to and based

6:09 — on that we can derive whatever the current time should be for the project and along with this provider we created a use playback hook which let any component subscribe to the current time whether or not the project is playing back and a few other helpful functions while necessary functions like playing pausing seeking and so on and this was really great this gave us a very very reliable interface for reacting to the current time because the idea was here to let react do the hard

6:39 — work for us because well react is very very good at rerunning functions whenever dependent state changes that's kind of what it does right but it was almost too good because we soon ran into performance issues so because of the way our api was set up any component that used this used playback hook would receive the current time value but the problem was that that current time value changed every single frame so any component that used that state and that was pretty much every component on the canvas would re-render every single

7:11 — frame and in react re-rendering is slow because you have to rerun the entire render function not just like the small bit of the ui that depends on the time anything else within that component's got to re-render as well that would cause most of that components children to re-render and then react has to also do a diff to figure out if anything needs to actually change in the dom the point is it's not advisable to rely on react for rendering things at 60 frames a second but needing to re-render the component

7:41 — every frame isn't really necessary because even though our current time might change every single frame in our captions example the index of our current word wouldn't that would only change i don't know once every 300 milliseconds or so because words are generally longer than one frame right and previously this wasn't an issue because we were just running a loop in every component that needed the time and we ran it outside of the react render loop and then whenever any sort of derived state so the active word for

8:11 — example would change then we would call a set state to re-render our component so it was pretty efficient so then we thought instead of having all these different loops scattered around our code base like independently figuring out what the current time is what if we had one central loop that calculated the current time and then components could efficiently react to that and subscribe to that without them all having to re-render every single frame and out of this came to standardize hooks the first one we actually borrowed

8:41 — from the react redux library and we called it use time selector so here's the idea we have a request animation frame which when the project is playing runs every single frame that calculates the current time and that dispatches a time change event and then any component that wants to subscribe to that current time change could do so by calling the use time selector and then that component would then run a function every single frame or whenever time changed to determine a

9:12 — new resultant value and if that value changed it would re-render and as you can see the only part of this whole flow that actually involves react i.e the expense of computation is the last step and that happens a lot less often than everything else does so the idea behind a selector is to run a cheap computation often and then only trigger a more expensive computation whenever that cheap computation returns a new result the expensive computation being a re-render in this case

9:44 — and another thing that was very cool about the selector was that we would call the function whenever current time changed because of any reason so either because of playback so whenever that request animation frame loop was called while playing or whenever the play started value changed meaning regardless of whether the user was playing back the project seeking or seeking while playing that selector would be called and any dependent ui would rerun their calculations and update

10:14 — or even if someone deleted like the scene in front of the currently active scene and the active word that should have been highlighted uh in that caption changed because that scene was deleted it all just ran reliably no matter why the current time changed and we also had to make sure that the videos were played and paused and seeked to the right time depending on the global time of the project so the second hook that we created is borrowed from react and we call it use time effect and

10:44 — this hook quite simply lets you fire side effects whenever the current time changes so in here we could simply seek the video element to the correct position and it kind of works like use time selector where you can pass it a function that will be called whenever current time changes the only difference is it doesn't return anything because it's meant for side effects in fact internally use time selector is actually built using use time effect pretty neat right and because this is such a foundational

11:14 — piece of our application we wanted to make sure that it always worked meaning that we probably needed some tests at first i thought it would be a bit of a challenge testing all of this because it involves behavior over time and request animation frame but it ended up being surprisingly easy so let's say we wanted to test whether playing and pausing worked so ideally what we would do is we would test our hook the way that it's used in real life right because testing your software the way someone is going to use it

11:44 — or the way some other code is going to use it is the best way to test it right so hypothetically what we would do is we would set up a test like this that would render our use playback hook it start playing it'll wait a second and then check the current time to make sure that it is set to a second so we're testing that it started playing and that it's like moving forward the correct amount of time then we would pause and then maybe wait like another second and then make sure that it actually paused after a second

12:15 — so make sure that the current time is still one second instead of two seconds pretty simple right that's exactly how it would be used and how we would expect it to behave but we don't actually want this test to run for two seconds so we needed some sort of a way to run this test without real time being a factor so because this hook is based on the actual javascript date we needed a way to mock the date so we use this library called mock date

12:45 — which works with jest and how this works is you render the hook and then we set the time to zero so if you called date.now in javascript you would get zero back we start playing and then we can time travel forward by a second by setting the date to a thousand and then we make sure that it's playing so make sure that the current time is a thousand then it moved forward the right amount of time pause and then time travel forward another second so set the actual time to two

13:17 — seconds and then again make sure that it's paused so the same test but not involving any real time which is pretty cool the other thing that we were missing was the fact that our use time effect and our use time selector hooks depended on request animation frame while the project was playing and we couldn't actually run that ingest because just doesn't support request animation frame so what we did was we replaced request animation frame with a set timeout and used just use fake timers feature so

13:48 — this turned off uh real time in timeouts ingest so let's take a look at how this actually works so first before each test we enable the use of fake timers and then we replace the real request animation frame implementation with a custom one that just calls a set timeout that resolves after 50 milliseconds and then in our test so here you can see we have a test that mounts the use time effect hook and has some sort of test function and we want to make sure

14:20 — that this test function is called whenever the time changes so first we set the time to zero just like we did in our previous test then we start the playback so we use our use playback hook here and then we advance the time by 50 milliseconds and also trigger a frame by telling jest to also time travel 50 milliseconds which would trigger one uh core of our set timeout and this would give us a way to frame by frame advanced time

14:50 — so that we can test things very very granularly to test that our test callback was called once by our use time effect for that one frame that we advanced it by and using this time traveling principle we could test anything that depended on our time system including making sure videos were being seeked to the right time and that the correct caption word was being highlighted and all of the tests could run faster than real time now one tricky thing that we still had issues with was keeping our video elements synced up with the rest of the

15:20 — items because what happens is you press play and we record the play started value immediately right so time starts to advance but because the html5 video elements on the page take a few milliseconds to start or they could be buffering or something or the browser could just be busy or there could be bluetooth delays all manner of reasons the video elements might actually start playing quite a bit later than than our internal time is storing so there were a few solutions that we thought of there but the simplest

15:50 — solution and the one that we're using right now is we start playing and the time starts advancing but then all of the video elements on the page call back to the time system periodically to tell the time system what their internal state is so if there's any kind of drift between the two the video element is going to tell the time state to actually seek to the correct time so basically the video elements start playing because of the time system starting to play and then now and

16:20 — then they call back to actually update their real time and sync the time system back up with a video element but there is a very promising proposed web standard called the web timing api and we're going to be experimenting with that pretty soon we haven't looked at it yet because it's still very early and there is a uh javascript implementation of it but we haven't really looked at it much i just thought i'd flag it though in case anyone else wanted to play around with it because it does seem like a really really good solution to keeping video elements in sync

16:51 — now looking towards the future the way that we've set up this time system actually lets us do pretty interesting things the first and what we're currently working on is something called frame by frame rendering so the way our recording works now is we open up the canvas in a headless browser we play it back and screen record the page and this has actually worked really well for us in the last while but we've had quite a lot of quality and frame rate issues just because we're dependent on the browser and the recording system always performing perfectly because everything's got to happen in real time

17:23 — right but right now we're actually rebuilding our rendering engine to render videos frame by frame so what this means is we're opening up the app in a headless browser just like we would for screen recording it but instead of screen recording it we take a screenshot of every single frame now without this deterministic time system we wouldn't be able to just like call a function on the page to advance the time frame by frame we wouldn't be able to just arbitrarily seek to a specific time or frame in our video and be confident that

17:55 — everything displays correctly another thing we'll be adding soon is animations again we can leverage the time system we've built to create interpolation based animations where for a given timestamp or for a given frame we output a specific css value so for example from time 0 to time 200 milliseconds we could interpolate an opacity value from 0 to 1 and that gives us a fade in animation over 200 milliseconds and what's cool about this because again it doesn't rely on css

18:26 — animations like it's completely stateless this will work with our frame by frame rendering so what have we learned from all this well we've learned that the best way to handle time in the browser is to derive the properties of your ui elements directly from your time in a declarative way and that the best way to structure your time system is to build a single source of truth for your time and a standard and centralized way to derive values and run side effects from changes to that time we've also learned that you should do

18:56 — this by taking advantage of the library that you're using to build your ui be it react or vue or svelt or even angular also we've learned that you should be conscious about not overusing your ui libraries and leaving often running calculations outside of the expensive render cycle to more efficiently update our ui and finally we've learned that the most important thing to focus on is giving the user a delightful performant and most importantly reliable interface to

19:26 — work with so that's been our journey to creating a time system for the web i'll post these slides on twitter if anyone wants to reference them later so you can find them there and one last thing i'll mention uh is that we're looking for talented front-end and video engineers to join our team we've got a lot of interesting problems to solve and we'd love for you to join us in solving them uh if you want to have a chat with us about that you can get in touch directly with me or with milk milkvideo's founder lenny on twitter thank you so much i hope you've uh

19:56 — learned something along with me or at least had some ideas um and i'll see you on twitter [Music]