Eli Mallon 0:04 Yeah. 0:06 All right. 0:06 So that's how Streamplace VOD works. 0:09 I hope you all enjoyed the presentation. 0:11 My name's Eli Mallen. 0:13 I'm the CEO and founder of Streamplace. 0:15 I don't care about my carabiner update version. 0:21 So the crux of this talk was If you are going to make a system that has both VOD and live— can that be muted at all? 0:31 The— you should probably start with live. 0:33 Live video is more difficult than VOD. 0:35 It is much more difficult— it is much easier to generalize live video to VOD than it is to go the opposite direction. 0:41 So we started out with live. 0:43 We're pretty happy with that. 0:45 And, you know, knock on wood. 0:47 Of course, as soon as I walked out of the other room, people started reporting playback issues over there. 0:50 But overall, everything's been going really well. 0:53 At all of this. 0:55 So I'll start by sort of explaining how Streamplace— a quick explanation of how Streamplace Live works, because I've talked about this a lot before. 1:03 And then we'll talk about how we're adapting that for a decentralized VOD primitive. 1:09 What does it mean to have something like, you know, what would a decentralized YouTube look like? 1:15 What things do you need in order to make that happen? 1:20 So yeah, this is just about Streamplace. 1:22 There's been 3 Streamplace third-party clients created since the beginning of this conference. 1:30 So I don't know if that reflects poorly on our front end or if it just represents the entrepreneurial spirit of the app proto community. 1:38 But this one's from jack.xyz. 1:41 And it shows all of the streams at once, really giving the Streamplace servers the maximum workout to give everybody what they're looking for there. 1:50 Before we do anything else, I found this MP4 file on the ground. 1:55 Let's see what it is. 2:01 You can all imagine the great noises Trabio is making here. 2:06 What's going to happen? 2:07 What's going to happen? 2:14 Wait for it. 2:16 Yeah, we got them. 2:17 Yeah. 2:19 So that MP4 file I found on the ground turned out to be my glorious victory over Trabio. 2:25 So how do I know that? 2:26 How do I know it's Eli's live stream that happened there? 2:30 And that has to do with Stream Places signing format. 2:35 So we needed to— so decentralized VOD is something that's kind of existed before, right? 2:41 You could argue— This will be a popular opinion. 2:43 You could argue video NFTs are kind of like a decentralized VOD, right, where you've got an IPFS reference to something that's like out there and not controlled by any one party. 2:53 Lots of other like IPFS-based, like there's like DAT-based live streaming that maybe falls into that. 2:59 But for the most part, we needed to invent what a decentralized live stream is. 3:03 So what we did was we took the first thing that happens is you generate a signing key, same as the one kind that's used to secure your app proto repository. 3:13 We then take that signing key. 3:16 You then configure your OBS instance just like we're doing here to stream into StreamPlace. 3:23 You actually provide it with that signing key so it has the capability of signing on your behalf. 3:27 If you really care about your key custody, you should run StreamPlace locally and then syndicate it out to a server. 3:34 Then this is basically every live stream you've ever watched. 3:37 If you haven't messed with video infrastructure before, is just a bunch of short video files that are thrown into your face in sequence, and it provides the illusion of a livestream. 3:46 So that's how we do it. 3:48 Each MP4 file comes in. 3:50 We give each one of them an embedded signature using technology from the C2PA, the Coalition for Content Provenance and Authenticity. 3:59 Cool format. 4:00 It provides a mechanism for giving you an MP4 file that contains an embedded signature. 4:06 Over a hash of all of that MP4 file, except for that part of the signature. 4:11 That's really tricky to get right. 4:12 And that's one of the things I really like about the CTPA's tooling is you've got this— a lot of decentralized media projects have external signatures. 4:23 So you might have— this actually included, like, Blue Sky videos, right? 4:26 How do you know it's my Blue Sky video? 4:29 It's because there's a blob out there that is referenced by a record in my repository. 4:34 And my repository has a signature on the root commit. 4:37 So that's how you know it's mine, right? 4:39 What I like about this format is I can have— this is the MP4 file on the ground example I just gave, right? 4:44 You can have arbitrary MP4 file on your computer. 4:47 And all of that signing and provenance data is intact and embedded in the MP4 file. 4:51 So I like that a lot. 4:52 It's something we really wanted to preserve as we moved from live to VOD. 4:57 And lots of very cool metadata in here. 5:00 We're not doing too much with this mechanism yet. 5:02 I spent a long time talking to Trezzy yesterday about embedding this. 5:07 Part of the embedded metadata here should be that I'm playing Silksong, right? 5:10 And then if you care about Silksong, you can watch that livestream and get that data from, like, pretty much any direction, right? 5:17 Like, you can download it from sketchiestwebsiteintheworld.ru and still be confident that you're looking at the right thing. 5:25 And then I'm playing Silksong. 5:27 Those— yeah, that's this example. 5:30 I have— yeah, we've made Streamplace as easy to run as possible. 5:33 So you can run your own Streamplace node, syndicate my streams. 5:38 They get copied over in this sort of trustless, unauthenticated way. 5:41 And then you see it over there. 5:43 So the thinking here is we probably won't beat Twitch by making a bigger data center or buying more bandwidth than Amazon. 5:51 But we might be able to beat them by following the example of BitTorrent. 5:54 Right? 5:55 That you can have lots of decentralized infrastructure that's replicating all of this, but through the magic of digital signatures, you know that the video is authentic. 6:04 So what's the problem? 6:07 This isn't very convenient. 6:09 So when I say slice the video up into 1-second MP4 files, I'm being very literal about that. 6:13 If you run Streamplace on your computer, you can navigate to this and see lots and lots and lots of 1-second MP4 files. 6:19 And when a computer is processing them all and feeding to you as a live stream, that's OK. 6:23 Afterward, when you want to work with this as a VOD, it's a pain in the ass. 6:27 So we wanted— this was one of the design goals of VOD for us is how do we handle that? 6:34 And unfortunately, because we used regular MP4 files, they're non-concatenable. 6:39 You can't take two MP4 files, squish all the bytes together, and get valid output. 6:45 That's just not how MP4 works. 6:47 It's not valid data there. 6:49 So you'd have to mux it, and the signature that you get there wouldn't match the input. 6:55 And all of a sudden, you've broken this entire cool BitTorrenty provenance mechanism that I was just describing. 7:01 So this is the other— this is the other reason we were already looking at forking C2PA. 7:09 So here we have a list of signing algorithms. 7:13 As you can see here, we have— the good and virtuous ECDSA signature over a SHA-256 hash over the p256 curve, but not over the secp256-k1 curve, the evil shady curve that's used inside BitTorrent and Ethereum and App Proto, right? 7:40 So I think— My theory here is that somebody in Adobe had to sell some exec that they weren't making blockchain software here. 7:49 So we didn't do that kind of signing. 7:51 We only did this kind of signing. 7:52 So technically, for as long as Stream Places existed, it hasn't been C2PA compatible. 7:58 And I had been looking for a while to formalize that. 8:00 So— oh, yeah. 8:02 This is the other— so this is the other— so Adobe's out there right now lobbying, especially European governments and the state of California. 8:14 California to require provenance data on everything that you post on social media. 8:20 This is the reason for the CTPA's existence, getting us ready for a world filled with deepfakes. 8:24 This is not inherently a bad idea. 8:28 You know, once anybody can deepfake anything, how do you ever trust what you're looking at? 8:32 No one technical mechanism can solve that problem. 8:35 But one way you could be more confident is if there was provenance data embedded in the livestream, right? 8:40 Where it says, oh, this video was taken from Eli's camera and then edited using this software and makes it way— It makes its way everywhere, right? 8:47 But the reason Adobe is doing this lobbying is because they want to try and do some regulatory capture, right? 8:52 So they want to sell governments these expensive Creative Cloud subscriptions and be like the only software that you could possibly use in order to do all of that stuff. 9:01 Also, the part I think is really funny is they want you to go pay like a CA like $107 a month for a signing certificate. 9:09 Like Let's Encrypt and everything never happened. 9:12 And of course, like— in the @proto and decentralized web communities, we have DIDs and totally other stacks of how we would handle some of this stuff, right? 9:23 How we would handle key custody, how we would handle authenticity, that kind of thing. 9:27 PLC directory's got its own sort of take on everything. 9:30 So yeah, some good ideas. 9:34 Needs to be modernized and not try and pay these vendors a bunch of money for certificates. 9:41 I said all of these things out loud a lot, and the folks, the good folks at the IPFS Foundation and the Dazzle Project were very, very receptive to it. 9:50 So they've given us a grant. 9:52 Everything you see so far, everything you see past this point in the presentation is a result of the grant that we got from the IPFS Foundation. 10:01 So we're working on two specs with them, Stupa and Muxel. 10:07 STUPA is the Simple Standard for Provenance and Authenticity. 10:10 This is the simpler of the two standards. 10:13 It's very much— thank you, Ted. 10:15 Look at that. 10:15 Ted got me a glass of water. 10:16 Everybody thank Ted. 10:17 That's so awesome. 10:23 STUPA is our— I don't know what makes a fork a hostile fork, but it's definitely a fork of the CTPA that takes out all the things— it actually doesn't take out anything. 10:34 Like, if you want to pay DigiCert $200 a month, you totally can. 10:37 But we have some different opinions about that. 10:39 We have DIDs, right? 10:40 So instead of signing it with some cert that you're given by a CA, you sign it with a signing key that has a provenance chain back to your @proto identity, right? 10:52 And then we also make it consistent with Dazzle. 10:57 The C2PA signatures— C2PA manifests, rather, already use CBOR. 11:01 So we're just upgrading it to use Drizzle. 11:04 Drizzle dazzles, a very strict subset of CBOR used in appproto, right? 11:10 That's, that's Stupa. 11:10 That's all of Stupa. 11:11 The rest of the presentation's about Muxel, 'cause it's way more complicated. 11:14 This is just a little change that we're making here. 11:16 Would love if they upstreamed it. 11:18 Would love if there were DIDs and K256 signing in, in, in C2PA. 11:25 So let's talk about the design goals for Muxel. 11:29 This is basically what changes do we need to make to those MP4 files that I described before in order to make them appropriate for VOD, so you don't have thousands and thousands of MP4 files on your computer after a long livestream. 11:43 So we want to keep the property that they're self-certifying. 11:46 It's awesome that you can just find one of these files and then go verify its provenance chain. 11:51 Shouldn't be— you shouldn't require external data, right? 11:55 As a design goal, we want to support livestream to VOD so that what video engineers call DVR. 12:00 Basically, if you're watching a live stream, that you can seek back and see what happened 1 minute ago, 2 minutes ago, 3 minutes ago, that kind of thing. 12:08 It would be nice if it's a useful format. 12:10 I don't want to give you some opaque CBOUR blob or something. 12:13 A design goal of this is that at the end of it, you have— at the end of a 24-hour live stream, you should have a 24-hour MP4 file on your computer. 12:21 Easiest to work with, maximally compatible, preserves all of the provenance information I was describing. 12:27 And it should be streamable with minimal overhead. 12:30 I'm not going to get too deep into media over QUIC in this presentation, though I would love to. 12:36 But basically, we don't want to do a bunch of extra effort to go from the canonical format that we're going to use for all of this to actually streaming it to users. 12:49 We shouldn't need to re-encode or anything like that. 12:53 Unfortunately, that is an impossible series of constraints. 12:58 Fundamentally, the reason being that if the output is going to be just an MP4 file on your computer, that is completely different than the format that Media over QUIC uses that's going to be useful for sending things to users, in particular with things like multiple tracks. 13:11 We really want to do multitrack video, multitrack audio, 720p, a bunch of other— 1080p, different audio formats, AAC and Opus for different things. 13:22 All of that. 13:23 Complexity we would like to be able to package in this one file. 13:25 And that, like, skyrockets the bandwidth that you would use to send it out to people, right? 13:30 You don't want to send them every rendition. 13:32 You want to send them just one. 13:34 So how do we get around that? 13:37 And the answer is we mux. 13:39 This is why it's called Muxl. 13:43 So the key insight here is that the archival format So this is to say the MP4 file on your computer, the streaming format, which is the format the video's in when it goes over Media Over QUIC and streams to a user, and the self-certifying format that has this intact signature that you use to validate that the media is who it says it is and get all the nice metadata. 14:10 We— they don't all have to be the same format. 14:15 As long as we can deterministically mux between them. 14:20 As long as if you have one of these formats, there is a deterministic algorithm you can use to turn it into any one of these other formats. 14:28 So we've accomplished this with WASM primarily. 14:31 WASM, especially the latest version of WASM, has a fully deterministic profile. 14:35 Even short of that, you know, Rust compiled to WASM and then you pipe some bytes through it, very, very likely to get the same bytes out through the output. 14:44 So this is— yeah. 14:47 So let's get into the different formats that we're using inside of here. 14:50 I'm going to get deep into MP4 file specifics. 14:55 I wouldn't wish this upon anyone, having to dive this deep into the MP4 specs. 15:00 I had to pay $300 for a PDF file. 15:05 So yes, this is the whole point of Stream Places. 15:09 These are the hard work we do, the hard problems we solve, so none of you You have to, right? 15:15 So the base primitives here that are part of Muxl. 15:21 First, we have video metadata. 15:25 I borrow a lot of this terminology from a spec called Hang. 15:29 That's from Luke Curley, who created the Media Over QUIC spec. 15:35 So this is— catalog is his term for this. 15:38 But this is all of the metadata that you need to understand the video. 15:42 Video decoders won't just accept some random video bytes. 15:46 You have to say, no, these video bytes are, for example, 1920 by 1080 at 60 FPS. 15:52 You've got the time scale and then a canonical track ID for everything that goes into this, right? 15:57 How many channels is the audio? 15:59 I don't have it listed in here, but this would have stuff like what language is the audio. 16:02 If we've got the audio translated into several different languages, that would be included in this metadata. 16:07 This is nice to separate out because from a low-latency transmission, perspective, you don't want to repeat this stuff, right? 16:15 You don't want to say, hey, 1080p video, 1080p video. 16:18 Like, you know it's 1080p video because the last 500 segments were 1080p video, right? 16:22 So we send the— we can send this once at the start and not have to repeat it. 16:27 I'm not quite getting into the app.proto stuff here, but just at the bottom there, a little teaser of what this might look like if you wanted to put it in an app.proto repository. 16:36 This could be a place.stream.muxl.catalog record. 16:40 We'll get to how it actually works in a minute. 16:43 This lets us make init segments. 16:45 Now we're talking about MP4 atoms. 16:47 This is fun. 16:50 So this data represented as like JSON or CBORE or whatever and stored in your app.proto repository is enough to create— actually deterministically create the start of the MP4 file, which is called the init segment, the very beginning of the MP4 file that contains the file type atom telling you This is an MP4 file. 17:11 And the move atom that describes, for example, the duration of the video, what all the tracks are, there's a bunch of— this is like the MP4 encoding of all that metadata that I just described. 17:22 This is the beginning of the Muxl file in its segment. 17:26 We then split up the video into Muxl segments. 17:33 This, again, is borrowed— this is— pretty much verbatim Hang CMF, which is part of the media over QuickSpec, which is awesome because that means it's going to stream very efficiently to users eventually. 17:44 And so this is just repeated. 17:46 Every frame gets its own what's called a MOOF and an MDAT, so little segments, little slivers of the MP4 file that represent each frame, each video or audio frame, essentially. 18:00 Easy enough, but we're going to do a lot with these in a minute. 18:04 And then we go back to that STUPA specification I was talking about a second ago, the CTPA stuff. 18:13 So this is— we can take— this is the canonical signed format. 18:18 This is the thing that you— when you get this format, you can validate the signature, validate the provenance claim, know this is actually, definitely Eli's CTPA-signed video file. 18:30 This is actually me beating Trabio or whatever. 18:33 And this is— this is great. 18:38 This is not necessarily the best format for archival and playback, right? 18:43 So one key insight and, like, the, like, eureka moment I had when making this is, like, okay, so this is, like, the canonical thing, but you wouldn't ever write this to your hard drive. 18:54 Or if you did because you're repeating the beginning of the file, the file type stuff, this would get you right back to 1-second MP4 files on your computer over and over and over, right? 19:05 So while this is like— you consider this like the base unit of StreamPlay streaming or a VOD or what have you, this— yeah, you'd never actually write one of these to the hard drive unless you're debugging or something like that, right? 19:17 What would you write to the hard drive, you ask? 19:20 You would write a Muxl archive. 19:23 Blake3. 19:24 I know— hang out with that proto people. 19:26 I know a lot of protocol engineers. 19:28 Every one of them has had a moment where they take me aside and they're like, Eli, you should really be using Blake3 for this stuff. 19:35 And I agree with them eventually. 19:38 So this is what you would write to your hard drive. 19:41 So this is the init segment. 19:43 This is the part where you say it's 1080p video. 19:46 Then you have all of these. 19:49 Each of these is what's called a GOP segment, so a keyframe of video. 19:52 Video followed by a bunch of other frames and then a keyframe and then a bunch of other frames. 19:56 And the cool thing about this is we've totally messed up the signature now, right? 20:03 Because we took a— the signature is over this file with an init segment and a video GOP and an audio GOP and then a manifest, right? 20:11 This is the thing that we signed. 20:13 This doesn't look like that at all. 20:15 I've even just kind of put the manifest at the end there. 20:17 But because we have deterministic muxing tools, we can recover this. 20:22 We can recover the signed canonical thing, right? 20:27 This format, however, is really, really useful for VOD. 20:31 Because if you are— I'll show you why. 20:36 See? 20:37 It's like this. 20:39 So this is an HLS playlist. 20:42 This is how you— you don't know it, but most of the time you've played video on the internet, you've had one of these happening in the background. 20:51 And what this does is takes all of those— this is just the video playlist. 20:55 This just tells you how to do the video. 20:56 The audio is referenced in a different manifest. 21:01 And basically what this gives you is a byte range in an index into that file. 21:08 So you can say this here says this is 1 second of video. 21:11 You can find it at this index of this file. 21:12 This is 1 second of video. 21:13 You can find it at this index of this file. 21:16 And this is all you need for efficient VOD playback. 21:19 If you're doing it live, There's disadvantages to doing it this way. 21:22 You probably want to use Media over QUIC and some of the other formats that we talked about. 21:25 But for VOD, this is really, really good. 21:28 And we've achieved one of our design goals because all of these— it's just referencing different byte ranges in archive.muxl.mp4, right? 21:37 So we've— the thing that is— I've been saying on your computer, but the thing for video, the thing that's in your app proto repository is just one big MP4 file. 21:48 That's really easy to understand. 21:52 The other— the other thing we get from this is the Blake3 SID. 22:00 The magic of Blake3, why was it that all those video engineers were telling me that I needed to use Blake3? 22:05 It's because this Blake3 hash is a Merkle tree over this file that you see here, which means you can request just that part that says track 1, GOP 1 there. 22:18 And through cryptographic verification, you can be 100% confident that that is the correct data that you've pulled out of there, right? 22:27 You can verify, yes, this is in fact part of this Blake3 SID, which is really, really useful when you're doing something like a CDN. 22:34 Because then you can make all these— you can have— be backing it with just one big file. 22:40 You can make these byte range requests. 22:42 You still get the cryptographic integrity that like, Oh yes, that hash matches what I saw in Eli's @proto repository. 22:50 So this is a really, really good format for this. 22:54 Cool. 22:54 So yes, 3 different formats that we have here. 23:02 So the archival format, which is the big MP4 file that goes in your @proto repository, the self-certifying format, which is the part that has the CTPA signature attached. 23:13 And the streaming format, which is just when you're actually sending it out, you just send— you just send this part, right? 23:19 You stream this stuff, each frame of video and audio directly to users. 23:25 So for all the @proto people that didn't care about all the different structures of the MP4 file, how do you use all of this? 23:33 We're not quite ready for it, but the very first VODs available on StreamPlace are going to follow this format, and it's going to be all of the talks from this conference. 23:41 They will be published to— at ProtoRepository. 23:44 And you'll be able to work with them. 23:45 This is a little preview of what this looks like. 23:51 So we have a— yeah. 23:54 So the man— as you might expect, the lexicon type for this is a place.stream.video. 23:59 It's got things like creator, some of this metadata in here. 24:04 I'm pretending this is the one for this talk. 24:06 I don't have a VOD for this talk, because I'm still doing this talk. 24:09 So you don't know what the hash is yet. 24:12 It's got— it's just, as I said, it's just a blob. 24:14 It's a video MP4 file. 24:16 But it is very, very easily cacheable. 24:19 You've got a nanosecond duration in there. 24:23 We— the design constraints for this, I've been targeting that the maximum length of one of these is going to be 24 hours, because— At a certain point, it gets sort of ridiculous. 24:33 Yeah, yeah, yeah. 24:35 And I don't want to, like, overflow the nanosecond duration. 24:37 But if you've got a 24/7 livestream, that seems like a reasonable primitive that, like, each day of streaming that you would stream into would become its own VOD with a 24-hour duration, right? 24:47 And then it's got a livestream reference. 24:49 So, yeah. 24:51 So all of this will be available— yeah. 24:53 I was hoping to have all of this available. 24:54 And I was like, oh, I'm going to announce it in this talk. 24:56 And it's going to be big. 24:57 And we're, like, really, really close. 24:58 But I didn't want to— push a bunch of changes to Streamplace right before we did all the live streaming and screw everything else up, right? 25:04 So I did try to do that. 25:09 We did that. 25:10 I did at about 7:10 this morning. 25:13 I'm like, all right, let's push the Muxl changes. 25:15 Let's see. 25:16 And then immediately that laptop couldn't stream into Streamplace anymore for whatever reason. 25:21 So I was like, nope, nope, back, revert. 25:23 We're going to— we'll do it. 25:24 We'll do it after. 25:26 So that's all by way of saying this is all very, very close. 25:30 And we're hoping to give it to the community very, very soon. 25:35 And yeah, and then build it with everybody, right? 25:37 So we don't have a frontend for it. 25:38 I'm going to challenge the— we're going to do a little VOD jam and say, hey, @proto-community, here's the videos. 25:43 Build me a frontend for it. 25:44 Given that without me asking, we got 3 frontends for Streamplace in the last 3 days, I feel like in the vibe coding era, we're going to get some submissions if you submit one of these. 25:55 You're going to earn a badge next to your chat name in Streamplace from now until the end of time. 26:03 So, yeah. 26:05 That's it. 26:06 That's how Streamplace works VOD. 26:08 We have a few online questions. 26:19 Great. 26:20 Which I don't fully understand. 26:21 Awesome. 26:21 So there was a question about format. 26:25 Why MP4? 26:27 Yeah. 26:27 And not something like ABC1 or something? 26:33 Yeah, yeah. 26:34 So the big alternative— why MP4? 26:37 There's a couple of reasons. 26:39 One, because it's the most ubiquitous media format. 26:42 And that's the most highly compatible across everything. 26:46 The second is— It gives us the— the media over QUIC stuff tends to support it a lot already. 26:55 Also, the FM— so this is like CMAF. 26:58 Back to this thing. 27:00 This is like CMAF MP4 playback. 27:02 And the advantage here is it lets you separate out— it gives you that lack of redundancy that I was describing where the init segment isn't— there are other formats that embed all of that init data in every segment. 27:12 But we don't actually want that because it's redundant. 27:15 For a long video. 27:16 So yeah, those reasons. 27:17 [Speaker:AUDIENCE] Why VP9 instead of H.264? 27:22 [Speaker:STEVE GROVE] Sure, sure, sure. 27:24 The question was why VP9 instead of— or why H.264 instead of VP9 or AV1. 27:30 No reason other than maximum compatibility right now. 27:32 We really, really want AV1 in StreamPlace. 27:37 We fully intend to support AV1 as much as we do H.264, especially when we do stuff like Like, I want Streamplace to do a lot of things that no other streaming platforms do. 27:46 Like, for example, like 4K 120 FPS AV1 video would be so sick. 27:53 You can't even do it. 27:54 I've tried to do it with H.264. 27:57 It just like doesn't— no encoder can even come close to handling that. 28:01 So yeah, yeah. 28:02 In terms of codec support, we want to support like every codec that it makes sense to. 28:05 We just started with this because we're not quite in a world where I could probably get away with AV1 only and we would have needed an H.264 backup anyway. 28:13 So in that context, that's why we started with H.264. 28:15 But yeah, definitely want to support lots of codecs. 28:24 Thank you.