Daniel Holmgren 0:01 That was fantastic, Trezzy. 0:03 I hope a bunch of people are lighting up their scrapers and attacking so many other industries. 0:10 The fire safety talk was one that would pair really nicely with yours because they had the same pain. 0:16 Thank you. Speaker B 0:17 Yep. Daniel Holmgren 0:20 Dame. 0:24 Dame? 0:26 Oh, it's you? 0:31 Dame, Dan, it's fine. 0:35 Come on down. 0:38 This convenient slider doesn't work for me. Speaker B 0:52 Probably at the podium, because I'll be changing slides a lot. Daniel Holmgren 0:57 This is not Dame. 0:59 It's Dan, and he's going to talk to us about protocol governance and hard decentralization. Speaker B 1:09 Hello? 1:12 Hello? 1:17 Okay, hey everybody. 1:18 My name is Daniel Holmgren. 1:19 Uh, I'm the head of protocol at Bluesky. 1:22 Uh, just a heads up, I've been losing my voice all week. 1:25 I thought maybe if I drank 6 Modellos at the afterparty last night that would loosen it up, and the jury's out on if that's working or not, so I'm hoping that I'm good for the, for this talk. 1:36 Uh, to tell you a little bit about, so I, I lead protocol, uh, development and design at Bluesky. 1:42 To tell you a little bit about our team, right now it's technically just me and Brian Newbold on it. 1:46 Although we collaborate closely with other folks from the Atmosphere team. 1:49 Without going into our whole org structure, the reason that we split out this protocol department is that protocol design and government, governance just looks pretty different from running a startup. 1:59 You want to make slower, intentional decisions and you don't want it to get caught up in sort of the, the market and the product lifecycle of the startup. 2:07 So we put up a little bit of an internal firewall for that department so we can go a little bit slower and we can be focused on what we really care about with the protocol. 2:16 Which is the emergence of a functioning, resilient, multi-stakeholder atmosphere. 2:24 So I want to start with a provocation, which is that it's logically impossible for Bluesky to decentralize the network. 2:30 We really want the network to be decentralized. 2:32 It is impossible for us to do it because we are the party, right? 2:37 And even if we funded somebody else to do it, they're still downstream of us. 2:40 And it's not actually— do they actually have the authority? 2:42 Do they actually have the power? 2:43 In We're the central entity, we can't decentralize it. 2:49 We can provide the tools and structures by which to decentralize it, but we need others to use those tools and structures to take control, power, and authority in the network to actually make this a resilient multi-stakeholder network. 3:03 Uh, so that's what I'm going to be talking about today is those tools and structures and how we address that. 3:07 The agenda is pretty brief. 3:09 We're going to talk about the IETF, some recent developments with PLC governance and then hard decentralization, which is kind of getting into the actual threats of what having a big entity in the middle of the ecosystem could do. 3:22 Uh, before we do that, I'm on the big stage and I wanted to talk about the real governance question, which is how do we capitalize and stylize AT protocol? 3:31 Again, this is a multi-stakeholder network, so this is just my opinion. 3:34 It just happens to be the correct opinion. 3:36 You can see AT protocol @proto, all lowercase, not a capital A, and atmosphere. 3:44 The rest of these are out, especially capital A, capital T, capital P Proto. 3:48 That one is terrible and it needs to be killed. 3:53 Okay, now on to the real stuff. 3:55 Uh, talking about the IETF or who gets to decide what the protocol actually is. 4:01 So, what is the IETF? 4:02 It's the Internet Engineering Task Force. 4:05 Uh, it's the standards body for the internet. 4:07 It standardized all the protocols that you know and love. 4:09 I'm sure that you're familiar with a lot of these: SMTP for email, HTTP for the web, TLS— it gives you the little green lock, uh, that tells you that your website's secure— WebSockets, OAuth, and more. 4:22 Uh, to tell you a little bit about the vibe of the IETF, it's a pretty weird, idiosyncratic organization. 4:28 I have a quote here from, uh, Snarf that I think captures it pretty well. 4:32 He said, standard— standardization is Standards bodies or bureaucracy for good, which I think sums it up. 4:37 It's pretty bureaucratic, it's pretty pedantic, but it's ultimately a very idealistic organization and it's a very neutral, credible body and they do a very good job sort of stewarding and owning these sorts of internet protocols. 4:51 So it's a mix of institutional but it's also grassroots. 4:53 You kind of have like a lot of old internet heads there, you have activists, you have academics and researchers, you also have folks from the big tech companies, Google and Cloudflare. 5:01 And others. 5:03 Uh, and that also gives it this mix of idealism and pragmatism. 5:06 People have conversations about keeping the internet decentralized, but also they care a lot about the internet working well. 5:12 Uh, and so they have this very pragmatic approach. 5:15 Uh, the kind of motto there is that decisions are made by rough consensus and working code, which I think actually gives a good sense of that idealism and pragmatism. 5:25 They want to see something working and they want to get a bunch of diverse voices in the room and have general agreement on it. 5:31 It's open membership, anybody can join, you don't need to pay money to join, you can just hop in the mailing list, start participating, you can show up to meetings and start participating, and companies are not members of it, individuals are members. 5:43 And I put kind of here because most people have an affiliation with a company and it's like, they kind of wear, it's pretty clear that they are affiliated with companies, uh, but it is still, it kind of gets at their, their ethos, uh, in a way, which is that these are, these are engineers coming together trying to reach consensus on protocols. 6:02 So the exciting news is we have a working group at the IETF. 6:05 Woo! 6:11 Uh, also huge kudos to Brian on that. 6:14 Brian really led up the standards work. 6:16 In all honesty, he should be the one up here presenting all of this. 6:18 He's just done an incredible job at that. 6:21 Uh, the working group is called Authenticated Transfer, or ATP. 6:25 Uh, this is the URL for it. 6:26 Uh, Um, the responsible area director is Andy Newton, the working group chairs are Shuping Peng and Mallory Nodel. 6:35 Brian and I got the chance to chat with all three of these folks, uh, earlier this week. 6:39 Generally got very good vibes from them, they're ready to hit the ground running, they're kinda like, start engaging on the mailing list, draft up some stuff, uh, we wanna have a lot of energy at the first meeting, so, uh, the vibes were good, I'm excited about this. 6:54 This is our charter in very small font. 6:56 You can read the entire thing on the IETF data tracker. 7:01 Uh, this was on the note of bureaucracy for good. 7:04 This was 4 months of going back and forth on a mailing list to get that nailed down to decide what we were going to work on. 7:10 Uh, and to break down kind of what's in that, the, what the charter for this working group is for is the structure of the repo and how we encode records in it, the sync protocol for public repositories, the ATURI scheme for addressing records in those repositories, and then interface requirements and selection criteria for an identifier resolution system. 7:31 Uh, this was kind of the hardest part of nailing down the charter, which is that we did not want to pull DIDs into the IETF. 7:38 We did not want to wade into the quagmire of specifying identity. 7:43 Uh, so we kind of There's an identity-shaped hole in the spec and we're specifying the shape of that hole. 7:51 We are not going to be— this working group is not going to be doing lexicon, any microblogging or social semantics at all. 7:59 Identity, as I mentioned, labeling, and the permission data work that we're working on right now is outside of the working group as well. 8:05 Though parts of this may— we may recharter the work. 8:09 If everything goes well for both parties, we may recharter the working group in the future and address some of these. 8:15 So, just to call it out, this is your chance to contribute to the technical direction of the protocol. 8:20 As I said, this is open membership, anybody can hop in here. 8:23 If you think that we screwed it up and the protocol is stupid or ugly or whatever, this is your chance to say it and actually have an impact on the protocol. 8:30 You, in, in some very real sense, you are on an equal playing field as Brian or I, uh, on that mailing list and in that room. 8:39 Uh, so, just to give you a sense for participation, as I said, anyone can participate. 8:43 There's 3 in-person meetings a year, one in North America, one in Europe, and one in Asia generally. 8:49 They have a very good remote-first experience, uh, and usually if you're attending remote and only for one day, they'll give you a free ticket, uh, so that is, that is great. 8:59 Uh, the first working group meeting will be, will probably be at IETF 126 in Vienna, which is this summer, July. 9:07 We may do a remote interim meeting beforehand, uh, and I will keep people updated on that. 9:12 But regardless, follow along on the mailing list, it's atp- ietf.org. 9:16 Brian and I are gonna be trying to send out some stuff on there, uh, somewhat soon. 9:20 So hop in and let us know your thoughts. 9:23 Okay, on to the next one. 9:24 PLC, or what about that centralized server at the bottom of the stack? 9:28 Uh, for those that don't know, PLC is the identi— identifier, not identity, the identifier system for the AT protocol. 9:36 Uh, it's where your keys are stored, uh, or yeah, it describes your public keys and your service endpoints and it's kind of the centralized server run by BlueSky right now at the bottom of the stack. 9:49 My hot take is PLC isn't that bad. 9:51 The threat model is actually pretty limited, uh, everything is cryptographically signed, uh, the threat model of what, whoever is running the PLC directory, what they can get away with is not that much. 10:01 They can't arbitrarily change your identity or anything like that. 10:04 They can essentially do denial of service attacks, removing operations from your identity or refusing to serve your identity. 10:12 Uh, those aren't good, but just to note the threat model is pretty limited. 10:16 All that being said, it still needs some work. 10:18 Uh, and so we're sort of thinking about maturing governance of PLC as we go along. 10:25 We'd like to use DNS and ICANN as an example, um, mostly for an analogy of iterative maturation of governance. 10:32 DNS has issues, ICANN has issues, don't overthink the analogy, but you can kind of see how DNS evolved over time. 10:39 Started as a text file that got distributed to different hosts in the network, moved to a single server that was a computer under the desk, transitioned to multiple root servers, ICANN got set up as a governance body for it and ICANN continues, uh, to evolve in their governance, uh, being more and more multi-stakeholder, taking into account other countries outside of the US more. 11:01 Um, and so this is sort of the model that we look at with PLC is, uh, You know, like we don't have to get it to the end state of like this is the perfect PLC forever or something tomorrow. 11:11 It's how can we iteratively move that forward as the like kind of threat model for the atmosphere evolves. 11:18 So the exciting news to announce is that Public Ledger of Credentials organization has officially been founded as a Swiss association. 11:32 Again, huge kudos to Brian for leading this up as well as Matt, our head of legal. 11:36 Which, if you guys can find, I don't know if Matt's here, but if you guys can find Matt, you should talk to Matt. 11:41 He's like protocol-pilled now, it's very fun. 11:43 Uh, so we put a lot of thought into who is going to be on the board for PLC. 11:50 Like, we wanted a board that people in the ecosystem could really trust to run this thing, and I think that we found the perfect one, which is, it's just gonna be Roscoe. 12:00 I think Roscoe's just gonna run this. 12:06 Uh, no, I'm kidding about that. 12:07 This is, this is going to be the actual board, uh, for the PLC Association. 12:12 There's 5 members, uh, Brian, who is on the BlueSky team, he's a protocol engineer on the BlueSky team, uh, he's the best person from the PLC or from the BlueSky team, uh, for this. 12:23 Richard Barnes, uh, he's the co-founder of Let's Encrypt, which has, uh, some analogies to PLC, it looks pretty similar to PLC. 12:31 He's also the co-author of the MLS, ACME, HPKE, all of these deep crypto IETF standards. 12:36 Uh, specifications. 12:38 Wendy Seltzer, who is an internet lawyer and open standards advocate. 12:41 She's here at the conference. 12:42 You should find her and talk to her if you're interested. 12:49 Filippo Valsorta, who is a cryptographer. 12:52 He maintains the Go cryptography library and he's also a transparency log aficionado and is working on the Project Sunlight implementation. 12:59 He is also here at the conference. 13:00 You should come talk to him. 13:04 And then Tyler and I don't know how to pronounce her last name unfortunately, but she's a a security and privacy engineer at Google. 13:11 Uh, she also lives in Switzerland, which is quite nice for, uh, organizational reasons. 13:17 Um, yeah, so just to, just to note again, this is like the first step in the evolution of PLC. 13:24 This isn't the end state of it, we're going to continue working on this. 13:27 Uh, we also haven't actually transferred any assets over to PLC. 13:31 Uh, that is something that we will be doing in the coming months and we'll give updates as that happens. 13:34 So, The first order of business is getting the domain name over to PLC. 13:40 Uh, alongside these organizational improvements, we also have some technical improvements to PLC. 13:44 We shipped streaming replication and a replica reference implementation that provides full verification of all directory operations. 13:52 Uh, there's a number of people that are running this in the network already, and the longer-term plan is a sophistication of that, which is transparency logs. 13:59 And you know that we're gonna do that because we have Filippo on the board. 14:05 Uh, great. 14:06 Now, so those are sort of like the governance updates on like kind of these core parts of the protocol. 14:12 This next section is about hard decentralization or that's great and all but what happens when Blue Sky kills API access or servers are actually just physical objects that can be destroyed, unplugged, or seized. 14:23 This is kind of the real politic of running a multi-stakeholder network where you have one big person at the center of it What are the particular threats and what can they get away with? 14:33 So I wanna call out that decentralization isn't just one thing and we need to think through what those threats actually are. 14:39 I'm, numbers are important but I'm somewhat uninterested in like what percentage of people are on what thing or what, it's what are the actual threats that come from BlueSky being outsized in this ecosystem? 14:51 Let's think through those threats and think through how BlueSky can address them and how the network can address them. 14:57 So, we're gonna do some threat modeling on what BlueSky could do and then we're gonna talk through some of the solutions. 15:03 So, the most common thing that people think of is BlueSky shuts down AppView API access. 15:08 People are familiar with this cause a lot of social companies in the past have done it. 15:11 This is actually sort of the threat that I'm least concerned about, uh, because data access is guaranteed by the open hosting PDS layer. 15:18 As long as the hosting layer, that's The network inherits its openness from the PDS layer and as long as the hosting layer remains locked open, then these other app view implementations can emerge as the need arises. 15:31 I've somewhat jokingly in the past called this lazy decentralization. 15:35 That it's if you trust that the hosting layer is open, then you don't need to decentralize the app layer yet. 15:41 But you have to trust that the hosting layer is open and we'll talk about that in a second. 15:45 Uh, still the things that we need for this are tools for backfilling the network and syncing the network and also app view implementation code. 15:54 Uh, so now to talk about if we screwed up the hosting layer, this would be BlueSky shutting down access to the PDS or access to the open data that's hosted by the PDS. 16:03 Uh, to note, BlueSky, if BlueSky were to do this, it destroys all of the feeds, apps, and labelers running on the protocol. 16:11 So the pain of destroying the network of open applications needs to be severe enough that BlueSky cannot consider shutting down our PDS app. 16:18 PDS access. 16:20 A smaller version of this attack would be Bluesky may stop serving data to particular competitors. 16:25 So if one big competitor emerges in the ecosystem, maybe we keep sharing data to all of the feeds and labelers out there, but we cut off access to that one larger competitor. 16:36 The things that we need to prevent this are a thriving network built on open data. 16:41 If people are relying on open data, then we're incentivized to not cut off access to it. 16:45 And then specifically on this cutting off access to particular competitors, all of the data is signed and rebroadcastable. 16:52 That means that you can get it from any source. 16:54 You can get it from the BlueSky PDS or you can get it from another PDS. 16:58 So we need multiple sources to get data from. 17:04 Uh, the next sort of threat is BlueSky stops crawling non-BlueSky PDSs. 17:09 So these are folks that have migrated their account off of the BlueSky PDS and the BlueSky app view stops indexing non-BlueSky PDS data. 17:17 Their motivation to do that would be to monopolize hosting. 17:20 Uh, what we need to prevent this is a critical mass of users on non-BlueSky PDSs and other applications that do index non-BlueSky PDS data. 17:30 And that, I wanna call out that, that critical mass, basically we need the critical mass cause if BlueSky were to stop indexing this, you need it to screw up people's experience of the app and they would get pissed because there's data that they want to see that they're not seeing in the BlueSky app. 17:44 Uh, I want to call out that I don't think that we need like an even distribution of users across hosting infrastructure in the network in order to achieve this. 17:54 You can imagine 5% of the network, if we're even less than that, if it's an important 5% of the network, that would be disruptive enough that if you cut off access to them, people would be upset and leave the BlueSky app. 18:06 This threat also becomes much less substantial if people can just go to another app that is indexing both BlueSky content and content from non-BlueSky PDSs. 18:16 So having other applications in the network, uh, helps prevent this threat. 18:22 Uh, another threat is BlueSky messing with accounts. 18:25 Uh, examples of this are preventing users from migrating between PDSs, uh, deplatforming users, especially if we don't have just cause, preventing users from using certain apps or competitors. 18:36 The things that we need this or ba— need to prevent this are basically account recovery or export tools. 18:42 So backups of repositories if BlueSky is refusing to serve your repository to you. 18:47 User-held recovery keys in case BlueSky is refusing to sign account migration operations on your behalf. 18:54 And then just a place to go. 18:55 Uh, non-BlueSky PDS is running in the network that you can migrate to. 19:00 Uh, and then the final threat is, uh, BlueSky just losing access to accounts. 19:05 Uh, this may be, uh, sort of out of ill intentions or this may be external forces. 19:11 So going out of, uh, business, you know, a data center blowing up, legal action, servers getting seized, something like that. 19:18 I would expect in most cases BlueSky should be able to help ease this transition. 19:22 For instance, like the most likely one of these is probably BlueSky going out of business and we should be able to help ease the transition. 19:28 If that were the case, even if we stopped providing PDS services, it's very cheap for us to keep some PDSs online and rotation keys as people move off. 19:36 In extreme cases where BlueSky is refusing to do that or unable to do it for some reason, we essentially need the, what we got from the last threat, which is backups of repositories and user-held recovery keys. 19:50 So pulling out the things that we found that we needed for each of these, it's basically AppView implementation and Sync tooling, uh, nice. 20:01 More users on non-BlueSky PDSs, so a more diverse open data hosting layer. 20:06 Uh, consumer key management and recovery tools, putting, uh, rotation keys in the hands of users. 20:12 Other data sources and backups for repositories. 20:15 And then a thriving ecosystem built on open data. 20:18 So I wanna walk through each one of these and talk about what the BlueSky team can do, what we are doing, and then what folks in the ecosystem can do if you're concerned about these particular threats. 20:33 So first on AppView implementation and sync, relays are cheap now with Sync 1.1. 20:39 Fig is running a relay for under $5 a month. 20:41 That's awesome. 20:42 There's over 10 full network relays running in the network. 20:45 Uh, we seem to be getting pretty good at doing this one. 20:47 So you can get your data from a lot of different sources. 20:50 Uh, there's lots of new sync tools coming out. 20:52 Uh, we publish tap. 20:54 Hydrant, the entire Fig Proto Microcosm universe is helping with sync, and then Black Sky is actually running a full network app view, which is incredible. 21:12 So in some sense, I kind of think this one's solved, although we can always do better on it. 21:15 But there's also novel approaches like Red Dwarf that provide a Blue Sky-like experience with no app view. 21:21 And so there's ways to rethink what this looks like in the network where maybe, I think it's amazing that Black Sky is running a full network app view. 21:28 I think that you can probably provide Blue Sky-like experiences for significantly cheaper if you make certain trade-offs in how you construct this. 21:38 The next one, Kuba joked that we should take a shot every time this graph is shown at the conference and I told him I got one more for him. 21:48 The next one is more users on non-BlueSky PDSs. 21:51 Uh, this is a graph of weekly users posting from non-BlueSky PDSs, and you can see that the trend line is pretty good. 21:57 So in some sense, we just gotta keep cooking on this and doing what we're doing. 22:01 Uh, what BlueSky, what we can do and what we're planning to do to help with this is putting an account migration UI in the PDS. 22:07 Um, we think that users, especially non-technical users, probably just have a better trust relationship with the BlueSky PDS, or yeah, with the BlueSky PDS and would trust an account migration UI there a little bit more to get over to non, uh, BlueSky PDS hosting. 22:23 We want to improve the PDS distribution, especially for mid-sized deployments, things in the 1,000 to 100,000 user range. 22:30 Right now it works pretty well for self-hosters and small groups of people. 22:33 It's a little bit of a pain for mid-sized deployments. 22:36 And then we also want to make the relay more, uh, self-serve. 22:40 Both for third-party relays so that it's easier for them to get proactive write notifications whenever PDS is right, but also for our relay doing dynamic rate limits so that you guys don't have to DM us whenever you run into that and you get cut off from the relay. 22:55 Um, what, what you all can do for this to improve this is keep cooking and also backup and recovery systems. 23:03 We talked about backup and recovery systems in the context of sort of BlueSky threatening people's repositories and keys, but it's also important for migrating off to non-BlueSky PDSs because you don't necessarily, you know, you're taking on an operational risk whenever you do that. 23:21 You kind of know BlueSky has money and we, and you know how we operate stuff and you know that we're probably keeping your data secure. 23:27 Whenever you move to another, uh, PDS or even self-hosting, you take on some operational risk and having backup and recovery systems can give you peace of mind to do that. 23:39 Uh, the next thing that we called out was other data sources and backups for repositories. 23:44 Uh, Phil, Fig, Bad Example just announced Hubble, which is funded via a grant from Blue Sky. 23:50 I'm super stoked about this project. 23:52 Hubble's a full network mirror of public repositories. 23:55 The relay used to offer this. 23:56 Whenever we switched to Sync 1.1, it no longer mirrors all the public repositories in the network. 24:02 Uh, and so this, this is a really is that it provides a full network copy of all of the data. 24:06 It's intended to increase network resilience. 24:09 This is a backup of your repository. 24:11 If your PDS crashes and burns and you lose your repository, you can go get it from here. 24:15 If BlueSky tries to lock you out of it, you can go get a backup of your repository here. 24:19 Or if BlueSky shuts down our APIs, then you can go sync repositories from Hubble. 24:24 All of the code will be open source and permissibly licensed. 24:27 Phil's planning on running an instance of this, but anybody can run an instance of this and provide a full network mirror. 24:34 Uh, consumer key management and recovery tools. 24:36 I actually think that there's a really great product opportunity here and I'm kind of surprised that nobody's done it. 24:41 You can take advantage of phone TPMs for storing recovery keys in a very secure way. 24:46 Uh, this application could also audit PLC and notify users of any unexpected operations and I, uh, just to note, I also think that this would partner well with a backup or recovery service, especially whenever we put out permission data. 25:00 And then the final thing is just a thriving ecosystem built on open data. 25:04 And this is, I think, the biggest part where you all come in. 25:06 Apps built on the atmosphere, atmospheric websites, feeds, labelers, social media tools, anything that requires that the data be open. 25:14 In some sense, all of the previous threats that we talked about are— they're pretty protocol-brained and I just don't think a lot of users are going to care or engage about that. 25:23 And I don't think that we're going to get to like an even distribution of users by focusing on key recovery or non-Blue Sky PDSs or anything like that. 25:30 The most important thing for the resilience of the network is that users need to understand that they're participating in a multi-stakeholder network, that they're participating in the atmosphere, that they're not participating in Blue Sky. 25:41 And so if we're building things that run on open data and users fall in love with OpenSocial and these experiences that you can only build on open data, that is the most important thing that we can do to keep Blue Sky honest and to keep the OpenSocial web running. 25:57 So the best way to ensure the resilience of the network is to build something that people love so much that they would revolt if it ever went away. 26:07 Sweet, thank you. Daniel Holmgren 26:16 We're— that's great, it's awesome. 26:19 We should get that done probably by next week, week after next, so it's great. Speaker B 26:23 Yeah, easy. Daniel Holmgren 26:25 We're heading into the coffee break and we have a couple of minutes. 26:29 Questions? Speaker B 26:37 So users knowing that they're participating in the atmosphere, not just blue sky. 26:43 How do we do it? 26:44 What's the branding? 26:45 That's a great question and we should figure that out. Daniel Holmgren 26:48 This gentleman just showed you an ASCII presentation. Speaker B 26:54 This presentation came from our designer. 26:56 I want to call that out. 26:59 No, that's a good question. 27:01 I think that that is— I mean, I don't have the answer for you right now, but that is the core question that we need to figure out this year is how do we start to build intuition for users about what the atmosphere is, what they're participating in, what an atmosphere count is, and what this whole OpenSocial thing is about. Daniel Holmgren 27:19 How is the Swiss Association funded? 27:22 Is that like a grant from Blue Sky, and how does this thing continue even if one of your scenarios, Blue Sky went away? Speaker B 27:30 That's a good question. 27:31 Right now it is funded via a grant from Blue Sky. 27:35 The exact details of that haven't been nailed down just yet, although it will probably provide some sort of ongoing funding. 27:41 And then the association is going to be working to provide, you know, more sustainable funding