Samuel
ok, let's break it down. at the core of atproto is the data. it lives own your own computer (or someone hosts it for you). think of it as a hard drive with JSON files on it data is structured (i.e. has a type), and can references other records on other peoples' computer with a URI like a hyperlink
Diagram labelled network state Two areas, my data storage and your data storage In my data storage, there is a post and a like. the like has an arrow to a post in your data storage
Lu wilson@todepond.com· 2 Nov 2024
okay let me get this straight... it's like this, right? to participate in this particular microblogging site we're all on, it has to go through the first-party appview?
75 184 654
Samuel
ok, let's say we want to see some posts. we can ask our PDS for some records, but that's not very interesting because it's just my stuff. we have a hyperlink to the post I liked, but that's a bit inconvenient because I'd need to go all the way to Lu's machine to see what it is
The above diagram with an arrow pointing to my data storage written beneath the arrow is "curl https://pds.mozzius.dev/fictional.getAllRecords" and beneath that is content from my data service again 2 1 76
Samuel
in order to make sense of the network, we're going to build a cache of all the content, so we can query it without needing to check each machine individual. think of it like Google's search index, except instead of googlebot scraping all the data, the PDSes offer a websocket that sends data instead
The data storage boxes now have "websocket" arrows pointing to a database symbol labelled "network cache" 3 1 81
Samuel
ok, we're getting somewhere. but to make sense of this big pile of data, we're going to offer some opinionated APIs. for example, a "getTimeline" API that returns all "post" records, with a computed count of how many "like" records are pointed at them now, we can fetch a timeline!
Now the "network cache" is in a box labelled AppView. also inside the appview is a box labelled "machine that calculates vote counts" beneath that is curl for "https://api.bsky.app/getTimeline" which returns posts from both data storages, and one of the post has a field of "likes: 1" 2 7 106
Samuel
this "AppView" service provides a "view" of the network. It will ignore records it doesn't know or care about, and derive useful information from the data it has now, since the data just sits on the web, and AppViews are downstream of the data, you can run another one and arrive at the same view!
The previous diagram, except the AppView section has been duplicated with the duplicate labelled "Alt AppView" 2 7 89
Samuel
so, imo the network (with multiple AppViews in play) looks like this. when you use a client, you ask an AppView to give you an opinionated view of the whole network. It acts as a cache in front of any number of hosts, which unlocks stuff like global search and accurate like counts (wild, right!)
Same diagram as the last one except the curl command has been replaced with a box with a butterfly emoji in it labelled "App" 6 4 80
Samuel
and to wrap things up, this is what writes look like. your client writes the new record to your PDS, which flows out down to the AppViews' caches, where it'll finally reach your client again (we're thinking of letting AppViews make writes on your behalf, but that's an optional UX/DX optimisation)
A green arrow labelled "write" goes from the App to your Data Storage at the top of the diagram. the Websocket arrows from your PDS to the AppViews are colored green along with the databases in the AppViews 7 16 97
Samuel
@samuel.fm
the "relay" is probably a red herring when thinking about the shape of the network. its just a many-to-one websocket service with some caching, so consumers that want to watch all events on the network (to make a feed, or whatever) can just watch the one websocket instead of 700+ as of Nov 2024
2 1 63
Samuel
running a PDS (your data store) is cheap. $5 VPS cheap, can have loads of users on the same host, and is immune to how busy the network is. if we gain 100mil users, you'll still be able to use a $5 VPS because it's only your data and only your traffic
4 7 104
Samuel
running a relay is more expensive but affordable (see whtwnd.com/bnewbold.net... ) running an AppView is tricker. this is a high-scale high-throughput service that requires seriously beefy machines. we don't expect people to be running them for fun on an Rasp. Pi, but we do want it to be feasible
2 4 73
🌺 Eva/Kitsundere 🦊 "Mama"
After reading through this, I definitely feel like I have a better appreciation for both what ATProto is and how the general network works behind the scenes with a client. The cache on a websocket "relay" is super nifty and seems like a really creative solution to optimize interactions
1 13
🌺 Eva/Kitsundere 🦊 "Mama"
I've been extra curious how it works because of the whole "users are websites" deal and the ability to host your own PDS..my mind went straight to things like traffic and the costs for each user, BUT here it appears the PDS itself is relatively cheap to maintain vs the larger cache bsky interacts w/
1 10
Samuel
yeah exactly. every user a website, but since data is cryptographically verifiable (Authenticated Transfer Protocol, see?), you don't need to fetch it directly from your machine to know it's really from you. this allows trustless network caches to sit between your machine and all the traffic
2 4 46
Ian Hatcher-Williams
one question i have is how is the index of DIDs is distributed? i understand that the main app view is subscribing to all the various PDS, but if that goes down, who holds the record of all possible PDS? does that need to function like DNS servers?
1
Maël
Is there a list of appviews ? What would be the incentive for a organisation to create an appview, appart from decentralisation ?
3
Zetaphor
I realized I didn't leave a link, sorry! lowendbox.com/blog/1-vps-1... I am running two boxes from Racknerd, one of which is my PDS. The PDS box had its RAM upgraded bringing it from $12/year to $23/year
$1 VPS – 1 USD VPS Per Month (Updated October 2024) - LowEndBox
Can you really get a VPS for only a dollar a month...and if you can, how reliable are the hosts? Believe it or not - and you don't have to believe us because we've got links for you! - there are a nu...
lowendbox.com
 1 1 2
Liberal Rebound
What's the trade off? I'm paying like 5x that on Hetzner, although the specs are beefed up and I'll probably be running other things on it. But like would the low specs impact a small PDS with only a few users?
1 2
Zetaphor
None that I've found so far! I'm running my PDS for my fairly active user account without any issues at all. The way the infrastructure is designed, your PDS is acting as a database/JSON API. You don't need a whole lot to handle that. It also helps that the Bluesky appview, their CDN caches media
https://skyzoo.blue/stats/zetaphor.com
skyzoo.blue
 1
Zetaphor
I believe it's also caching more than just media, so in the context of Bluesky it's an extremely light serve. You can do a LOT with a small amount of resources I'm also serving a TTS service from a similarly sized server for a small but active userbase in a VR game
1
Zetaphor
The PDS runs in docker, so you could probably rig something up. Not sure if that is possible with Racknerd though as I'm pretty sure you need to contact their support to allocate more resources. They give you such low pricing by provisioning you on older hardware
1
Liberal Rebound
Yea, I mean damn guess this is just another example of how much tech goes to waste, when such a clever well thought out protocol can at least in part function without issue on old hardware, but we cycle through beefier over powered hardware and don't even take advantage of it fully
1
Olivier Forget
This is such a key difference compared to AP. AP just doesn’t scale well. I suspect that’s why you’ll eventually get users with huge followings here but not so much there.
3
andwhyisit
I want to self-host my pds one day, but I wanna use Rocky Linux as the distro so that I can use my existing vps. Is there a red hat installer yet?
C
Do you think PDSes might end up with an allowlist of friendly relays? For example, if a state actor DDoSes PDSes by asking for lots of backfills and/or subscriptions
1