standard.site -style lexicon for hosting datasets on atprot…

Samuel

@samuel.fm

standard.site -style lexicon for hosting datasets on atproto? anyone have $500?

Standard.science domain available for $517.50

I don’t have specialist knowledge here but I’d imagine you’d want some sort of metadata (description etc) and then just an array of blobRefs

2 5

Samuel 27 Jun

@jimray.bsky.team what about hosting a lexicon design challenge with a $517.50 grand prize

2 9

Samuel 27 Jun

we shall get huggingface onto atproto inshallah

2 15

alice 27 Jun

or i can put atproto on huggingface bsky.app/profile/alic...

alice@alice.mosphere.at· 27 Jun

a nice byproduct of my backfill is i'm now the proud owner of one of the largest dataset of bluesky posts, roughly all 2.6 billion of them, with all their metadata from the post lexicon living in 552GB zstd'd parquet files, which for now live on a simple hetzner storage box

why not both 😈

🤔

I was working on this :)

Erica "digifox" Kovac 27 Jun

I feel like datasette's developer/maintainer might be interested in this: @simonwillison.net

Al Merose 27 Jun

Atproto would make an ideal zarr.dev store!

Zarr

A specification, implementations in 10 languages, and an ecosystem of datasets and tools, built by an open community.

zarr.dev

dad (aka miguel)27 Jun

you don’t even know how right you are, we’re around there too 💀

1 2

Samuel 27 Jun

godspeed

Andrew Lisowski 💻27 Jun

@byarielm.fyi

1 1

Al Merose 27 Jun

My friend @tegnicholas.bsky.social wrote a whole blog post about how this could work! tom-nicholas.com/blog/2025/sc...

Science needs a social network for sharing big data - Welcome

Imagine being able to visit one website, search for any scientific dataset from any institution in the world, preview it, then stream the data out at high speed, in the format you prefer. We have the ...

tom-nicholas.com

tierney cyren 27 Jun

I’ll chip in $100

Al Merose 27 Jun

Sorry to spam you, but I’d help you build this. DM me.

Al Merose 27 Jun

CC: @tegnicholas.bsky.social

Sam 27 Jun

We need crowdfunding on atproto

B. Prendergast 27 Jun

@emily.space maybe you might have ideas about this

Emily Hunt 27 Jun

That'd be really cool! I think the right venue for this would be the IETF or similar, though. It would take a LOT of input to make something generic enough to work. Also, @vmx.cx's matadisco records already contribute here too

1 3

Al Merose 27 Jun

Maybe, or maybe not! I think a lot of scientific data may actually secretly be Zarr: github.com/zarr-develop...

GitHub - zarr-developers/VirtualiZarr: Cloud-Optimize your Scientific Data as Virtual Zarr stores, using xarray syntax.

Cloud-Optimize your Scientific Data as Virtual Zarr stores, using xarray syntax. - zarr-developers/VirtualiZarr

github.com

Denis 27 Jun

How does hosting stuff on atproto work? Won’t it be centralized?

Samuel 27 Jun

you put it own your own server, atproto serves as the distribution layer

1 5

Al Merose 27 Jun

This is very similar to an idea I’ve had — I think the designs would be similar on atproto to torrents: github.com/zarr-develop... Furthermore: decentralized, replicated stores would be essential for many domains of science, especially climate science

A Torrent ZarrStore · zarr-developers zarr-python · Discussion #3036

Coming back from Cloud Native Geo, a major theme was how to build resilient data infrastructure (or ecosystem). A common idea that came up in many discussion groups was could we somehow store scien...

github.com