Keep in mind that you’re going to be retrieving and storing a huge amount of data running these scripts
And you are adding to the overload of lemmy.world, beehaw, lemmy.ml, etc who have all the popular content communities. Federation has a lot of overhead, as does having to distribute a community one vote at a time to 500 subscribed servers.
pend my time on Lemmy scrolling “All”, which I think is a pretty common thing.
There was a lot of advice handed out back in June that the answer to scaling Lemmy was to go create instances. The reason it works is because "All" is empty on a virgin system ;) With no data in the database, the logic Lemmy hands to PostgreSQL works real fast ;)
The real war is hate media memes and messages that trickle down to those obedient to voices from the clouds, electric voices these days. In the old days, books that said burning bushes gave out signals.
Is your intention to have local copies of content from popular servers and read it locally? Major communities like news., memes, etc?
Many people seem to think this is offloading the major servers like lemmy.world - but I think the opposite is true in my measures of how lemmy_server performs. There is a lot of overhead to each additional instance in Lemmy 0.18.3 backend. Lemmy code does a lot of work to keep each of these subscribing servers updated with every post, comment, vote, person - attempted in real-time.
Keep in mind with 0.18.3 there is a new "dead server" check that cuts off communication that won't show up on the block list. There have been reports in !fediverse@lemmy.ml recently that this has caused some issues. notable posting: https://lemmy.ca/post/2626714
Storage was still pretty expensive, and there we transitions in computing from originally paper terminals to screen and people didn't have a sense of long-term retention of personal messages (I guess many people probably felt that way about SMS messages on mobile). There also wasn't really a way to look at a user's "profile" like you have on Lemmy - to see everything you post in any topic - which a search-engine provided a way to search for your name across a time period.
Me and my brother were talking to each other
About what makes a man a man
Was it brain or brawn, or the month you were born?
We just couldn't understand
Really, are you going to ignore what it says? The opening?
It implies a flat /c/a /c/b /r/a /r/b system "until now"? Or am I wrong?
Perhaps you aren't faniliar with how under-utilizes naming dots matter in domain names?
smtp.chemistry.science.oranic.org has been in the Internet (Usenet) conventions for a VERY long time! Forgotten, burred in $$$$$$ wealth. "Windows"... Everywhere. Owning the words. TradeMarks.
Gentoo Linux agrees... the very root of the concept of "federated" comes from Usenet, which did not have a flat hashtag style group, a flat subreddit /r/name /c/name kbin magazine convention.
It was a big deal when we got an archive we could search of all content...
"The Deja News Research Service was an archive of messages posted to Usenet discussion groups, started in March 1995 by Steve Madere in Austin, Texas. Its powerful search engine capabilities won the service acclaim, generated controversy, and significantly changed the perceived nature of online discussion. This archive was acquired by Google in 2001."
Up until now, social containers like groups, communities, or subreddits on all the largest social networks have existed as fundamentally separate locations on a single hierarchical level.
"Up until now"... Uh... no, Usenet... was the open standard for social media. Created in 1979. A foundation of the Internet. Just as much as e-mail was.
The official lemmy-ui works fine, but it has never been a top priority for the Lemmy project; understandably so, as they've been focused on pioneering an ActivityPub-enabled forum backend.
They are actively developing a new Rust front-end too.
Something to consider... Lemmy runs fast with no data in it. Latency issue is tied to how much data they have stored in PostgreSQL. If they aren't holding full copies of all the remote communities, sure it is faster, but your searches and All aren't going to turn up much.
And you are adding to the overload of lemmy.world, beehaw, lemmy.ml, etc who have all the popular content communities. Federation has a lot of overhead, as does having to distribute a community one vote at a time to 500 subscribed servers.