Why not Fanout via static files or CDNs in the Fediverse?
Why not Fanout via static files or CDNs in the Fediverse?
Current Fediverse Implementation
From my understanding, the prominent fediverse implementations implement fanout via writing to other instances.
In other words, if user A on instance A makes post A, instance A will write or sync post A in all instances that have followers for user A. So user B on instance B will read post A from instance B.
Why this is Done
From my understanding, to prevent a case where post A is viral and everyone wants to read it, and instance A's database gets overwhelmed with reads. It also serves to replicate content
My Question: Why not rely on static files instead of database reads / writes to propagate content?
Instead of the above, if someone follows user A, they can get user A's posts via a static file that contains all of User A's posts. Do the same for everyone you follow.
Reading this file will be a lot less resource intensive than a database read, and with a CDN would be even better.
Cons
- posts are less "Real time". Why? Because when post A is made, the static file must be updated (though fediverse does this already), and user B or instance B must fetch it. User B / instance B do not have the post pushed to them, so the post arrives with a delay depending on how frequently they fetch. But frequent fetches are okay, and easier to handle heavy loads than database reads.
- if using a CDN for the static files, there's another delay based on the TTL and invalidation. This should still be small, up to a couple minutes at most.
Pros
- hosting a fediverse server is more accessible and cheaper, and it could scale better.
- Federation woes of posts not federating to other instances can potentially be resolved, as the fanout architecture is less complex (no longer necessary to write to a dozens or hundreds of instances for a single post).
- Clients can have greater freedom in implementing how they create news feeds. You don't have to rely on your instance to do it. Instances primarily make content available, and clients can handle creating news feeds, content sorting and filtering (optional), etc.
What are your thoughts on this?
How do you expect your feed to be updated?
If a CDN is involved, we would have to properly take care of the invalidations and what not. We would have to run a batch process to update the CDN files, so that we are not doing it too often, but doing it every minute or so is still plenty fast for social media use cases.
Have to emphasize that I am not expert, so I may be missing a big pitfall here.
So I have to constantly check all files from everyone I follow for new entries in order to have a working timeline?
Sorry, I meant your timeline, where you see other peoples posts.