8mo ago

Seems .world and .ee federation are broken

Is it a .world or a .ee issue?

Seems .world to .ee is broken so i wont be seeing anyone's comments from .world and the rest of the fediverse probably wont see this.

Heres ya issue: https://grafana.lem.rocks/d/bdid38k9p0t1cf/federation-health-single-instance-overview?orgId=1&var-instance=lemmy.world&var-remote_instance=lemm.ee

Edit found more info

25 comments

it's not just lemmy.world.
of the larger instances, the following have trouble sending activities to lemm.ee currently:
lemmynsfw.com -> lemm.ee: 2.81d behind
sh.itjust.works -> lemm.ee: 1.04d behind
lemmy.world -> lemm.ee: 22.5h behind
i pinged @sunaurus@lemm.ee on matrix about 30h ago already about the issues with federation from lemmynsfw.com, as it was the first one i noticed, but I haven't heard back yet.
- Seems like theres some widespread DNS problems again today
https://grafana.lem.rocks/d/bdid38k9p0t1cf/federation-health-single-instance-overview?orgId=1&var-instance=lemmy.world&var-remote_instance=lemm.ee
- Wait, how do they get that data remotely? I was looking at my instance vs world and I saw there's like the +1 hour from a week or so ago when I upgraded to latest mbin lol.
  I guess they're looking at common activities and when they appear on each?
  
  lemmy has a public api that shows the federation queue state for all linked instances.
  it provides the internal numeric id of the last activity that was successfully sent to an instance, as well as the timestamp of the activity that was sent, and also when it was sent. it also includes data like how many times sending was unsuccessful since the last successful send. each instance only knows about its own outbound federation, but you can just collect this information from both sides to get the full picture.
  there is also https://phiresky.github.io/lemmy-federation-state/site to look at the details provided by a specific instance.
Huh.
I can see the post from my .world account
- Yeah but u cant see this comment from my .ee account
- Mind u lemmy without .world is an interesting experience
  
  its about 25% of my instance traffic. its noticeable, but theres definitely a shift towards spreading content out.
I know there was a problem where federation was lagging by a couple of days, but I thought it was fixed? I'll hit up the Admins.
- Nar its lagging now. Ps Can only see this cos im on a .world alt
I'm seeing this 12 hours later lol
- Its starting to catch back up
Looking at incoming request. .world is working OK for me. They seem to be batching stuff like I'll get nothing for 30 seconds, then over 3 seconds like 50+ requests.
Of course I don't know if their queue is backed up and I'm getting delayed stuff. I'd need to stop processing and look into the incoming queue to see what they're sending.
Bit of an edit. Looking at incoming again I can see under newest items, an entry from world that was 11 minutes old. Oh I have an idea. I'll see if this edit gets there in a timely manner.
Spoiler alert, it was instant.
Oh ignore me. It's specifically between those two instances I guess.
- Aparently its between big instances and lemm.ee only. All of said instances use cloudflair. I suspect that cloudflair has blocked/rate limited the larger instances from reaching lemm.ee.
  
  I think it must be hit/miss. Because I think those edits I made would have gone from my instance to world and then from world to .ee, and it was happening within seconds.
  So, presumably random stuff is being dropped or delayed?
I see it? Maybe I just haven’t noticed but I think .world is fine for me?
Edit: nvm you might be right. I just checked my posts in !futurama@lemmy.world. They normally get 60-100 up votes but the last one only got 3.
Admins have been talking about it, it looks like it has something to do with banning/unbanning users. Sometimes a ban/unban doesn't want to propagate to .ee and that causes a logjam somehow.
Smarter people than me are looking at it!
- If its that bloody bot that bans u if ur comment karma is to low thats gonna be funny. I got banned/unbanned like 10 times in an hour cos my comment karma was fluctuating near the bots threshold.
  
  That thing is such a nuisance. Makes the modlog nearly unusable.
  
  Til there is such a thing

25 comments