Skip Navigation

User banner
Posts
13
Comments
262
Joined
2 yr. ago

  • (which created rows of that magnitude for updates does sound like federation messages outbound to me)

    rows=1675 from lemmy.ca here: https://github.com/LemmyNet/lemmy/issues/3165#issuecomment-1646673946

    It was not about outbound federation messages. It was about counting the number of comments and posts for the sidebar on the right of lemmy-ui to show statistics about the content. site_aggregates is about counting.

  • The issue isn't who fixed it it, the issue is the lack of testing to find these bugs. It was there for years before anyone noticed it was hammering PostgreSQL on every new comment and post to update data that the code never read back.

    There have been multiple data overrun situations, wasting server resources.

  • Right Now

    Working, this comment time

  • Fresh as of comment time:

  • What if instead of doing 3 things each time you posted a comment, it did 1300 things. And it did the same for everyone else posting a comment.

    Yes, that is what was happening in Lemmy before lemmy.ca called it out with AUTO_EXPLAIN PostgeSQL on Saturday, 8 days ago.

  • Latest, at the time of this comment: still over 4 SECONDS

  • huh?

    Please explain in detail what "huh" means in this context.

    As I said in the comment you replied to: do you know of the history of site_aggregates PostgreSQL table?

  • What are you asking for? lemmy.ml is the official developers server, and it crashes constantly, every 10 minutes it ERROR out, for 65 days in a row.

  • No. Care to explain please?

    On Saturday July 22, 2023... the SysOp of Lemmy.ca got so frustrated with constant overload crashes they cloned their PostgreSQL database and ran AUTO_EXPLAIN on it. They found 1675 rows being written to disk (missive I/O, PostgreSQL WAL activity) for every single UPDATE SQL to a comment/post. They shared details on Github and the PostgreSQL TRIGGER that Lemmy 0.18.2 and earlier had was scrutinized.

  • Do you know of the site_aggregates federation TRIGGER issue lemmy.ca exposed?

  • Probably too many users.

    if local.lemmyusers > 15, crash constantly because of PostgreSQL nonsense logic and Rust ORM.

  • 5801ms, terrible

  • That’s why I use lemm.ee

    1993: God, how we would love it if someone could tell us anything was “just that simple”, and then of course when you see a pie chart you go “Oh, a pie chart…”. I mean, it has more religious meaning now than a crucifix to see a pie chart. I mean, because…. why is that so popular? Because it reduces complexity. The complexity is very real but his little soundbites - 1993

    @garpunkal@lemm.ee - do you know of the history of site_aggregates PostgreSQL table?

  • As of the time of this comment, now crashing

    AGAIN Down

  • BACK ONLINE! YEY!

  • It’s mostly DDOS attacks though, not the influx of normal users.

    Really? As site_aggregates table getting 1500 rows updated on every single new comment and post local insert is just the tip of the iceberg of how nobody has scrutinized the PostgreSQL performance. Thank you to lemmy.ca last weekend for looking at AUTO_EXPLAIN on their data.