Skip Navigation

InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)SH
shaserlark @ shaserlark @sh.itjust.works
Posts
11
Comments
378
Joined
10 mo. ago

I am trying to connect qbittorrent and wireguard.

Jump

    • why won't Lemmy let me comment or post unless Idisconnecte from my vpn

    Jump

  • Don’t ever mention Winnie the Pooh

    • Spanish PM calls to end social media anonymity, force digital ID

    Jump

  • No porn and drugs but „free speech“? Yeah right, no thanks. If my account on mastodon gets banned on an instance I go somewhere else.

    Of course if fediverse becomes too centralized the couple instances left might just defederate from everyone else, but OTOH what protects me from a couple individuals downvoting me into oblivion on Bastyon?

    They’re both decentralized in their own way but communities have to fight against malicious actors that attack the decentralization.

    • Spanish PM calls to end social media anonymity, force digital ID

    Jump

  • Typical politician, identifies the problem only to draw the absolute wrong conclusion.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Thanks for the reply, still reading here. Yeah thanks to the comments and reading some benchmarks I abandoned the idea of getting an Apple, it’s just too slow.

    I was hoping to test Qwen 32B or llama 70b for running longer contexts, hence the apple seemed appealing.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Congrats on being that guy

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • You’re aware that there’s the OpenAI API library right? https://github.com/openai/openai-python

    It’s really nothing fancy especially on Lemmy where like 99% of people are software engineers…

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Are you drunk?

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Yeah I found some stats now and indeed you’re gonna wait like an hour to process if you throw like 80-100k token into a powerful model. With APIs that kinda works instantly, not surprising but just to give a comparison. Bummer.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Thanks! Hadn’t thought of YouTube at all but it’s super helpful. I guess that’ll help me decide if the extra Ram is worth it considering that inference will be much slower if I don’t go NVIDIA.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Yeah I was thinking about running something like Code Qwen 72B which apparently requires 145GB Ram to run the full model. But if it’s super slow especially with large context and I can only run small models at acceptable speed anyway it may be worth going NVIDIA alone for CUDA.

    • Who else has unlocked the achievement?

    Jump

  • Proud of you. Done it a long time ago. Would do it again either

    • Pro-Israel judge poised to take over ICJ presidency

    Jump

  • Seems like that extra $150 million extra in hasbara money is already in good use judging from your genocide denial and post history.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Meh, ofc I don’t.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Thanks, that’s very helpful! Will look into that type of build

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • I understand what you’re saying but I’m coming to this community because I like having more input, hear about the experience of others and potentially learn about things I didn’t know about. I wouldn’t ask specifically in this community if I wouldn’t want to optimize my setup as much as I can.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Interesting, is there any kind of model you could run at reasonable speed?

    I guess over time it could amortize but if the usability sucks that may make it not worth it. OTOH really don’t want to send my data to any company.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • I’d honestly be open for that but would an AMD setup not take up a lot of space and consume lots of power / be loud?

    It seems like in terms of price & speed, the Macs suck compared to other options, but if you don’t have a lot of space and don’t want to hear an airplane engine constantly I’m wondering if there are options.

    • Using Mac M2 Ultra 192GB to Self-Host LLMs?

    Jump

  • Yeah the VRAM of Mac M series is very attractive for running models at full context length and the memory bandwidth is quite good for token generation compared to the price, power consumption and heat generation of NVidia GPUs.

    Since I’ll have to put this in my kitchen/living room that’d be a big plus but idk how well prompt processing would work if I send over like 80k tokens.

    • Danish king changes royal coat of arms in apparent rebuke of Trump over Greenland row

    Jump

  • Might be true if you’re white and not Muslim. It’s not mutually exclusive to be nationalistic and socialist at the same time if you think about it.