marauding_gibberish142 @ marauding_gibberish142 @lemmy.dbzer0.com

Posts

16

Comments

399

Joined

4 mo. ago

If you consider 4 B580s as enterprise, sure I guess

OP, I have been facing the same situation as you in this community recently. This was not the case when I first joined Lemmy but the behaviour around these parts has started to resemble Reddit more and more. But we'll leave it at that.

I think I have a solution for you if you're willing to spend $2-$3 a month - set up a VPS and run a Wireguard server on it. Run clients on your devices and the raspberry pi and connect to it.

As for your LAN: from the discussion you linked, it seems that Jellyfin will use the CAs present in the OS trust store. That's not very hard to do on Linux but I guess if you have to do it on Android you'd have some more trouble. In either case, using a reverse-proxy (I like HAProxy but I use it at work and it might be more enterprise than you need, for beginners Caddy is usually easier) will fix the trouble you're having with your own CA and self-signed certs.

I am interested in the attack vector you mentioned; could you elaborate on the MITM attack?

Unfortunately, if you don't have control over your network, you cannot force a DNS server for your devices unless you can set it yourself for every individual client. If I assume that you can do that, then:

1. Set up DNS server on Pi
2. Set up CA on Pi
3. Create root CRT, CSR and server certs from it (bare-minimim setup)
4. Copy over this stuff to Jellyfin image/VM, and copy root cert to clients trust store.
5. Run reverse proxy in front of Jellyfin and configure the correct IP address of the reverse proxy with an A record in your DNS server.
6. Configure reverse-proxy with server/application cert.
7. Use RethinkDNS on Android to pass everything through the wireguard server hosted on the VPS, and set private DNS to the DNS server hosted on the Pi.

I think that should do it. This turned out more complicated than I imagined (it's more of a brain dump at this point), feel free to ask if it is overwhelming.

I see. Thanks

I see. That solves a lot of the headaches I imagined I would have. Thank you so much for clearing that up

Thanks, but will NPUs integrated along with the CPU ever match the performance of a discrete GPU?

Thanks for the tip on x670, I'll take a look

Thanks for the comment. I don't want to use a networked distributed cluster for AI if I can help it. I'm looking at other options and maybe I'll find something

Your point is valid. Originally I was looking for deals on cheap CPU + Motherboard combos that will offer me a lot of PCIe and won't be very expensive, but I couldn't find anything good for EPYC. I am now looking for used supermicro motherboards and maybe I can get something I like. I don't want to do networking for this project either but it was the only idea I could think of a few hours back

I'd prefer that you reply with examples/an explanation of what I'm doing wrong instead of cursing

Thanks

I see. I must be doing something wrong because the only ones I found were over $1000 on eBay. Do you have any tips/favoured listings?

Used 3090s go for $800. I was planning to wait for the ARC B580s to go down in price to buy a few. The reason for the networked setup is because I didn't find there to be enough PCIe lanes in any of the used computers I was looking at. If there's either an affordable card with good performance and 48GB of VRAM, or there's an affordable motherboard + CPU combo with a lot of PCIe lanes under $200, then I'll gladly drop the idea of the distributed AI. I just need lots of VRAM and this is the only way I could think of.

Thanks

Thank you, and that highlights the problem - I don't see any affordable options (around $200 or so for a motherboard + CPU combo) for a lot of PCIe lanes other than purchasing Frankenstein boards from Aliexpress. Which isn't going to be a thing for much longer with tariffs, so I'm looking elsewhere

Thank you, but which consumer motherboard + CPU combo is giving me 32 lanes of PCIe Gen 4 neatly divided into 2 x16 slots for me to put 2 GPUs in? I only asked this question because I was going to buy used computers and stuff a GPU in each.

Your point about networking is valid, and I'll be hesitant to invest in 25Gbe right now

Thanks but I'm not going to run supercomputers. I just want to run 4 GPUs separately because of inadequate PCIe lanes in a single computer to run 24B-30B models

I'm not going to do anything enterprise. I'm not sure how people seem to think of it this way when I didn't even mention it.

I plan to use 4 GPUs with 16-24GB VRAM each to run smaller 24B models.

I don't understand your point, but I was going to use 4 GPUs (something like used 3090s when they get cheaper or the Arc B580s) to run the smaller models like Mistral small.

It is because modern consumer GPUs do not have enough VRAM to load the 24B models. I want to run Mistral small locally.

Thank you, I'll take a look

Thank you for the links. I will go through them