slacktoid @ slacktoid @lemmy.ml

Self-hoster/FOSS

---

Read more

Posts

0

Comments

941

Joined

4 yr. ago

Thank you for all your insight!! This is really helpful

I need to mess with tabbyapi. Doesn't help that there's like 2 tabbys, one is tabbyapi and the other is tabbyml. I am guessing tool support is at its infancy stage.

I mean of course the VCs will say that. They aren't the ones working. ¯(ツ)_/¯

Damn! Thank you so much. This is very helpful and a great starting point for me to mess about to make the most of my LLM setup. Appreciate it!!

Absolutely disgusting(ly cute)

Dude! That's so dope. I would really like your insights in how you tuned MoE. That would be a game changer as you can swap out unnecessary layers from the GPU and still get the benefit of using a bigger model and stuff.

Yeah it's a little hard to do inference with these limited VRAM situations and larger contexts. That's a massive pain

You're good. I'm trying to get larger context windows on my models so trying to figure that out and balance token throughput. I do appreciate your insights into the different use cases.

Have you tried larger 70b models? Or compared against larger MoE models?

Not to be that guy (he says as he becomes that guy) but the GPL is not a permissive license, BSD and MIT are. Tho imo GPL is the better and probably best license.

Also what models and use cases did you run it for? And what was your context window?

Lolol yes, it's a weird straight circle indeed.

However applying those fixes and then learning to fix it is a great way to learn how to troubleshoot and unb0rk your system.

Treat the response like you would of an LLM, it needs to make sense to you, you need to make sure they aren't messing with you or have given you a fix that only works in their case. Usually the best fixes are the simple ones. And it seems like even with the longer ones you're able to figure out your simple fixes which is awesome!

Tbh education was failing students long before AI. It started as soon as a degree requirement became the barrier of entry for a job. Companies don't care about what you learnt, just you have a degree and you have a high GPA. Students are just playing the game appropriately. learning was never the objective of education after a point.

I just map it to the abxy that steam suggests. Works great when I wanna keep using the trackpad or joystick at all times.

But states rights? What happened to states rights?

You can also host it yourself.

What about a gentle little girl who has cocaine in her body vs a big wild man? Place your bets!! Mines on cocaine.

Oooh interesting. Thank you!

Can you define unstable? I would like to know more about that

Lots of RAM, 32+, and compile in tmpfs.

Welcome to eartf

But then how will the normies pay for premium to not see ads