noneabove1182 @ noneabove1182 @sh.itjust.works

Posts

89

Comments

149

Joined

2 yr. ago

Moderating

---

it apparently has larger implications than we might think, there was a big discussion in thebloke's discord

Yes that's a good comment for an FAQ cause I get it a lot and it's a very good question haha. The reason I use it is for image size, the base nvidia devel image is needed for a lot of compilation during python package installation and is huge, so instead I use conda, transfer it to the nvidia-runtime image which is.. also pretty big, but it saves several GB of space so it's a worthwhile hack :)

but yes avoiding CUDA messes on my bare machine is definitely my biggest motivation

lollms-webui is the jankiest of the images, but that one's newish to the scene and I'm working with the dev a bit to get it nicer (main current problem is the requirement for CLI prompts which he'll be removing) Koboldcpp and text-gen are in a good place though, happy with how those are running

for me it's painfully obvious when a phone is 60hz vs 120hz, i run mine at 120 and my wife doesn't care and runs at 60.. so yeah obviously some people just do not care or can't see it, others like me need it to be high refresh haha

Probably just because of the MOE (mixture of experts) lol

I actually just recently started having really good experiences with exllama on only 13B models, specifically I found the orca tuned ones to perform really well

I need a citation for that for sure, I know until very recently all software updates were non-OTA, meaning you had to drive to a dealership to get the software applied, which means dealers were hesitant to issue them, that could all be incorrect now and it's certainly incorrect for some of them, I'm positive there are car companies that put out OTA like tesla, i just don't know who they are

that last edit you added is probably the worst part, because it takes away from how solid Toyota and others are because it ruins the entire metric, Toyota is likely crushing it, and entirely possible Tesla is actually really really bad, but without the RIGHT metrics we can't actually draw any good conclusions, it's not just bad for tesla but for the whole market

Yeah there's definitely been some aggregious recall issues, but the problem is the stats include minor things that only required a quick OTA, so it skews the numbers awkwardly and means we can't properly judge the real problems they had

If they separated the numbers, we might see that either Tesla has very few real recalls, Tesla actually does have a lot of real recalls but also happens to have software ones, or it's about normal

And without separating all we can do is guess

100%, this number is skewed by the fact that tesla will basically "recall" for any minor issue because it's a simple software update, I imagine a lot of companies try to avoid recalls as aggressively and for as long as possible because it's a significantly bigger burden on them

I say this as someone who drives a Tesla but is still extremely judgemental of Tesla

That will certainly be amazing, but for now it's actually not bad to use either oobabooga web UI or koboldcpp to run the inferencing and provide a rest endpoint, cause you can trick basically any program into treating it as if it's OpenAI and use it the same way

Yeah I'm using it with home assistant :)

Basically I'm using oobabooga for inference and providing an API endpoint as if it were OpenAI, and then plugging that into Microsoft's guidance, which I then give a tool. The tool takes as input the device and the state, and then calls my home assistant rest endpoint to execute the command!

This is a great write up, thank you so much! Subscribed to your Lemmy too :)

I'm really intrigued by Llongma and super curious to see if it actually does anything vs scaling llama2, on llama 1 going from 2k->3k was pretty trivial (eventually), so I wonder what llama2s trivial scaling would be.

Also curious to see if it combats the U shaped attention in any way, I imagine little, but ready to be delightfully surprised!

Thanks again for making this, love contributions like this for our community!!

I'm trying to find a way to use it with Guidance to control my smart home, actually really doable with only a 13b model

I guess it depends on what you mean by usable, I think people have had success with ROCM, it's not as solid as CUDA of course but it's been more than usable

Security patches are out monthly less than a week after Google releases, OS updated are slower but have been getting better, only major downside is the lack of commitment to more than 2 years of OS, real kick in the shins for such an expensive phone but alas I'm a sucker for all it's other offerings

Camera is nice and especially unique so that is a bonus, but for me it's the physical fingerprint scanner, SD card slot, headphone jack, front firing speakers, and no notch in a flagship from a reputable company, could drop several of those requirements and not find any options and this has all of them

I'm devoted to Sony, can't switch to anything else.. they're the only one watching all the features I care about

There's a few things, one of the biggest is the commercial license and the 4k context

they may have added that recently, cause i wanna say it didn't but on my last phone (2 months ago) it did