rufus @ rufus @discuss.tchncs.de

Posts

12

Comments

1,377

Joined

2 yr. ago

I think an LLM integrated into the IDE would be better suited when it comes to projects that aren't backed by a company like Microsoft who have a large amount of GPU compute to spare for their users.

Or it'd be bart of a CI pipeline. AFAIK that is theoretically already possible. You could configure the existing CI to feed the code through some form of AI code check.

I think we're already getting there. Lots of newer phones include AI accelerators. And all the companies advertise for AI. I don't think they're made to run LLMs, but anyways. Llama.cpp already runs on phones. And the limiting factor seems to be the RAM. I've tried Microsoft's "phi-2", quantized and on slow hardware, it's surprisingly capable for such a small model. Something like a ternary model would significantly cut down on the amount of RAM that is being used which allows to load larger models while also making it faster, everywhere. So I'd say yes. And it would also allow me to load a more intelligent model on my PC.

I think the doing away with matrix multiplications is also a big deal, but has little consequences as of today. You'd first need to re-design the chips to take advantage of that. And local inference is typically limited by memory bandwidth, not multiplication speed. At least as far as I understand.

I'd say if this is true, it allows for a big improvement in parameter count for all kinds if use-cases. But I've also come to the conclusion that there might be a caveat to that. Maybe the training is prohibitively expensive. I don't really know, at this point there is too much speculation going on and I'm not really an expert.

Reading up on the speculation on the internet: There must be a caveat... There is probably a reason why they only trained up to 3B parameter models... I mean the team has the name Microsoft underneath and they should have access to enough GPUs. Maybe the training is super (computationally) expensive.

HDMI 1.4 has a less than half the bitrate of the oldest DisplayPort standard. And regarding the newer HDMI standard: read the article. The forum are being jerks...

I think you're completely right with that assessment. Journalist used to be a reputable profession. And explaining things and processing raw information into something that can be consumed by the reader, deemed important. Especially getting it right. There is a whole process to it if you do it professionally. And curating content and deciding what is significant and gets an audience is equally as important.

Doing away with all of that is like replacing your New York Times with your 5-year-old and whatever she took from watching the news.

Hmm, I mean there is also publication bias. You're more likely to edit a Wiki page if you found a solution.... But you're also likely to rant and ask for questions if it's really bad.... There is a bit in the middle where it doesn't work that well. What I find super annoying if I find my question already posted 2 years ago and there isn't a solution posted underneath. That means someone either got it working and didn't update their post... or they moved on and it's impossible. But you're right, this really mostly happens to obscure and niche problems. Not if it's a ThinkPad or Dell laptop midel that has already sold millions of times. But somewhat likely if it's a newer high-end gaming mainboard or niche server that isn't common amongst the Linux-folks.

I read the other good advice with the firmware (UEFI) update. If there's still Windows on it, maybe run Lenovo Vantage and let it update the firmware.

I'd say it probably works out of the box.

I usually have a look at thinkwiki and the arch wiki. Since they don't have dedicated guides for this model, it usually means it's not supported at all and no one even tried, or it's a smooth ride and there just are no issues. Since it's not a niche product, I'd say it's the latter. And it's an older model without extravagant hardware... it should work fine.

Depending on where you're located a kidney could get you anything form that amount of money and more, to jail time. The internet says most people get less... So it's probably the illegal organ selling that gets you a good amount of money. But I doubt it's quick easy and clean... You could sell your blood, sperm and participate in clinical trials... That won't get you as much, though. And I don't know if you have to wait for that money.

I'd say sue some company if you're in the US... But that also takes years...

I dont think there is a method that is both legal and quick.

$24.300 is a lot of money... A bank loan? Rich relatives? Selling something big like a new car or a part of a house?

As far as I understand, their contribution is to apply what has proven to work well in the Llama architecture, to what BitNet does. And add a '0'. Maybe you just don't need that much text to explain it, just the statistics.

They claim it scales as a FP16 Llama model does... So unless their judgement/maths is wrong, it should hold up. I can't comment on that. But I'd like that if it were true...

It's not perfect. I've seen anwers that sounded good get more upvotes than the correct anwer. This especially reinforces urban legends.

And we have some idiots around. Happens to me occasionally that I give short and concise and correct answers, but they get one or two downvotes.

I completely agree on the removing downvotes isn't a good thing.

Yeah, I think comments shouldn't vanish along with a deleted post. And posts and comments shouldn't get automatically deleted alongside an account. Somehow everything is just the other way around here. And it's rarely a good thing.

Good question and self-reflection on your side!

The other people here are correct. You need to sign up for a first-aid / first-responder course.

It is practice and rehearsal. I did an extensive training back when I was a first responder. We learned what to do and the steps not to forget, sentences to memorize all of that. And then we had to exercise for days, do it over and over again. And that's what makes these procedures available to you. Especially in emergency situations where adrenaline is rushing in, emotions are on a high level, somebody is seriously hurt and bystanders are distracting everyone. Theoretical knowledge sometimes useless in those situations. The human body isn't made to think very thoroughly when overwhelmed by adrenaline. You need to have a feeling for what you're doing, have the mnemonic appear immediately in your head, the procedure stored in your muscle memory and remember what you did the last 3 times when that happened.

Go to an organization and do a course. If you can, do a 2-day one (if that's available in your country.) The short ones don't really focus on practice.

And it's a nice skill to have, anyways.

Boycott Nestlé

Do this first, boycott other companies who make a profit with child-labor and exploiting the poor. Don't buy from companies that steal people's water. Destroy the rainforest or harm animals unnecessarily.

I think this is far more important. Feel free to boycott shrinkflation on top. I try to do all of that if I get that choice and can afford it and have the knowledge available to me. But those products are also still on the shelves.

I'm not sure. Afaik the research is happening. And AI related stuff always happens faster than I can imagine. Ultimately I want the LLMs to hallucinate. They should be able to combine ideas and come up with new and creative answers and be more than just autocomplete. I think what we need is the LLM knowing what it knows and what is made up, and a setscrew. I can see this happening with a higher level of intelligence and/or a clever architecture. I'm not an expert on machine learning myself, however that is what I took from news, companies struggling with their chatbots and everyone wanting their AI assistant to provide factual information. And I don't see anything ruling that out completely. I mean we humans also sometimes get things wrong or mis-adjust our level of creativity. But I think the concept of facts can be taught to LLMs to some degree, they already seem to grasp it. And concepts have been proposed and things like AI agents that come up with ideas and other agents that check for factuality are in active use. Along with the big tech companies making their AIs cite the sources. In my eyes, progess is being made.

But this is why I currently don't use LLMs for important and unsupervised stuff, and i try to avoid them when I need correctness. However... I really like to tinker with them, do AI assisted storywriting, or have them come up with 5 creative ideas for a birthday party for my wife. That works well, and with a bit of trickery you can make them output more than the most obvious ideas. And I'm impressed by their ability to code, but as I said it's still far away from being useful to me. I currently don't fear for my job. And I additionally struggle with the size of models I can do inference with and their respective intelligence... We're in the Linux community here, so I think I can be open... I don't like big tech companies doing my compute and providing me with closed and proprietary services. I don't use ChatGPT, only open-weight models I can run myself. They aren't as smart, but I don't want the future of humankind to be shaped by services and good will of big tech companies.

Or took the wrong substance...

Anyways, the "spying" is probably not OP's most pressing issue.

Why did you suddenly become paranoid about specifically your hardware chips spying on you?

If you're trolling that's okay. If not: Please talk to someone. A real person who can check if you're alright.

I think so, too. I mean the traditional history search and command option suggestions are instant and come at no additional cost. I don't know how fast ChatGPT is, I only ever play around with local LLMs. And roughly exploring what Github Copilot is about, just made my laptop fans spin on max and started to drain the battery really fast. Would be the same for an 'AI' terminal. And when asking the LLMs for shell commands I got mixed results. It can do easy stuff. So I guess for someone who wonders how to find the IP address... It'll do the trick. But all the things I tried asking some chatbots that would have been really useful to me, failed. It hallucinated parameters or did something else. And I needed to google it anyways or open the man page.

I'm not sure, I currently don't see me using such tools. I like talking to chatbots and have them draft stuff and provide me with ideas. But I also like computers in the other way, that they are machines that just follow my orders and don't talk back. And when working in the terminal or coding, it seems to distract me if suggestions pop up and I need to read them and decide what to do, or occasionally laugh... For me it seems to work better if I think about something, have an idea in my head and type it down without discussing it with the machine... I mean not 100% of the time, sometimes a suggestion helps... But I think I rather have the chatbot in a separate window and only loosely tied into my workflow if at all. And I don't like proprietary and cloud-based products for something like this.

There are lots of fish shell extensions, zsh stuff and loads of things that make suggestions, autocomplete, remember your shell history and remember frequently executed commands and visited directories. All of that works WAY better than the AI suff. (And sometimes also has nice pop-up menus.)

So compared to plain bash without autocomplete and Ctrl+R it may be useful. It is probably a step back for everyone else. Especially if they roughly know what they're doing.

But I didn't try this specific software. Maybe I would if it were free software and connected to a local LLM.