GenderNeutralBro @ GenderNeutralBro @lemmy.sdf.org

Posts

1

Comments

1,001

Joined

2 yr. ago

I don't see any mention of whether this uses local models or cloud models. I'm not interested in sending anything I care about it into the cloud.

Spot on.

SNW is good but I don't think we'll ever see a return to the old TV format of 20+ episode seasons. You can't do random episodic stories all that well in 6-12 episodes. Short seasons have no room to breathe.

Even Futurama has this problem with the two new Hulu seasons, and that's without the burden of an overarching plot to keep moving forward.

In Settings > Home, there's a Sponsored Shortcuts checkbox.

If you think this isn't related to human rights, then you've missed the point.

People have the right to use technology, and indeed we effectively need technology to exercise our right to free speech. You cannot have one without the other. Not anymore.

The right way to think about this that they are arbitrarily banning a topic of discussion simply because it is not dead-center average. This isn't even a legal issue, and the justification is utter nonsense (Facebook itself runs on Linux, like >90% of the internet). No government has officially asked them to do this, though the timing suggests that it is unofficially from the Trump administration.

This is about exerting control, establishing precedent, and applying a chilling effect to anything not directly aligned with their interests. This obviously extends to human rights issues. This is a test run.

Maybe if they distilled the coder version of qwen 14b it might be a little better but i doubt it. I think a really high quant 70b model is more in range of cooking functioning code off the bat. Its not really fair to compare a low quant local model to o1 or Claude with on the cloud they are much bigger.

That's a good point. I got mixed up and thought it was distilled from qwen2.5-coder, which I was using for comparison at the same size and quant. qwen2.5-coder-34b@4bit gave me better (but not entirely correct) responses, without spending several minutes on CoT.

I think I need to play around with this more to see if CoT is really useful for coding. I should probably also compare 32b@4bit to 14b@8bit to see which is better, since those both can run within my memory constraints.

Sounds cool. I'm using LM Studio and I don't think it has that built in. I should reevaluate others.

I'm not entirely sure how I need to effectively use these models, I guess. I tried some basic coding prompts, and the results were very bad. Using R1 Distill Qwen 32B, 4-bit quant.

The first answer had incorrect, non-runnable syntax. I was able to get it to fix that after multiple followup prompts, but I was NOT able to get it to fix the bugs. It took several minutes of thinking time for each prompt, and gave me worse answers than the stock Qwen model.

For comparison, GPT 4o and Claude Sonnet 3.5 gave me code that would at least run on the first shot. 4o's was even functional in one shot (Sonnet's was close but had bugs). And that took just a few seconds instead of 10+ minutes.

Looking over its chain of thought, it seems to get caught in circles, just stating the same points again and again.

Not sure exactly what the use case is for this. For coding, it seems worse than useless.

Yeah. If you want to be on the cutting edge of storage, look for a mobo that has PCIe gen5 m.2 slots. But really, PCIe gen4 m.2 drives are still pretty darned fast. You can get some with >7GB/sec transfer rates. Do you need >12GB/sec transfer from disk? Probably not. Is it cooooool? Sure. :)

This is a popular SSD these days, very good for the price: https://us-store.msi.com/PC-Components/Storage-Devices/Solid-State-Drive/M482-NVMe-M2-2TB-Bulk . If you want something high-end, look for an SSD with DRAM cache. Useful if you're writing massive amounts of data regularly, like video mastering or something like that, generally overkill otherwise.

I've been on the Ryzen x700 line for a long time now, first the 1700 and now on the 7700. No complaints, they rock. So I'd start by looking at the 9700. 9900 has more cores (and uses significantly more power), 9600 has fewer cores. Single-core performance is basically the same across the board, so it just depends on whether your workload can use a lot of cores or not. The "X3D" chips have additional CPU cache that supposedly improves performance in some workloads (notably in gaming). So if that's important to you, the 9800X3D is the natural choice.

Fortunately, my worn is not LLM related but just simple neural networks, but I don’t know how that might affect best practices for hardware.

Okay. If this is something you already do on existing machines, you'll be in good position to know how much memory you actually need, and then maybe give yourself a little room to grow. My guess would be that you're not working on massive models so you'll probably be fine with a mid-range card.

At the same time, a lot of AI/ML stuff is becoming mainstream and requires a ton of VRAM to get good performance. If you do any work with graphics, audio, or video, you might find yourself running large models without really thinking about it. There are lots of use cases for speech recognition models, for example, which are quite large. Photoshop already has some medium-sized models for some tasks. Noise reduction for audio can also be quite demanding (if you want to do a really good job).

As for system RAM...the world of DDR5 is indeed complicated. I don't think there's a huge need to go over 6000MHz RAM, and faster RAM brings some compatibility issues with some mobos/CPUs. It's also usually faster to use two sticks than four. So 2x32GB would be better than 4x16 in general.

For GPUs in particular, new gens with more VRAM are on the way, so buying the high-end now might leave you with something that feels obsolete by the time you grow into it. If you spend $750 now and $750 again in 2-3 years, you might end up better off than if you spent $1500 today and waited twice as long to upgrade. Particularly if you are able/willing to sell your old equipment to offset upgrade costs.

VRAM is king for AI workloads. If you're at all interested in running LLMs, you want to maximize VRAM. RTX 3090 or 4090 are your options if you want 24GB and CUDA. If you get a 4090, be sure you get a power supply that supports the 12V HPWR connector. Don't skimp on power. I'm a little out of the loop but I think you'll want a PCIe 5.0 PSU. https://www.pcguide.com/gpu/pcie-5-0-psus-explained/

If you're not interested in LLMs and you're sure your machine learning tasks don't/won't need that much VRAM, then yes, the 4070 Ti is the sweet spot.

logicalincrements.com is aimed at gaming, but it's still a great starting point for any workload. You'll probably want to go higher on memory and skew more toward multi-core performance compared to gaming builds, IMO. Don't even think about less than 32GB. A lot of build guides online skimp on RAM because they're only thinking about gaming.

But any 50 watt chip will get absolutely destroyed by a 500 watt gpu

If you are memory-bound (and since OP's talking about 192GB, it's pretty safe to assume they are), then it's hard to make a direct comparison here.

You'd need 8 high-end consumer GPUs to get 192GB. Not only is that insanely expensive to buy and run, but you won't even be able to support it on a standard residential electrical circuit, or any consumer-level motherboard. Even 4 GPUs (which would be great for 70B models) would cost more than a Mac.

The speed advantage you get from discrete GPUs rapidly disappears as your memory requirements exceed VRAM capacity. Partial offloading to GPU is better than nothing, but if we're talking about standard PC hardware, it's not going to be as fast as Apple Silicon for anything that requires a lot of memory.

This might change in the near future as AMD and Intel catch up to Apple Silicon in terms of memory bandwidth and integrated NPU performance. Then you can sidestep the Apple tax, and perhaps you will be able to pair a discrete GPU and get a meaningful performance boost even with larger models.

This will be highly platform-dependent, and also dependent on your threat model.

On PC laptops, you should probably enable Secure Boot (if it's not enabled by default), and password-protect your BIOS. On Macs you can disable booting from external media (I think that's even the default now, but not totally sure). You should definitely enable full-disk encryption -- that's FileVault on Mac and Bitlocker on Windows.

On Apple devices, you can enable USB Restricted Mode, which will protect against some attacks with USB cables or devices.

Apple devices also have lockdown mode, which restricts or disables a whole bunch of functionality in an effort to reduce your attack surface against a variety of sophisticated attacks.

If you're worried about hardware hacks, then on a laptop you'd want to apply some tamper-evident stickers or something similar, so if an evil maid opens it up and tampers with the hardware, at least you'll know something fishy happened, so you can go drop your laptop in an active volcano or something.

If you use any external devices, like a keyboard, mouse, hard drive, whatever...well...how paranoid are you? I'm going to be honest: there is a near 0% chance I would even notice if someone replaced my charging cables or peripheral cables with malicious ones. I wouldn't even notice if someone plugged in a USB keylogger between my desktop PC and my keyboard, because I only look at the back of my PC once in a blue moon. Digital security begins with physical security.

On the software side, make sure you're the only one with admin rights, and ideally you shouldn't even log into admin accounts on a day-to-day basis.

If you're running a consumer level GPU, you'll be operating with 24GB of VRAM max (RTX 4090, RTX 3090, or Radeon 7900XTX).

90b model = 90GB at 8-bit quantization (plus some extra based on your context size and general overhead, but as a ballpark estimate, just going by the model size is good enough). You would need to drop down to 2-bit quantization to have any hope to fit it in a single consumer GPU. At that point you'd probably be better off using a smaller model will less aggressive quantization, like a 32b model at 4-bit quantization.

So forget about consumer GPUs for that size of model. Instead, you can look at systems with integrated memory, like a Mac with 96-128GB of memory, or something similar. HP has announced a mini PC that might be good, and Nvidia has announced a dedicated AI box as well. Neither of those are available for purchase yet, though.

You could also consider using multiple consumer GPUs. You might be able to get multiple RTX 3090s for cheaper than a Mac with the same amount of memory. But then you'll be using several times more power to run it, so keep that in mind.

We need a lock before the lock screen to deal with all the functionality you can now access without unlocking it.

Sometimes I accidentally turn on the flashlight when the phone is in my pocket. I call it asslighting.

According to BabyCenter.com user data, it's the 13,388th most popular boy's name. https://www.babycenter.com/baby-names/details/root-1704004

Are we so desperate that we want what is basically malware ported to Linux? Ew. I didn't tolerate that shit when I was running Windows, and I'm sure not going to start now.

I'll just keep on voting with my wallet, and not pay money for such user-hostile products.

10 years ago, all I wanted was a brick-sized phone with a battery that won't quit. Now that we have cheap and reliable portable power banks, it's dropped down on my priorities list.

I guess a 22Ah battery would have a longer effective lifespan, too. I mean, if it drops to 50% capacity after a few years, that's still more that double most phones. So that would potentially solve my longevity issue.

This link is more Lemmy-friendly: !signalgroups@moist.catsweat.com (note that you might need to refresh the page after loading it, if this is the first time someone is loading it on your Lemmy instance).

Typically, I use a slow-charger overnight (a plain ol' USB type-A charger, which I think means 5W max), then top-up as needed during the day with USB-PD fast chargers. I generally do not top up to 100% during the day. I have adaptive charging enabled in settings.

That said, I'm a heavy phone user, and I've never had a phone that reliably lasts me a full day. According to aBattery, my current phone is at 750 charge cycles, which is just about 1 per day since I bought it. I'm not up to date on all the latest developments in battery tech, but I think it's normal for a battery to drop to 80% of its original max charge after 500 cycles. I don't think I have a dud on my hands, just an ordinary battery that is aging as expected. Like I said, it's still "fine". It hasn't started unexpectedly shutting off or anything like that.

I still have my old Pixel 2 (now 7 years old) that I occasionally use as a wi-fi device. I used that phone heavily for 2 years and very lightly for the remaining 5. I'm lucky if the battery lasts half an hour at this point; it's basically a desktop device now.

Also works on Twitch with the added benefit of NOT playing ads (you still get breaks, just with a placeholder screen instead of the commercial).

mpv has yt-dlp support built in, so it can just play the streams directly.