Couch "Spud" Berner @ StubbornCassette8 @feddit.nl

Posts

0

Comments

7

Joined

2 yr. ago

What are the hardware requirements on these larger LLMs? Is it worth quantizing them for lower-end hardware for self hosting? Not sure how doing so would impact their usefulness.

Please conduct yourself.

I've been getting around that mostly with Chocolatey and other PowerShell scripts on Windows. I'm sure the same can be done on Linux.

Are you hosting the desktop or server version of Trilium?

Also not bashing Trilium. If it works for you, great. It's a self hosted solution to keep your notes. Can't complain about that!

Have you considered using Obsidian paired with Syncthing?

I keep my Obsidian notebook in Syncthing folders and find it works well enough across Windows and Android devices. The plugins transfer too. You would only have to trust the authors when setting up Obsidian for the first time after pointing to the right directory.

You will have conflicts with certain files if you open Obsidian on multiple devices at the same time. The note.md files should be preserved, which I think is what is really needed.

Oh wait, I think I misunderstood. I thought you had local language models running on your computer. I have seen that be discussed before with varying results.

Last time I tried running my own model was in the early days of the Llama release and ran it on an RTX 3060. The speed of delivery was much slower than OpenAI's API and the material was way off.

It doesn't have to be perfect, but I'd like to do my own API calls from a remote device phoning home instead of OpenAI's servers. Using my own documents as a reference would be a plus to, just to keep my info private and still accessible by the LLM.

Didn't know about Elevenlabs. Checking them out soon.

Edit because writing is hard.

Can you share details? Been thinking of doing this with a new PC build. Curious what your performance and specs are.