Florida ocean temperatures peak to almost 100 degrees amid heatwave: "You really can't cool off"
Terrasque @ theterrasque @infosec.pub Posts 0Comments 299Joined 2 yr. ago
Terrasque @ theterrasque @infosec.pub
Posts
0
Comments
299
Joined
2 yr. ago
"I had the misfortune to come across a leaked video of your CEO
<Google name>
having some really questionable sexual intercourse with a really sketchy character, and it was truly disgusting. I can not in good conscience support a company led by such a horrible individual"If they want feedback, give them feedback.
I need to be able to access it from both phone and PC, and in emergency from a random PC/phone. Also, I used the "share as web link" a lot in notion.
In addition it would be nice to be able to access it at work without having to install anything.
AppFlowy isn't web based? I couldn't find an option to just run a server and access it via browser anywhere in the install instructions.
My experience too. The few times I've been stuck and decided to try chatgpt, it's been completely unhelpful, at best suggesting basic things that I checked within the first 5 minutes of troubleshooting.
That was the best case. Worst case it'd sprout some plausible looking nonsense that took time to check and dismiss.
With the occasional hostile takeover
First of all, this link is just to C# bindings of llama.cpp and so doesn’t contain the actual implementation.
I know, it's my code. I refactored it from some much less readable and usable c# code. I picked it because it more clearly shows the steps involved in generating text.
How do you know LLMs can’t look ahead? [...] How do you know it hasn’t written out the entire response in memory already after which it only shows you the first word?
Firstly, it goes against everything we know so far of how they operate, and secondly.. because they can't.
If you look at the C# code, the first step is in _process_tokens function, where it feeds the context into llama_eval. That goes through each token and updates the internal memory / model state. Since it saves state, if you already have processed some of the tokens you can tell it to skip them and start on the new ones.
After this function you have a state in memory, the current state of the LLM, as a result of the tokens it's seen so far.
When we are done with that, we go to the more interesting part, the _predict_next_token function. Note that that takes a samplingparams parameter. It then set some options, like if top k is not set it's set to length of the model's vocabulary (number of tokens it knows about), and repeat_last_n, if not set, is set to the length of the existing context.
The code then gets the model's vocabulary, aka all the tokens it knows about, and then it generates the logits. The logits is an array the length of the vocabulary, with a number for each token showing how likely that one is the next token. The code then adds any specified token bias to that token's number. Already here, even if it already had a specific answer in mind, you can see problems starting.
Then the code adds token repetition penalty, based on the samplingparams. This means that if a token repeats inside the given history, it's value will be lowered according to the repeat_penalty. Again, even if it had a specific answer, this has a high chance of messing that up. The same is done for frequency and presence. For more details of what those native functions do, you can see the llama.cpp source - they have the same name there.
After all the penalties are applied, it's time to pick the token. If the temp is 0 or lower, it just picks the highest rated token (aka greedy sampling). This tends to give very boring and flat responses, but it's predictable and reproduceable, so it's often used in benchmarks of various kinds.
But if that's not used (which it almost never is in "real" use), there are several methods. You have MiroStat, which tries to create more consistent quality between different answer lengths, and the "traditional" using top-k, top-p and temperature.
Common for them is however that internally it produces a top list of candidates, and then pick one at random. And that's why a LLM can't plan ahead.
When a token ID is eventually produced it returns the new ID, that gets added to context, the text equivalent of the token is looked up and sent back to the UI, and the new context is fed into llama_eval and the process starts again.
For the LLM to even be able to plan an answer ahead it must know of all penalties and parameters (or have none applied), and greedy token prediction must be used.
And that is why, even if it had some sort of near magical ability to plan ahead that we just don't know is there, at the end of the day it could still not plan a specific response.
Just ask if you want some clarification.
As for GPU, I'm waiting.. IMHO it's just too expensive now. And sadly, Nvidia is currently the only game in town. Some software works on amd, but just about everything works on Nvidia.
That said, my PC has 48gb system ram, and I can run 65b models on it with about 1s per token. With a few layers offloaded to my 10gb GPU. That would otherwise require 2x 3090 or 4090 (2x4090 would be about 20x faster though..)
No reason? It's probably meetings, then more meetings, add some meetings, and you guessed it, meetings.
Like the follow tab mentioned it's probably first product owners meetings to agree on what a user would expect.. and there's always someone having a wild opinion or two that needs to be "hashed out". Then when that's done it's meetings with the UX team, then they have a meeting on their own, then a new meeting with product owner, UX and designers, then after that frontend team is in the loop, then back to UX and prod owner, then a new round, then it's time for backend to come in, first one with PO and frontend, then a technical one to agree on how to do it, then database team is involved, they refuse to change a small thing and expected functionality needs to be changed, back to PO, UX, frontend, backend, and then finally maybe a dev or two can sit down and add it. Which takes 2 hours. After six weeks of meetings.
And then comes testing of course, and signoff on the functionality.
"Fast" is nowhere in enterprise development.
Another thing, llama.cpp support offloading layers to gpu, you could try opencl backend for that for non-nvidia gpu's. But llama.cpp can also run on cpu-only, with usable speed. On my system, it does about 150ms per token on a 13b model.
koboldcpp is probably the most straight forward to get running, since you don't have to compile, it has a simple UI to set launch parameters, and it also have a web ui to chat with the bot in. And since it use llama.cpp it support everything that does, including opencl (clblast in launcher)
LLM's don't ingest information as such. The text gets broken into tokens (parts of words, like "catch" can be "cat" + "ch" for example), and then run through training. Training basically learns the statistical likelyhood of which token follow an array of existing tokens. It's in some ways similar to a markov chain, but of course much more complex. It has layers of statistics, and preprocessors that can figure out which tokens to give higher precedence in the input text.
Basically the more parameters, the more and subtler patterns it can learn. Smaller models are often trained on fewer tokens than bigger ones, but it's still a massive amount. IIRC it's something like 1T tokens for 7 and 13, and 1.4T tokens for 33b and 65b. In comparison to the models I linked, ChatGPT 3.5 is rumored to be 175b parameters.
In addition to just parameter size, you have quantization of the numbers. Originally in a model each parameter number is 16bit float, it turns out you can reduce it to 8bit int or even 4 and 3 bit with not too much hit at complexity. There's different ways to quantize the parameters, with varying impact on the "smartness" of the model. By reducing the resolution of the numbers, the memory needed for the model is reduced, and in some cases the speed of running them is increased.
When it comes to training, the best results have been achieved with full 16bit fp, but there are some techniques to train on quantized models too. The results I've seen from that is less impressive, but it's been a while since last I looked at it.
Edit: I mentioned qlora previously, which is for training quantized models. I think that's only available for gpu though.
Edit2: This might be a better markov chain explanation than the previous link
Crazy to see my already quite slim image being reduce all that much!
*size.original='1.0 GB' *
I mean.... I don't know what that does, of course, but I would rarely use "quite slim" to describe that :D
"size.original='66 MB' size.optimized='33 MB'"
This one's nice though
You can probably run a 7b LLM comfortably in system RAM, maybe one of the smaller 13b ones.
Software to use
- https://github.com/ggerganov/llama.cpp - command line. Basic, flexible.
- https://github.com/LostRuins/koboldcpp - Precompiled llama.cpp with ui - easy to start with
Models
In general, you want small GGML models. https://huggingface.co/TheBloke has a lot of them. There are some superHOT version of models, but I'd avoid them for now. They're trained to handle bigger context sizes, but it seems that made them dumber too. There's a lot of new things coming out on bigger context lengths, so you should probably revisit that when you need it.
- https://huggingface.co/TheBloke/orca_mini_v2_13b-GGML - the q3_K_M.bin perhaps - might still be too big, depending on what you're running in the background
- https://huggingface.co/TheBloke/orca_mini_3B-GGML - very small model. Not sure how well it'll do
- https://huggingface.co/TheBloke/airoboros-7B-gpt4-1.4-GGML
- https://huggingface.co/TheBloke/vicuna-7B-v1.3-GGML
- https://huggingface.co/TheBloke/WizardLM-7B-V1.0-Uncensored-GGML
Each have different strengths, orca is supposed to be better at reasoning, airoboros is good at longer and more storylike answers, vicuna is a very good allrounder, wizardlm is also a notably good allrounder.
For training, there are some tricks like qlora, but results aren't impressive from what I've read. Also, training LLM's can be pretty difficult to get the results you want. You should probably start with just running them and get comfortable with that, maybe try few-shot prompts (prompts with a few examples of writing styles), and then go from there.
To be safe I should download backups once a month or so.
Maybe look into borg and https://www.borgbase.com/ - they give 10gb free. I sat it up for some important data I would want to keep if utter disaster struck yesterday, and was pretty straight forward.
You could also set up a more ghetto time machine like rsync with https://github.com/laurent22/rsync-time-backup if you have a machine on your network with ssh access from outside.
The issue here is that you are describing the goal of LLMs, not how they actually work.
No, I am describing how they actually work.
it cannot achieve this via rudimentary statistics alone because the model simply does not have enough parameters to memorize which token is more likely to go next in all cases.
True, hence the limitations. That would require infinite storage and infinite compute capability.
Also, going “one token at a time” is only a “limitation” because LLMs are not accurate enough.
No, it's done because one letter at a time is too slow. Tokens are a "happy" medium tradeoff.
The space token effectively just makes it reflect on the conversation.
It makes a "break" of the block, which lets it start a new answer instead of continuing on the previous. How it reacts to that depends on the fine tune and filters before the data hits the LLM.
To be clear, I do not believe LLMs are the future.
I have just said that LLM's we have today can't fix the problems with false data and hallucinations, because it's a core principle of how it operates. It will require a new approach.
You could add a rocket engine and wings to a pogo stick, but then it's no longer a pogo stick but an airplane with a weird landing gear. Today's LLM's could give us hints to how to make a better AI, but that would be a different thing than today's LLM's. From what has been leaked from OpenAI GPT4 has scaling issues so they use mixture of experts. Just throwing hardware at it is already showing diminishing returns. And we're learning fascinating new ways of training them, but the inherent problem is the same.
For example, if you ask an LLM if it can give an answer to a question, it will have two paths to go down, positive and negative. Note, at the point where it chooses that it doesn't know how to finish it, it doesn't look ahead. But it sees for example that 80% of the answers in the texts it's been trained on starts with a positive, then it will most likely start with "yes" - and when it does that it will continue to generate an answer - often very convincing and plausibly real looking answer, because it already committed to that path.
And as for the link about teaching it backspace token, the comments there are already pointing out the issue:
It's interesting that in the examples (Table 3 on page 21), the model uses the backspace token to erase the randomly-added token from the prompt, but it does not seem to ever use the token to correct its own output. I'm curious how frequently the model actually uses this backspace token in practice - and if the answer is "vanishingly rarely", what is the source of the improved Mauve score and sample diversity they show? Is it just that the different training procedure gives an improvement?
For it to use the backspace, wouldn't it have to predict the wrong token with greater confidence than the corrected token? I would think this would require more examples of a wrong token + correction than the correct token, which seems a bit odd.
Almost none of the text it's trained on has a backspace token, and to finetune it in is tricky since it's a completely new concept - and remember it's still doing token for token - so it would have to write a token and then right after find out that it's more likely to send a backspace token than to continue it. It's interesting, and LLM's can pick up on some crazy patterns, but I'm skeptical.
An entity that can consistently predict the next word of any conversation, book, news article with extremely high accuracy is quite literally a god because it can effectively predict the future
I think you're reading something there other than what I said. Look, today's LLM's ingest a ton of text - more accurately tokens - and builds up statistics of which tokens it sees in that context. So statistically if you see the sentence "A nice cup of " statistically the next word is maybe 48% coffee, 28% tea, 17% water and so on. If earlier in the text it says something about heating a cup of oil, that will have a muuch higher chance. It then picks one of the top tokens at (weighted) random, and then the text (array of tokens) is fed in again into the LLM and a new prediction is made. And so on it continues until you stop the loop (usually from a end token or a keyword you're looking for). Larger LLM's are better at spotting more subtle patterns - or more accurate it got more layers of statistics that's applied - but it still has the fundamental issue of going one token at a time and just going by what's most likely to be the next token.
It many cases it does. For example, if GPT gives you a wrong answer, you can often just send an empty message (single space) and GPT will say something like: “Looks like my previous answer was incorrect, let me try again: blah blah blah”.
Have you tried that when it's correct too? And in that case you mention it has a clean break and then start anew with token generation, allowing it to go a different path. You can see it more clearly experimenting with local LLM's that have fewer layers to maintain the illusion.
This says nothing. You are effectively saying: “Until we can find a new approach, we can only expand on the existing approach” which is obvious.
But new approaches come all the time! Advances in tokenization come all the time. Every week there is a new paper with a new model architecture. We are not stuck in some sort of hole.
We're trying to make a flying machine by improving pogo sticks. No matter how well you design the pogo stick and the spring, it will not be a flying machine.
Some things are inherent in the way the current LLM's work. It doesn't reason, it doesn't understand, it just predicts the next word out of likely candidates based on the previous words. It can't look ahead to know if it's got an answer, and it can't backtrack to change previous words if it later finds out it's written itself into a corner. It won't even know it's written itself into a corner, it will just continue predicting in the pattern it's seen, even if it makes little or no sense for a human.
It just mimics the source data it's been trained on, following the patterns it's learned there. At no point does it have any sort of understanding of what it's saying. In some ways it's similar to this, where a man learned how enough french words were written to win the national scrabble competition, without any clue what the words actually mean.
And until we get a new approach to LLM's, we can only improve it by adding more training data and more layers allowing it to pick out more subtle patterns in larger amounts of data. But with the current approach, you can't guarantee that what it writes will be correct, or even make sense.
I knew an addict once.. he died.. do you think it's a joke? You should really consider changing your name
Deleted
Permanently Deleted
Jump
Sir, squirrels are not supported by our internet package. Until you stop using squirrels on our internet connection, I'm afraid we can't help you.
Sure about that "not a legal email character" part?
As a Norwegian: