Will this work with Qlora and 4bit inference?

[jordancole21]

Super excited about this project! I'm in the process of reading the paper now! But just curious, are there any plans to make this work for 4bit or 8bit finetuning, so it can be applied to the larger opensource models?

[martinjaggi]

yes currently it relies on standard finetuning to transform an existing model to longer context. conceptually, it should be perfectly reasonable to try LoRA-finetuning instead for more memory efficiency, or even Q-LoRA.

we're playing with LoRA at the moment. one change you'll have to make is to also unfreeze the initial embedding layer, to allow it to learn the new landmark tokens. otherwise, should work out of the box.

let us know if you manage to get it running by combining the two codebases. we can keep the issue open if other people share their experience as well

[juanps90]

Do you think it should be doable with 4 bit finetuning as implemented on this repo?

https://github.com/johnsmith0031/alpaca_lora_4bit

How big of a dataset would be necessary for the finetuning to work past the 2048 token mark, assuming a LoRA approach?

[Alignment-Lab-AI]

Do you think it should be doable with 4 bit finetuning as implemented on this repo?

https://github.com/johnsmith0031/alpaca_lora_4bit

How big of a dataset would be necessary for the finetuning to work past the 2048 token mark, assuming a LoRA approach?

https://discord.gg/n9hXaBPWxx

on my server i have a system set up and this is one of our active problems were working on, i plan on figuring out how to alter this to pass custom datasets through the tokenization system on this and run it with axolotl which has built in qlora functionality

its actually a very cool development stack that involves samantha which is a new model developed by ehartford. the server has some of the more notable open sourced researchers there and if you think you can figure out how to combine the requisite elements were paying for compute to run experiments and make attempts, including h100s or clusters of a100's if needed

honestly id happily pay for you to train a 65b llama fine tune if you can figure how to do it with custom datasets on this repo

[juanps90]

Do you think it should be doable with 4 bit finetuning as implemented on this repo?
https://github.com/johnsmith0031/alpaca_lora_4bit
How big of a dataset would be necessary for the finetuning to work past the 2048 token mark, assuming a LoRA approach?

https://discord.gg/n9hXaBPWxx

on my server i have a system set up and this is one of our active problems were working on, i plan on figuring out how to alter this to pass custom datasets through the tokenization system on this and run it with axolotl which has built in qlora functionality

its actually a very cool development stack that involves samantha which is a new model developed by ehartman. the server has some of the more notable open sourced researchers there and if you think you can figure out how to combine the requisite elements were paying for compute to run experiments and make attempts, including h100s or clusters of a100's if needed

honestly id happily pay for you to train a 65b llama fine tune if you can figure how to do it with custom datasets on this repo

Thank you very much. I don't feel qualified enough to take on such a challenge, maybe a more knowledgeable person can do this.

I joined the discord you sent and will be paying attention to any developments around increased context length on LLaMA models.

[eugenepentland]

I am currently training a 7B TheBloke-WizardLM-7B-HF model. I worked around all of the issues to get it up and running. It's presently generating the tokens for the original dataset. If all goes well and I'm confident in the results, I'll send you a message about seeing if I can work with you to get a 65B model.

[Alignment-Lab-AI]

I am currently training a 7B TheBloke-WizardLM-7B-HF model. I worked around all of the issues to get it up and running. It's presently generating the tokens for the original dataset. If all goes well and I'm confident in the results, I'll send you a message about seeing if I can work with you to get a 65B model.

keep me posted!

[eugenepentland]

Just wanted to give a status update. I've been able to create a QLoRA but haven't seen any improvement as of yet. There were a lot of settings I had to tweak to get it running and I'm not sure if the issue is settings I've tweaked or just from not training long enough.

I will release a fork later today with what I've done and an explainaition of the issues I'm running into if anyone else wants to take a crack at it as well.

[Alignment-Lab-AI]

Just wanted to give a status update. I've been able to create a QLoRA but haven't seen any improvement as of yet. There were a lot of settings I had to tweak to get it running and I'm not sure if the issue is settings I've tweaked or just from not training long enough.

I will release a fork later today with what I've done and an explainaition of the issues I'm running into if anyone else wants to take a crack at it as well.

ill happily take a look, maybe we can crack it if we work at it as a unit
how was the models behavior with just the landmark?

[windprak]

I could train 65b on 4 node 32GPU if someone wants to help me with the training script

[eugenepentland]

I worked with @Alignment-Lab-AI and we were able to reproduce the results as described in the paper. Here is a guide if you just want to be able to test out landmark attention (not qlora yet, that's still in the works)

This guide is what for spinning up the model on lambda labs, so you might be able to skip some of the steps. Leaving all of the information in here just in case. This was done at 3 AM yesterday so there may be an issue you run into when setting up that I forgot about. If you have any issues let me know.

Python version 3.11
Cuda Version 12
H100 Graphics card 80GB VRAM
Ubuntu

Getting the models

mkdir models
cd models
When cloning the repos, you may need to figure out how to get git lfs install working. I had to do something but I don't remember what I did
git clone https://huggingface.co/epfml/landmark-attention-llama7b-wdiff
git clone https://huggingface.co/huggyllama/llama-7b

Installing the required libraries

(If you don't have conda)
wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh
chmod -v +x Miniconda*.sh
./Miniconda3-py39_4.12.0-Linux-x86_64.sh
conda create -n landmark
conda activate landmark
conda install pytorch -c pytorch
cd landmark-attention/
pip install -r requirements.txt
cd llama/
pip install -r requirements.txt
pip install sentencepiece

Creating the tuned llama file. This is how the merged model is created

cd /landmark-attention/llama
python weight_diff.py recover --path_diff /home/ubuntu/models/landmark-attention-llama7b-wdiff/ --path_raw /home/ubuntu/models/llama-7b --path_tuned /home/ubuntu/models/llama-tuned

Running Inference (run_test.py)
If you want to run it on your own prompt, the easiest way is to comment out the llama_base_pipe and change the generate_prompt function. You need to use their llama_mem_pipe for inference, so you aren't going to be able to take the model and have it function as expected in oobooga or something similar as of yet

cd /landmark-attention/llama
python run_test.py

We were able to get context up to 25k working and getting the correct answer, but it is SLOW. I think there are some optimizations that still need to be made to improve the performance. More comprehensive testing for evaluation needs to be completed as well.

[windprak]

Yes, I already did the 7B reproduction with 8 A10080GBs but I want to also try using larger models up to 65B running on 4 nodes. I guess I have to use deepspeed like shown in Llama-X. Is anybody is up for some collaboration?

[eugenepentland]

You can join our discord and we can work out the details. We are trying to get this in the hands of people as soon as possible.

https://discord.gg/fPUqTUWG

[Alignment-Lab-AI]

Yes, I already did the 7B reproduction with 8 A10080GBs but I want to also try using larger models up to 65B running on 4 nodes. I guess I have to use deepspeed like shown in Llama-X. Is anybody is up for some collaboration?

There's a lot of optimization left to do, you can reach us at toasts discord or at mine

https://discord.gg/X7E34Am9sn

[eugenepentland]

Just giving a status update, we've been able to train a 3B model using 20GB of VRAM, and 7B model using 29GB of VRAM, but have not trained long enough to get results and are still working out how it will get merged with the original weights.

I've made a fork that will keep up with my progress on this.
https://github.com/eugenepentland/landmark-attention-qlora

[eugenepentland]

Hey everyone, I got it working! We are running a longer training overnight, but from training 500 steps on the 7B model, we were getting up to 7k tokens (I tried 32k but got OOM). The accuracy wasn't great (60% accurate at 7k tokens) from lack of training so we haven't released the model yet. We should have something released tomorrow though.

Exciting stuff!