Motivation

In the past few months, I have started to use Obsidian in order to take notes about different topics, since I notices that I often searched for the same pages. This way, I could organize my knowledge and make searching easier. While going through the vault, I thought that it would be great to be able to ask something like ChatGPT for contents in my vault. Of course, if run locally, the model can’t be as complex as their model. I don’t have such hardware at home. But after some research, I found a promising post: Reddit user /u/Steerwall posted about the same topic on Reddit

They mentioned privateGPT and h2oGPT in their post. Since privateGPT seems to require a different project in order to use an UI, which I kinda wanted I decided to first try h2oGPT They also mentioned the Obsidian Markdown Loader which could come in handy later on.

Setup

Installing h2oGPT

Hardware

First of all, I don’t think that my PC is the best choice for it, but it’s what I have available: Ryzen 5 3600, 32GB DDR4, RTX 3060ti

Especially the GPU with 8GB of VRAM should be a big hurdle for most models.

(Failed) Docker Setup

Since I recently reinstalled linux on my computer, I needed to setup docker. Docker and docker compose were easily installed, but I also needed some packages from NVIDIA, in order to get GPU access.

yay -S nvidia-container-toolkit
yay -S nvidia-container-runtime

I also added the nvidia runtime to the configuration following their documentation.

However, I still got the error Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/moby/267a5d1550f0495582592749890ef2022174a014c0556443878df79330d9eb46/log.json: no such file or directory): /usr/bin/nvidia-container-runtime did not terminate successfully: exit status 127: unknown.

Trying to run nvidia-container-runetime resulted in the error nvidia-container-runtime: error while loading shared libraries: unexpected PLT reloc type 0x00. On to more research!

My first step was a system update. It was about time I did that anyway. Sadly, this did not fix it. Well, at least my system is now updated.

After trying some stuff and finding multiple errors linked to docker, I decided to try the manual install route for now. If this end up working easier, I don’t need to bash my head against this.

Manual Install

After cloning the repository and following the steps to create a virtual environment, I ran the command recommended in the FAQ:

python generate.py --base_model=h2oai/h2ogpt-gm-oasst1-en-2048-falcon-7b-v3 --hf_embedding_model=sentence-transformers/all-MiniLM-L6-v2 --score_model=None --load_4bit=True --langchain_mode='UserData'

To my surprise, it just worked. The model was downloaded and the server was started. The model used about 5GB of VRAM. I talked to the model for a bit, before starting the next step.

Loading my Obsidian data into the model

The UI has the option to upload files in the column on the left. However, I want to automatically import my entire vault, since there are many files in many folders. For this, I simply had to create a folder called user_path inside the root directory of the project. For this, I symlinked my vault to this directory. Afterwards, I ran python src/make_db.py --add_if_exists=True, which should update the existing database with my files. Restarting the server allowed me to ask questions about the contents of my vault. That was surprisingly easy.

What I learned

I should not slam my head against a wall when I encounter a problem. The docker way sounded much easier, but took me a lot longer to try. With the manual approach, I was done after about an hours, which mostly consisted of waiting for download to finish.

I will see, what I can do with this new tool in my toolbox. It really is exciting to see.