Lokales [[ki|AI]] Modell. Offline und mit CPU.

Download models from [[https://gpt4all.io/index.html|Website]].

<code>
curl --location --header "Content-Type: application/json" --data "{\"temperature\": 0.7,\"max_tokens\": 150,\"model\": \"gpt4all-falcon-newbpe-q4_0.gguf\",\"messages\": [{\"role\": \"user\", \"content\": \"Was ist KI?\"}]}" http://localhost:4891/v1/chat/completions
</code>

<code>
pip install gpt4all
</code>

<code python>
from gpt4all import GPT4All
import sys

falcon = "D:\\gpt\\ggml-model-gpt4all-falcon-q4_0.bin"
hermes = "D:\\gpt\\nous-hermes-13b.ggmlv3.q4_0.bin"
lora = "D:\\gpt\\gpt4all-lora-quantized.bin"

model = GPT4All(falcon)

def run(str):
    output = model.generate( str, max_tokens=1000)
    print(output)
    
if __name__ == "__main__":
    run(sys.argv[1])
</code>

=====Links=====

  * [[https://www.youtube.com/watch?v=l-Ji31HlC68|GPT4ALL-LocalDocs]]