Lokales AI Modell. Offline und mit CPU.

Download models from Website.

curl --location --header "Content-Type: application/json" --data "{\"temperature\": 0.7,\"max_tokens\": 150,\"model\": \"gpt4all-falcon-newbpe-q4_0.gguf\",\"messages\": [{\"role\": \"user\", \"content\": \"Was ist KI?\"}]}" http://localhost:4891/v1/chat/completions

pip install gpt4all

from gpt4all import GPT4All
import sys
 
falcon = "D:\\gpt\\ggml-model-gpt4all-falcon-q4_0.bin"
hermes = "D:\\gpt\\nous-hermes-13b.ggmlv3.q4_0.bin"
lora = "D:\\gpt\\gpt4all-lora-quantized.bin"
 
model = GPT4All(falcon)
 
def run(str):
    output = model.generate( str, max_tokens=1000)
    print(output)
 
if __name__ == "__main__":
    run(sys.argv[1])

Links