4d-llm

ONNX Runtime GenAI is a library that provides optimised inference for generative AI models — particularly LLMs — built on top of the ONNX Runtime engine. This component provides a server CLI and a class-based generic API with callback functions to use ONNX Runtime GenAI from 4D.

llama.cpp is an open-source C/C++ library and CLI tool for running large language models (LLMs) locally on consumer hardware. This component provides a class-based generic API with callback functions to use llama.cpp from 4D.