The Transformative Impact of Local Language Models
It is not an exaggeration to assert that the advanced language models (LLMs) powering AI bots such as ChatGPT and Gemini are reshaping our world. Increasingly, individuals rely on these tools for programming, writing, summarising, and online searching. Regardless of the direction AI takes in the coming years, LLMs will undoubtedly play a critical role in that future.
If you utilise platforms like ChatGPT, Claude, or Perplexity, you are engaging with an LLM. However, what many may not realise is that these models can be run locally on your own computer. This approach offers significant advantages, including offline access and enhanced privacy, as it eliminates the need to transmit data to the cloud for external analysis or review.
Additionally, by opting for local models, you can avoid monthly subscriptions to AI companies and usage fees. Numerous LLMs are available for free download, with offerings from major organisations like Meta and Google. While these models may not possess the same level of sophistication or speed as their paid counterparts, they are sufficiently effective for everyday tasks, allowing you to select according to your specific requirements.
This local setup does require more maintenance, and you may miss the convenience of simply launching an application like ChatGPT. For instance, you will need to handle updates yourself. Nevertheless, the trade-off is a more personalised and private AI system, and getting started is relatively straightforward.
Getting Started with Local Models
You can run local language models on Windows, macOS, and Linux, although macOS tends to be the preferred choice among AI enthusiasts. The ecosystem is more unified and consistent on Macs, which are produced by a single manufacturer. Moreover, Apple Silicon chips integrate the CPU, GPU, and RAM, which is advantageous for AI models.
No matter which platform you select, having ample RAM is beneficial. The bare minimum is 8 GB; however, if that is your only option, you will face limitations in terms of the size of LLMs you can run and their operational speed. Ideally, 16 GB is preferable, while 32 GB or more is recommended for deploying larger and faster models. To achieve optimal performance, a dedicated GPU with substantial VRAM is essential. Any amount exceeding 8 GB will significantly enhance processing capabilities, as this memory type is specifically designed for the tasks performed by AI models.
For Windows users, a dedicated Nvidia GPU can greatly improve performance. Nvidia graphics chips are particularly proficient in executing AI processes compared to standard processors, which is part of the reason for the close relationship between Nvidia and the AI boom. These dedicated graphics cards also possess their own RAM, providing additional processing space for AI models.
While there are no strict minimum specifications for running a local LLM, maximising RAM and utilising a dedicated graphics card will undoubtedly be advantageous. Additionally, you will need software to run the model, as well as the model itself. There are numerous options available for both, allowing you to choose based on your specific needs.
Recommended Applications and Models
For newcomers, LM Studio Bionic is widely regarded as the best AI application for both Windows and macOS, and it is available at no cost. Other popular and reliable alternatives include vLLM, Llama.cpp, Ollama, and GPT4All, which are also compatible with various operating systems. However, these programs tend to be slightly more technical and complex.
Once you have selected your AI application, the next decision involves choosing an LLM. The software options mentioned previously will guide you to several choices, and there are online repositories of models, with Hugging Face being the most notable, boasting over three million available models to select from.
Step-by-Step Guide to Setting Up a Local LLM
There are numerous configurations available for local LLMs, but we will provide you with a step-by-step guide for setting up LM Studio Bionic on Windows to illustrate the process. After downloading the installation file from the website and launching LM Studio Bionic, click on ‘Create Project’ to begin. You will then need to name your first project.
A blank conversation window will appear: by clicking on ‘Choose a Model’ in the dialogue box and then selecting ‘Get Local Models’, you will access the main model selector of LM Studio Bionic. Each available LLM is listed along with its size, popularity, and various details. Pay attention to the ‘Team Selections’ for recommendations to get started. Smaller models may not be as powerful but will download more quickly and take up less space on your system.
Returning to the chat interface, the functionality resembles that of a standard AI chatbot application. In the input box at the bottom, you can select from your installed AI models (if you have more than one). To the left of this box, there is a “+” icon for sending images and files, provided the current model supports such features.
The navigation panel on the left allows you to switch between your projects. Accessing the ‘Settings’ link (located at the bottom left of the navigation panel) will bring you to the preferences screen of LM Studio Bionic. Here, you can customise various aspects of the software, from managing deleted chats to modifying on-screen interface elements.
Click ‘Settings’ and then ‘Library’ to manage your existing models, or use the ‘Explore’ menu option to discover new ones. If you require support for images and documents, look for ‘multimodal’ models.
The final component of the LM Studio interface worth familiarising yourself with is the right sidebar, which can be shown or hidden using a button in the upper right corner. This sidebar provides additional options for managing files across projects and, if necessary, allows the programme to access your computer’s file system.