Are Local LLMs the Answer for Confidential Information?

Most valuators are well aware of some of the benefits that large language models (LLMs) like ChatGPT, Claude, Gemini, and CoPilot can have for them, but many worry about the privacy implications — specifically, the risk of uploading confidential documents. Those documents could, for example, become part of the “training” for commercial, cloud-based AI tools and potentially pop up as part of response to another user’s query down the road. Read on to learn about a potential work-around, as well as its limitations.

The Local LLM Option

The privacy concern is entirely valid. But Colin Brown, founder and chief technology officer of Syncnet, which helps consultants leverage AI and software, says a local LLM could eliminate the risk. According to Brown, setting up a local LLM on your own computer has become much easier, thanks largely to Hugging Face, a repository of millions of open-source, pre-trained models that valuators can download to their own computers.

In a nutshell, you can download a model and give it a specialty (for example, reading industry reports). You then load the model with applicable documents. For a testifying valuator, for example, those might include relevant caselaw, valuation standards, samples of your best valuation reports, and similar materials the model can use as reference points when you submit prompts.

The model produces responses to your prompts based on those documents, without the documents ever leaving your computer. You can disconnect from the Internet for an even greater assurance of privacy.

Brown takes you through the detailed steps for setting up a local LLM in this video.

Local LLMs vs. Commercial LLMs

The idea of a local LLM is enticing, and it does effectively protect confidential information. Valuators need to be aware, though, of the substantial limitations.

To put it simply, the differences between a local LLM and the commercial models are like the differences between a Cessna and a Boeing 787. A Cessna is useful and capable in a certain narrow environment, but a 787 has vastly more thrust, payload, instrumentation, and range — and it’s backed by airport-scale infrastructure.

Local LLMs are compressed versions of the massive commercial models. They forfeit size and precision in exchange for avoiding the privacy risks associated with cloud infrastructure and commercial model training.

That said, it’s also worth noting the difference between a local LLM for some valuation-related tasks and the traditional manual methods, which could be described as the difference between jumping on a Cessna for 100-mile trip and walking to your destination. “If you download an open-source LLM that has been trained on the English language, basically anything that’s ever been recorded, it’s going to have a higher IQ than most of us,” Brown says.

In other words, local LLMs have their uses, but valuators should take into account the following significant caveats: 

1. Limited computing power

Commercial LLMs have exponentially more memory and bandwidth than local LLMs. The average desktop or laptop won’t have enough computing power to efficiently run a local LLM. Brown says you’ll need either a gaming laptop with a built-in GPU (graphical processing unit) or a desktop with a GPU with VRAM (like a NVIDIA graphics card) inserted. A commercial LLM, on the other hand, has thousands of GPUs, so it’s just no contest. Plus, a GPU can cost as much as your computer, and a gaming laptop can top $2,000. Still, Brown says, “it’s well worth it because your computer will run better.”

Not surprisingly, the smaller computing power means it can take a while to even get your local LLM up and running. Brown says that, for example, loading your documents into a local LLM isn’t a quick process. “It takes time because the model is going to convert the documents into a vector database. It could take a full day, but then they’re in there, and you can query the model and ask questions against those documents.”

2. Capability constraints

Local LLMs contain “static knowledge” — think of it as their brains being frozen on the date they’re trained with your documents. They don’t update or improve automatically unless you set up a RAG (retrieval-augmented generation). Commercial systems are constantly improving and “learning” new information.

Local LLMs also have limited “context capacity.” For example, you could load five 200-page annual reports, and the model might “forget” the beginning of the first report by the time it gets to the fifth. 

3. Instability

Stability in the AI context generally refers to uptime (versus downtime, or “crashes”) and output consistency (does the tool follow instructions or produce the same answer every time). Local LLMs suffer in both arenas compared with commercial models, for several reasons.

For example, as noted above, commercial models run on thousands of GPUs. To fit an LLM onto the 8GB or 16GB of VRAM typical for a local LLM, the model’s brain cells (known as weights) are compressed substantially. The compression can introduce “noise” compared with the precision you get from a model running on the power of a sprawling data center.

In addition, commercial LLMs undergo constant reinforcement learning based on human feedback, which keeps them from straying off-topic. Local models aren’t as well trained and therefore are more likely to go off script. (Bear in mind, though, that large LLMs can and do also produce inconsistent responses to the same query.) 

4. Hallucination risk

You’ve probably seen some of the widely published examples of AI hallucinations — when AI tools essentially make up things. This is why AI tools are sometimes compared to an intern eager to please you; they create things out of whole cloth to make you happy.

The noise that’s introduced into a local LLM by compression means the model is less “certain” when predicting the next word (remember, these tools are predictive, not logical). This increases the odds of hallucinations. For example, a local LLM might confidently provide a financial ratio, citation, or interpretation that doesn’t exist or is incorrect. The consequences could prove disastrous for a valuator.

Local LLM Use Cases for Valuators

In light of the above, local LLMs generally aren’t advisable for tasks involving analytical reasoning, financial interpretation, or valuation conclusions — basically, anything that must be defensible.

That’s not to say, however, that they don’t have their place. You could, for example, use a local LLM effectively to draft memos or summarize documents. You might upload a 50-page document and request a “Table of Key Risks” and financial ratios — but the prompt should request footnotes so you can verify the model’s response in the document yourself.

And, while it goes without saying that you wouldn’t use a local LLM (or any LLM) to write a valuation report, the model could still bring value to the process. Brown suggests submitting your report and prompting the local LLM to compare it with other reports you’ve done in the past and highlight any shortcomings in the current report.

Proceed with Caution

Local LLMs can’t compare with commercial LLMs in numerous critical ways, but they do have real uses for valuators, particularly when dealing with sensitive materials. Just keep the tradeoffs in mind.