How to choose an embedding model: Vector model evaluation and selection guide for Traditional Chinese and English enterprise knowledge bases

Last month we built a knowledge base QA system for an insurance company and discovered a retrieval problem during testing: the same question returned the correct policy clause when searching for "policy borrowing," but returned unrelated passages when searching for "policy pledge borrowing." Both terms mean the same thing in business context, yet the embedding vectors separated them substantially in vector space.

The problem was not the RAG architecture but the foundation: embedding model selection. The model failed to capture the semantics of Traditional Chinese insurance terminology, causing retrieval to fail.

Embedding model selection is frequently overlooked in RAG development. Teams often focus on tuning prompts and replacing LLMs, but few rigorously evaluate the model that converts text into vectors. Incorrect vectors at the foundation will cause even sophisticated generation models to produce coherent answers from wrong source passages.

Why Traditional Chinese and mixed Chinese-English content are difficult

Three specific challenges arise in practice.

Vocabulary gaps between Traditional and Simplified Chinese present the first challenge. Most open-source and commercial models train primarily on Simplified Chinese. Traditional Chinese terminology that differs from Simplified equivalents often does not align well in vector space. Without explicit handling, pure Traditional Chinese content shows recall rates dropping by 5 to 15 percentage points, based on testing.

The second challenge is inline code-switching. Enterprise documents from Taiwan frequently mix languages within a single sentence, combining Chinese narrative with English technical terms. Models must parse both Chinese semantics and English terminology in the same context, treating both as meaningful content rather than noise.

The third challenge is domain-specific terminology. Finance, healthcare, and manufacturing each use specialized vocabularies that general-purpose models have not been trained on extensively. Related terms scatter across vector space as a result.

Comparing the three main model categories

The comparison covers three categories: OpenAI, Cohere, and open-source BGE/e5 models. Testing used the same corpus of Traditional Chinese and Chinese-English mixed text from an enterprise knowledge base, measuring Recall@10.

Based on testing, each model shows distinct strengths. OpenAI offers simplicity with a single API call to begin and performs well on English and code tasks. However, its performance on granular Traditional Chinese semantics does not match specialized models. Cohere delivers strong multilingual consistency, and its int8 and binary vector options reduce storage and search costs to approximately one-quarter of baseline performance. The cost reduction becomes significant at scale. BGE-m3 remains the primary open-source option. It outputs dense, sparse, and ColBERT representations simultaneously, supports hybrid retrieval, handles both Traditional and Simplified Chinese well, and can run on customer premises. Organizations handling sensitive data or high-volume workloads value this capability.

Which model to try first

Use the following decision logic to select a model:

If data cannot leave your premises or your corpus exceeds one million documents and you want long-term cost control, use BGE-m3 self-hosted with hybrid retrieval. If data can use APIs and you need multilingual support with strong recall, select Cohere embed-multilingual-v3 with int8 compression enabled. If you want to evaluate RAG viability without large-scale data, OpenAI text-embedding-3-large is fastest, producing working demonstrations within two days. For pure Traditional Chinese content with dense terminology such as legal documents or medical records, use bge-large-zh or fine-tuned BGE-m3. Gather several thousand positive and negative examples, fine-tune the model, and extract improved recall.

Changing embedding models typically yields less recall improvement than optimizing chunking and adding a reranker. The effective approach prioritizes chunking strategy and hybrid retrieval first, then applies bge-reranker or Cohere rerank, and only then considers embedding fine-tuning. Reversing this order directs resources inefficiently.

The insurance company example illustrates this priority. Switching to BGE-m3 with hybrid retrieval and a reranker layer allowed both "policy borrowing" and "policy pledge borrowing" to retrieve the same set of clauses, improving Recall@10 from 71% to 92%. Model selection provides the foundation, but practical value comes from operational use. This is why we conduct one week of thorough retrieval evaluation before deploying any RAG system. Technically correct retrieval in testing does not ensure adoption. The defining measure is whether operational staff trust and use the system daily.