A language model only knows what was in its training data, frozen at a cutoff date. It cannot see your company's docs, last week's changelog, or anything private. Ask it a question outside that training data and it either says it doesn't know, or worse, makes something up with total confidence.

RAG fixes this by retrieving relevant text at the moment of the question and handing it to the model as context, so the answer is grounded in real, current, verifiable source material rather than memory.

Indexing happens once, up front. Query happens every time someone asks a question.

The vector DB is the only thing the two phases share. This decoupling is why RAG scales well — you can re-index documents without touching the model, and swap the LLM without re-embedding anything.

Chroma runs embedded with a built-in embedding model, so this needs no API key to try.

• Chunks too large or too small — either buries the answer in noise or splits it across boundaries.
• No reranking — top embedding matches aren't always the most relevant ones.
• No fallback for "not found" — the model should say it doesn't know rather than force an answer from weak matches.
• Stale embeddings — re-index when source documents change, or answers drift out of date silently.

Not a replacement for your normal database — a companion for anything that needs "find similar," not "find exact."

A normal database matches on exact values: WHERE user_id = 42. Chroma matches on meaning. It stores each piece of text as an embedding — a list of numbers capturing what that text is about — and lets you ask "what's most similar to this?" even when no words overlap.

This is the backbone of most retrieval-augmented generation (RAG) systems: embed a knowledge base once, then at question time retrieve the nearest chunks and hand them to an LLM as context.

The geometry that makes "find similar" a maths problem instead of a keyword problem.

Each sentence is a vector pointing away from the origin. The angle between two vectors — not the distance between their tips — is what cosine similarity measures. The cat and dog sentences point in nearly the same direction (small θ, high similarity); the Python sentence points somewhere else entirely (large θ, low similarity).

A long sentence and a short sentence about the exact same topic can produce vectors of different lengths. If you measured plain distance, the longer one would look artificially "further away" just because its vector is bigger — not because the meaning differs. Measuring the angle instead cancels out length and isolates direction, which is what actually encodes meaning.

The dot product on top captures how much the two vectors point the same way; dividing by both magnitudes normalises them to unit length first, so only the angle survives. Result ranges from -1 (opposite meaning) to 1 (identical direction) — most related-but-different sentences land somewhere around 0.3–0.9. Chroma stores this as cosine distance (1 − similarity), so lower always means closer, matching the convention of other distance metrics.

The diagram above is 2D so it's drawable, but real embeddings from models like all-MiniLM-L6-v2 or OpenAI's text-embedding-3-small have 384 to 1536 dimensions. Each dimension is one axis in that space — not something you can point to and label ("dimension 42 = animal-ness"), but collectively the axes encode aspects like topic, tone, formality, and tense, learned automatically during training.

Pick based on whether data needs to survive a restart, and whether multiple processes share it.

A collection is roughly a table — a named group of embedded documents.

Whatever turns your text into vectors. Swappable — but not mid-collection.

upsert is the one you'll reach for most — it lets you re-run an ingestion script without hitting duplicate-id errors.

Narrow the search space before similarity even runs.

There's also where_document for full-text filtering on the raw content, e.g. {"$contains": "python"}.

Comparing against every vector doesn't scale — HNSW makes it approximate but fast.

With a few hundred documents, brute-force comparison is instant. With millions, it isn't. Chroma uses HNSW (Hierarchical Navigable Small World graphs): a layered graph where the top layer has few nodes and long jumps, and the bottom layer holds every vector densely connected to its neighbours.

A query starts at the top, greedily hops toward whatever's closest, drops a layer once it can't get closer, and repeats — funnelling down to a precise local search. Roughly binary search, but for high-dimensional space.

Embed once, retrieve at question time, hand the result to an LLM.

The one detail that matters most for quality: chunking. Embedding a whole document as one vector averages away detail. Splitting into ~500-token pieces with a little overlap keeps each chunk sharply about one thing.