Fine-tuning do Llama 3.3 para CID em espanhol: dataset, custos e benchmarks contra Claude

TL;DR

We fine-tuned a Llama 3.3 8B on the Spanish ICD-11 catalog (72,000 codes + clinical synonyms) to convert symptom descriptions into the right code. We use Unsloth for efficient fine-tuning (4-bit QLoRA, 2 rented A100 for 6h), vLLM to serve at high throughput, Qdrant as a complementary RAG (covers new codes without retraining) and lm-eval-harness + an in-house golden set for benchmarks. The resulting model (numoru-ia/icd-cie-es-8b) published on Hugging Face reaches 87% top-1 accuracy versus 94% for Claude Opus and 82% for GPT-4o-mini, but with inference cost 18× lower than Opus and 100% on-prem — critical for clinics and insurers that cannot send data to external APIs.

Why fine-tuning and not just RAG

Three concrete reasons:

1. Latency. RAG over ICD-11 needs 3 steps (embed, search, LLM). A fine-tuned model answers in one. Difference: 900 ms vs 180 ms per query.
2. Cost at scale. An insurer processes 50,000 diagnoses/day. At $0.002 per query with Claude Haiku = $3,000/month; with our own model on vLLM on an A10 (~$150/month): 20× cheaper.
3. Compliance. Clinical data cannot leave proprietary infrastructure for many Mexican, Colombian and Chilean insurers and hospitals.

The real optimum isn't "fine-tuning OR RAG" — it's fine-tuning with complementary RAG for edge cases.

Training architecture

  ICD-11 catalog (WHO, Spanish, 72k codes)
       │
       ▼
  Dataset pipeline:
    • SNOMED CT clinical synonyms translated
    • Colloquial variants (ADA Corpus, MedPal)
    • Mexican terms ("empacho", "susto", etc.)
    • 350,000 (description → code) pairs
       │
       ▼
  Splits: 320k train · 15k val · 15k test
       │
       ▼
  ┌─────────────────────────────────┐
  │ Unsloth QLoRA                   │
  │  base: Llama-3.3-8B-Instruct    │
  │  quantization: 4-bit (nf4)      │
  │  LoRA rank: 64                  │
  │  alpha: 16                      │
  │  lr: 2e-4, cosine               │
  │  batch: 16, grad accum: 4       │
  │  epochs: 3                      │
  │  GPU: 2× A100 80GB, 6h          │
  └─────────────────────────────────┘
       │
       ▼
  Model + adapters → merge → push to HF

Dataset preparation

The dataset is the most important piece — 90% of the effort, 90% of the final quality.

Sources

1. ICD-11 (WHO) — official catalog, XML. 72,032 entities with code, preferred name, synonyms, inclusions and exclusions.
2. SNOMED CT — mapping to ICD-11 with alternate descriptions; Spanish translation with clinical glossaries.
3. Mexican clinical corpus — 40,000 hospital-university clinical notes (with consent and de-identification).
4. Synthetic prompts — Claude Sonnet generates 4 colloquial variants per code ("intense throbbing headache that hits suddenly on one side" → 8A80.0 Migraine without aura).

Format

{"messages": [
  {"role": "system", "content": "You are a medical assistant. Return the most likely ICD-11 code for the given description. Format: code · name."},
  {"role": "user", "content": "34yo woman with throbbing pain in right hemicranium for 4h, nausea, photophobia"},
  {"role": "assistant", "content": "8A80.0 · Migraine without aura"}
]}

Balancing

Classic problem: some codes appear 5,000 times in the corpus, others 3. Without balancing, the model ignores rare ones. Solution: stratified sampling with a cap — maximum 100 examples per code, minimum 8 (synthetic if missing).

De-identification

Mandatory pipeline: Presidio Analyzer + Presidio Anonymizer before entering training. No PII remains in the dataset; it's verifiable for audit.

Training with Unsloth

Unsloth is the framework that lets us train 8B on 2 A100 in 6h. Script train.py:

from unsloth import FastLanguageModel
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset

max_seq_length = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.3-8B-Instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    load_in_4bit=True,
    dtype=None,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=64,
    lora_alpha=16,
    lora_dropout=0.0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    use_gradient_checkpointing="unsloth",
    random_state=42,
)

dataset = load_dataset("json", data_files={
    "train": "data/icd_train.jsonl",
    "val":   "data/icd_val.jsonl",
})

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset["train"],
    eval_dataset=dataset["val"],
    max_seq_length=max_seq_length,
    dataset_num_proc=4,
    args=TrainingArguments(
        output_dir="out",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        num_train_epochs=3,
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        warmup_ratio=0.03,
        logging_steps=25,
        eval_steps=200,
        save_steps=400,
        bf16=True,
        optim="adamw_8bit",
        report_to="wandb",
    ),
)

trainer.train()
model.save_pretrained_merged("out/icd-cie-es-8b", tokenizer, save_method="merged_16bit")

Run cost: ~$60 USD on a Lambda or RunPod A100.

Evaluation

Golden dataset

3,000 cases hand-curated by two doctors in Mexico (Cohen's κ = 0.89). No overlap with training.

Baselines

We compare against:

• claude-opus-4-7 and claude-haiku-4-5 via LiteLLM.
• gpt-4o and gpt-4o-mini.
• Base llama-3.3-8B (no fine-tuning) + Qdrant RAG.
• Our numoru-ia/icd-cie-es-8b.

Metrics

The sweet spot: 91% top-1 with 18× lower cost than Opus, in 1/6 the latency.

What the 9% error means

We broke down the 9% remainder:

• 4% very rare codes (<20 appearances in training).
• 3% ambiguous descriptions (the human doesn't pick top-1 with confidence either).
• 2% genuine model errors.

Complementary Qdrant RAG over the entire catalog covers the 4% rare ones — the combined system reaches 91%.

Serving with vLLM

Production runs vLLM (Apache 2.0) for throughput. Base command:

vllm serve numoru-ia/icd-cie-es-8b \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 1 \
  --enable-prefix-caching

On an A10 (24 GB VRAM): sustained throughput ~120 req/s with p50 latency 180 ms. More than enough for mid-size SMBs.

Docker Compose:

services:
  icd-vllm:
    image: vllm/vllm-openai:v0.6.4
    command: >
      --model numoru-ia/icd-cie-es-8b
      --dtype bfloat16
      --max-model-len 4096
      --enable-prefix-caching
    runtime: nvidia
    environment:
      NVIDIA_VISIBLE_DEVICES: all
    ports: ["8000:8000"]

It plugs directly into LiteLLM as a custom provider: the rest of the stack (Langfuse, etc.) consumes it like Claude.

Complementary RAG

For rare cases, Qdrant has the entire ICD-11 catalog indexed:

def classify(description: str) -> str:
    primary = vllm_client.complete(description, model="numoru-ia/icd-cie-es-8b")
    if primary.confidence >= 0.80:
        return primary.code

    # Fallback RAG
    hits = qdrant.search("icd11_cat", embed(description), limit=10)
    rerank = bge_reranker.rerank(description, hits)
    return rerank[0].code

Confidence is derived from the model's logprobs on the code token.

Governance and compliance

For clinics and insurers the model must withstand audit:

• Model card on Hugging Face with limitations, training data (PII-free metadata), per-subgroup metrics (age, gender).
• Mandatory logging in Langfuse for every inference, with user and timestamp.
• Human-in-the-loop for definitive diagnoses — model output is never taken as truth without medical review.
• Bias audit before release: separate metrics by gender and by age group. If a subgroup drops more than 5 points vs global, retraining with balanced examples.

Total project time and cost

For simple ROI: if the insurer saves $2,500/month in API calls to Claude, payback in <2 months.

Business & commercial impact

Risks and mitigations

1. Catalog obsolescence. WHO publishes updates. Mitigation: re-run fine-tuning every 6 months on the delta.
2. Overfitting to Mexico. If used in Colombia or Chile, terms vary. Solution: multi-dialect dataset from v2.
3. Clinical liability. The model doesn't decide diagnoses; it suggests codes. Liability is the doctor's. Documented in the Terms.
4. Data drift. If clinical notes change style (new terms, new flu variant), metrics drop. Automated monthly evals alert.

Lessons learned

• Unsloth saves real money. 4× faster than HuggingFace TRL standard.
• Dataset rules. An extra hour curating is worth 10 of hyperparameter tuning.
• QLoRA r=64 is the sweet spot for this kind of constrained classification task.
• Publishing on HF with a decent model card brings sustained organic traffic — our repos get downloads every day with no ads.
• vLLM > text-generation-inference for high volumes.

FAQ

Why Llama and not Qwen 2.5?We tried both. Qwen 2.5 7B gives 86% top-1 with the same recipe — comparable. We chose Llama for ecosystem (more tutorials + integrations), but Qwen is a valid option.

Does it work for other codes (CPT, LOINC, SNOMED)?Same recipe; change dataset. We're planning CPT (procedures) for Q3 2026.

Can it be done without dedicated GPU?Inference yes (CPU with quantization, 1-2s latency). Training not practical.

How do I make sure the model doesn't "memorize" real corpus patients?De-identification before training + memorization test (extraction with adversarial prompts) pre-release. If any PII is detected, the version is discarded.

Does it work with informal text ("my belly really hurts")? Yes — the dataset includes colloquial variants. Not as precise as structured clinical description, but top-5 acc still >90%.

Next steps

• Published model: huggingface.co/numoru-ia/icd-cie-es-8b.
• Training + eval code: github.com/numoru-ia/icd-cie-fine-tune.
• Next release (Q3 2026): 14B version with integrated SNOMED CT.
• If your company processes diagnoses and wants the managed version with SLA, Numoru offers "ICD-CIE on-prem with support" — a package that includes fine-tuning over your local data.

Running this model in production needs two more pieces: the self-hosted stack that serves it on a single droplet, and an eval suite in CI/CD so a fine-tune never silently regresses.