How do you use Hugging Face's text-generation pipeline with open-source chat models like Mistral or Llama?

Open-source instruction-tuned models (Mistral-Instruct, Llama-3-Instruct, Qwen, Gemma) follow specific chat templates that structure the conversation into system, user, and assistant turns with special tokens. Using the correct template is critical — wrong formatting produces significantly degraded outputs because the model was fine-tuned to expect this exact structure.

The apply_chat_template tokenizer method and the text-generation pipeline with conversations input both handle template application automatically, provided you use a tokenizer from the same model family.

from transformers import pipeline
import torch

# Load with pipeline (handles chat template internally)
pipe = pipeline(
    'text-generation',
    model='mistralai/Mistral-7B-Instruct-v0.3',
    torch_dtype=torch.bfloat16,
    device_map='auto',
)

messages = [
    {'role': 'system', 'content': 'You are a concise Python expert.'},
    {'role': 'user',   'content': 'Write a one-liner to reverse a string.'},
]

output = pipe(
    messages,
    max_new_tokens=150,
    temperature=0.3,
    do_sample=True,
)
print(output[0]['generated_text'][-1]['content'])  # assistant's reply

# ── Manual apply_chat_template for full control
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B-Instruct')
model     = AutoModelForCausalLM.from_pretrained(
    'meta-llama/Meta-Llama-3-8B-Instruct',
    torch_dtype=torch.bfloat16, device_map='auto'
)

# apply_chat_template inserts model-specific special tokens
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,  # add the prompt prefix before assistant turn
)
print(formatted[:200])  # see the raw formatted string

inputs = tokenizer(formatted, return_tensors='pt').to(model.device)
with torch.no_grad():
    ids = model.generate(**inputs, max_new_tokens=200, do_sample=False)
decoded = tokenizer.decode(ids[0][inputs['input_ids'].shape[1]:],
                            skip_special_tokens=True)
print(decoded)