medium10 pts · first answer countsfine-tuningtokenization
Hindi and Arabic use more tokens
The same message costs about 2.5× more tokens in Hindi than in English with your model's tokenizer. How does this affect fine-tuning?
Discussion 0
Join free to post in the discussion.
Related questions
- Why LLMs use subword tokensMachine Learning Engineer
- What full fine-tuning changesMachine Learning Engineer
- Why begin from a pretrained modelMachine Learning Engineer
- Training only a new headMachine Learning Engineer
- Learning rate for fine-tuningMachine Learning Engineer
- Shape of SFT training dataMachine Learning Engineer
- Reading train and validation lossMachine Learning Engineer
- Meaning of PEFTMachine Learning Engineer