hard20 pts · first answer countsfine-tuningrlhf

Policy writes nonsense that scores well

In a PPO run with a very small KL coefficient, the policy starts producing repeated phrases like "Great question! Great question!" and gets top reward scores. What is the best next step?

A.Lower the KL coefficient furtherB.Increase the learning rate to escapeC.Remove the reward model and train on prompts onlyD.Raise the KL coefficient and patch the reward model

Discussion 0

Join free to post in the discussion.

Policy writes nonsense that scores well

Discussion 0

Related questions