The short answer is that you stop when you run out of time. Former world champion Garry Kasparov was said to have 3000 novelties in his repertoire. A novelty is a move that hasn't been played before and that changes the evaluation of a position. Kasparov was a professional player with a phenomenal memory and a team of grandmasters helping him with opening research. He earned his living by beating the best players in the world and every successful novelty meant a fatter paycheck at the end of a tournament.
The casual player with limited time needs to be practical. There are only so many hours in a day, only so many minutes which can be used for chess, and so many interesting things in chess competing for this limited time.
The best way to use your time is to keep your repertoire balanced. Don't go to move 18 in one variation, while the others all stop at move 4. Some variations will naturally be longer than others. There are certain popular lines with a long sequence of forced moves where your work will start at the end of these moves. These should be the exception.
How do you choose which move becomes part of your repertoire? You should have a reference or two that gives the known moves in your favorite lines. Caution! Developing a repertoire does not mean memorizing these references. Along with chess publications, there is a lot of material available on the Web. Check the 'Game downloads' link on the left sidebar for some useful sites.
