If you look at the interaction between form 8606 and 5329, it appears you only have to pay the penalty on rollover contributions *that are taxable*. Since you've already paid taxes on the rollover (say, 1 year ago), your basis is equal to the contribution, so you won't pay tax again. Note that the rollover contribution would be taxable if it was withdrawn the same year (and you had therefore not paid taxes)... the penalty would apply in this case. I'm not quite sure why they chose 5 years instead of 1.
Hmmm...I wasn't clear on this myself, so I tried to find out. Still not completely sure, but I think you have to pay a 10% penalty on withdrawals from a rollover account within 5 years. This looks like the proper IRS example to look at:
Example. In April of 2010, John, age 50, rolls over $20,000 from his traditional IRA to a Roth IRA. This is John's first contribution or rollover to a Roth IRA. There is no basis* in the $20,000 rollover. In August of 2010, John withdraws $10,000 from the Roth IRA. When John completes Form 8606 for 2010, he enters $20,000 on lines 16 and 18 of Form 8606, and $10,000 on line 20a and $10,000 on line 20b. Since the $10,000 distribution John received from the Roth IRA is nonqualified, he completes Part IV of Form 8606 and has a taxable distribution of $10,000 on line 36.
This $10,000 nonqualified distribution is an early distribution subject to the 10% additional tax. When John figures the amount to enter on Form 5329, line 1, he will enter $10,000, the amount from Form 8606, line 36. John does not include on line 1, the $10,000 allocated to Form 8606, line 18.
If you receive a distribution that is not a qualified distribution, you may have to pay the 10% additional tax on early distributions as explained in the following paragraphs.
* Wait, what the basis of a traditional IRA account anyway? Oh, here it is:
Basis. Your basis in traditional IRAs is the total of all your nondeductible contributions and nontaxable amounts included in rollovers made to traditional IRAs minus the total of all your nontaxable distributions, adjusted if necessary (see the instructions for line 2 on page 5).
Hope that helps.