I can completely relate to you and I know exactly where you're coming from. Whenever I took a day off the gym I thought that fat would accumulate and undo my previous weeks work. Wouldn't life be grand if our bodies didn't require rest? We could exercise all day and all night long, reaching our goals practically in a matter of days! Well, our bodies don't operate like that unfortunately.
Every time you exercise to any significant degree, you will put a strain on your system. The harder you exercise, the greater the strain. A very common theme on this website is that you need to exercise intensely enough as to put enough strain on your system to allow your body to adapt. It will adapt by becoming more efficient, for example becoming fitter or stronger. An example here is a typical natural bodybuilder - if you are going to build up a muscle group, you need to allow that muscle to recover before hitting it again.
Let's say you perform cardiovascular exercise over a week and place a significant strain on your system. But rather than taking a day off to allow your body to recuperate, you continue to strain your system. This will ultimately work against you - your body can no longer adapt to the physical demands, it is now simply trying to return to a satisfactory state of health. This is when you over-train - you become sick, your hormones jump all over the place, you can't sleep correctly and you become injury prone.
I'll be the first to say that even by knowing all of this information, this still doesn't stop those mind games inside your head when you are sitting down relaxing and recovering on your rest day. What you need is a plan of attack. Something that sets out exactly what you need to do in order to reach your goals. For example:
If you write that down for the week, then it's there in black and white. You cannot change your mind or 'feel' like doing something else. Generally speaking, if it's not on the plan then you don't do it - simple.
Honestly a similar approach to this got me through my 12 weeks. I had everything written down. By doing this, there was no subjectivity left to my training - it was straight forward and I just did what had to be done. Rest HAS to be done - there is no question about it. It is just as important as your training and your diet.