Generally speaking, it is recommended to limit the amount of sugar that you do have in your diet. This is particularly true for refined sugars such as those found in confectionery and other highly processed foods.
When it comes to fruits, vegetables and dairy, there isn't as much information out there on limiting these foods. The main reason is because these are natural foods and they are naturally in our diet and have been for many, many years. I am personally a firm believer that our diet should be predominately based on natural, whole foods, or foods with minimal processing. Fruits and vegetables fall right into this category, as does dairy.
However, as with any food, it is important to consume everything in moderation. Too much of any one particular food will not be healthy. For example, if you were to consume a few kilos of fresh fruit. I'm sure you'd be running to the toilet and shortly after!
Specifically, when we refer to sugar, the most common form of sugar found in fruit is fructose. When you consider a piece of fruit, there is a relatively small amount of sugar contained within it. For example, an apple weighing approximately 140 g contains just 17.6 g of sugar. When you consider that a piece of confectionery, just as sweet as a whole apple could potentially contain four times this amount of sugar, you can understand why there aren't so many recommendations out there to limit the amount of fruit that you consume. Fruits are quite low in sugars.
The same is true with dairy. 100 ml of milk contains just 4.7 g of sugar in total.
The other benefit of fruit and dairy is that they both generally have a much lower glycaemic index than what confectionery has. So whilst fruit and dairy are much lower in carbohydrates in total, they are also generally absorbed much more slowly over a sustained period of time.
The key to any fruit consumption is always in moderation. If you're having to 3 pieces of fruit a day, that's fine. Also be sure to moderate your amount of dairy consumption.