Does his belly feel hard or bloated? You mentioned already that he doesn't seem to be in pain.
Looking at
Dr. Greene's website for the age of your baby and the fact that you're breastfeeding, he says the time to be worried is if it's been a week with no stool...or sooner if your baby appears to be in pain without one. Going only once a week is normal as long as the stool is soft.