You have just moved into a new house, which could have had cats or other pets that used the bathroom in that room. I would clean that room thoroughly with an enzymatic cleaner to get rid of the smell–even though you can’t smell it, your cat can. Also remove all of the used clothing and was those really well too.
Put a litter box in that room with litter you know he likes. Also, get your cat neutered or spayed. If your cat isn’t neutered, he WILL mark territory because cats (male and female) are extremely territorial. Sounds like he has been traumatized by children, otherwise he would not have a fear of them. He needs to adjust to children by no sudden movements, loud noises, etc. and children can be taught how to be with a cat. No hitting, pulling on ears, tails, etc., shrieking. He shouldn’t be strange around your baby if he is taught that there is nothing to fear from children.
Another thought, is there could be cats outside marking territory and he can smell them. I would try and see if you can keep them away from the house and the dirt by deterrents that make loud noises if they get too close, or lights or sprinklers.
.