Let's say that the ex-husband of a divorced woman never went to court to get visitation rights?

If he still wants to see the kids, in a safe environment - say, the kids grandpa and uncle goes with kids to see their dad - wouldn't this be OK? Is there any way that could cause the courts to nullify the custody rights the mom has, or somehow put the kids in a foster care facility because the father saw the kids without going to court first?