Yeah, I read a study a year or two ago that not only do people assume that it's terrible for children to be left alone, the amount of perceived terrible increases based on moral judgements about the activities (and gender) of the person responsible for caring for the children in question.

http://collabra.org/articles/10.1525/collabra.33/
http://www.npr.org/sections/13.7/2016/08/22/490847797/why-do-we-judge-parents-for-putting-kids-at-perceived-but-unreal-risk
(Warning -- that study is really depressing.)