This whole "pedestrians must wait for every single direction to stop" problem seems to mainly be the result of traffic engineers trying to avoid having people in crosswalks being struck by drivers making left turns on a green light. Apparently this is one of the most common ways for a pedestrian to be injured in an intersection. I think the solution of having all lights be red when the walk signal is on is kind of excessive, but I'm not sure I have a better solution really.