This study challenges the consensus view that children can judge what someone is looking at from infancy. In the first experiment 2-, 3-, and 4-year-old children were asked to judge what a person in a drawing was looking at and which of two people was "looking at" them. Only 6% of 2-year-olds and young 3-year-olds passed both gaze-direction tasks, but over 70% passed an analogous point-direction task. Most older 3-year-olds and 4-year-olds passed all three tasks. Experiment 2 compared children's ability to judge what the experimenter was looking at with performance on the picture tasks. Three-year-olds performed significantly worse than 4-year-olds on the real life and picture gaze tasks. Performances on the two types of gaze task were highly correlated. Experiment 3 included stimuli with the additional cue of head-direction. Even the younger children performed well on these stimuli. These results suggest that, regardless of task format, children cannot judge what someone is looking at from eye-direction alone until the age of 3 years. Weaknesses in the evidence supporting the consensus view are highlighted and discussed.