The far-field patterns of atoms diffracted from a classical light field or from a quantum one in a photon-number state are identical. On the other hand, diffraction from a field in a coherent state, which shares many properties with classical light, displays a completely different behavior. We show that in contrast to the diffraction patterns, the interference signal of an atom interferometer with light-pulse beam splitters and mirrors in intense coherent states does approach the limit of classical fields. However, low photon numbers reveal the granular structure of light, leading to a reduced visibility since welcher-Weg (which-way) information is encoded into the field. We discuss this effect for a single photon-number state as well as a superposition of two such states.