The spectrum of terahertz (THz) emission in gases via ionizing two-color femtosecond pulses is analyzed by means of a semi-analytic model and numerical simulations in 1D, 2D and 3D geometries taking into account propagation effects of both pump and THz fields. We show that produced THz signals interact with free electron trajectories and thus significantly influence further THz generation upon propagation, i.e., make the process inherently nonlocal. This self-action contributes to the observed strong spectral broadening of the generated THz field. Weshow that diffraction of the generated THz radiation is the limiting factor for the co-propagating low frequency amplitudes and thus for the self-action mechanism in 2D and 3D geometries.