Holins are generally believed to generate large membrane lesions that permit the passage of endolysins across the cytoplasmic membrane of prokaryotes, ultimately resulting in cell wall degradation and cell lysis. However, there are more and more examples known for non-lytic holin-dependent secretion of proteins by bacteria, indicating that holins somehow can transport proteins without causing large membrane lesions. Phage-derived holins can be used for a non-lytic endolysin translocation to permeabilize the cell wall for the passage of secreted proteins. In addition, clostridia, which do not possess the Tat pathway for transport of folded proteins, most likely employ non-lytic holin-mediated transport also for secretion of toxins and bacteriocins that are incompatible with the general Sec pathway. The mechanism for non-lytic holin-mediated transport is unknown, but the recent finding that the small holin TpeE mediates a non-lytic toxin secretion in Clostridium perfringens opened new perspectives. TpeE contains only one short transmembrane helix that is followed by an amphipathic helix, which is reminiscent of TatA, the membrane-permeabilizing component of the Tat translocon for folded proteins. Here we review the known cases of non-lytic holin-mediated transport and then focus on the structural and functional comparison of TatA and TpeE, resulting in a mechanistic model for holin-mediated transport. This model is strongly supported by a so far not recognized naturally occurring holin-endolysin fusion protein.