The announcement of the direct detection of gravitational waves (GW) by the LIGO and Virgo collaboration in February 2016 has removed any uncertainty around the possibility of GW astronomy. It has demonstrated that future detectors with sensitivities ten times greater than the Advanced LIGO detectors would see thousands of events per year. Many proposals for such future interferometric GW detectors assume the use of silicon test masses. Silicon has low mechanical loss at low temperatures, which leads to low displacement noise for a suspended interferometer mirror. In addition to the low mechanical loss, it is a requirement that the test masses have a low optical loss. Measurements at 1550 nm have indicated that material with a low enough bulk absorption is available; however there have been suggestions that this low absorption material has a surface absorption of >100 ppm which could preclude its use in future cryogenic detectors. We show in this paper that this surface loss is not intrinsic but is likely to be a result of particular polishing techniques and can be removed or avoided by the correct polishing procedure. This is an important step towards high gravitational wave detection rates in silicon based instruments.