A mineral coating develops on the filter grain surface when groundwater is treated via rapid sand filtration in drinking water production. The coating changes the physical and chemical properties of the filter material, but little is known about its effect on the activity, colonization, diversity, and abundance of microbiota. This study reveals that a mineral coating can positively affect the colonization and activity of microbial communities in rapid sand filters. To understand this effect, we investigated the abundance, spatial distribution, colonization, and diversity of all and of nitrifying prokaryotes in filter material with various degrees of mineral coating. We also examined the physical and chemical characteristics of the mineral coating. The amount of mineral coating correlated positively with the internal porosity, the packed bulk density, and the biologically available surface area of the filter material. The volumetric NH4+ removal rate also increased with the degree of mineral coating. Consistently, bacterial 16S rRNA and amoA abundances positively correlated with increased mineral coating levels. Microbial colonization could be visualized mainly within the outer periphery (60.6 ± 35.6 μm) of the mineral coating, which had a thickness of up to 600 ± 51 μm. Environmental scanning electron microscopic (E-SEM) observations suggested an extracellular polymeric substance-rich matrix and submicron-sized bacterial cells. Nitrifier diversity profiles were similar irrespective of the degree of mineral coating, as indicated by pyrosequencing analysis. Overall, our results demonstrate that mineral coating positively affects microbial colonization and activity in rapid sand filters, most likely due to increased volumetric cell abundances facilitated by the large surface area of internal mineral porosity accessible for microbial colonization.