A common technique to fabricate silicon nanowires is to use metal particles (e.g., Au, Ag, Cu, Al) to catalyze the growth reaction. As a consequence, the fabricated nanowires contain small concentrations of these metals as impurities. In this work we investigate the effect of the metallic impurities on the electronic transport properties of silicon nanowires. The computational method is based on ab initio density functional theory together with nonequilibrium Green's functions. From the computed transmission functions we extract a scattering cross section to characterize the scattering strength of the different metal atoms. We find that Au, Ag, and Cu impurities have very similar scattering cross sections, while Al differs from the rest. Impurities located in the center of the wires scatter significantly more than impurities close to or at the surface. The results for nanowires are compared with bulk Si scattering calculations and good agreement is found. This agreement shows that the scattering results for the ultrathin nanowires (which are computationally feasible) are not dominated by finite size or surface effects, and indicate that the results can be extended to larger and experimentally more relevant wires.