Common genetic variants have been identified for adult height, but not much is known about the genetics of skeletal growth in early life. To identify common genetic variants that influence fetal skeletal growth, we meta-analyzed 22 genome-wide association studies (Stage 1; N = 28 459). We identified seven independent top single nucleotide polymorphisms (SNPs) (P <1 × 10−6) for birth length, of which three were novel and four were in or near loci known to be associated with adult height (LCORL, PTCH1, GPR126 and HMGA2). The three novel SNPs were followed-up in nine replication studies (Stage 2; N = 11 995), with rs905938 in DC-STAMP domain containing 2 (DCST2) genome-wide significantly associated with birth length in a joint analysis (Stages 1 + 2; β = 0.046, SE = 0.008, P = 2.46 × 10−8, explained variance = 0.05%). Rs905938 was also associated with infant length (N = 28 228; P = 5.54 × 10−4) and adult height (N = 127 513; P = 1.45 × 10−5). DCST2 is a DC-STAMP-like protein family member and DC-STAMP is an osteoclast cell-fusion regulator. Polygenic scores based on 180 SNPs previously associated with human adult stature explained 0.13% of variance in birth length. The same SNPs explained 2.95% of the variance of infant length. Of the 180 known adult height loci, 11 were genome-wide significantly associated with infant length (SF3B4, LCORL, SPAG17, C6orf173, PTCH1, GDF5, ZNFX1, HHIP, ACAN, HLA locus and HMGA2). This study highlights that common variation in DCST2 influences variation in early growth and adult height.