We present a theoretical analysis of the acoustic radiation force on a single small spherical particle, either a thermoviscous fluid droplet or a thermoelastic solid particle, suspended in a viscous and heat-conducting fluid medium. Within the perturbation assumptions, our analysis places no restrictions on the length scales of the viscous and thermal boundary-layer thicknesses δs and δt relative to the particle radius a, but it assumes the particle to be small in comparison to the acoustic wavelength λ. This is the limit relevant to scattering of ultrasound waves from nanometer- and micrometer-sized particles. For particles of size comparable to or smaller than the boundary layers, the thermoviscous theory leads to profound consequences for the acoustic radiation force. Not only do we predict forces orders of magnitude larger than expected from ideal-fluid theory, but for certain relevant choices of materials, we also find a sign change in the acoustic radiation force on different-sized but otherwise identical particles. These findings lead to the concept of a particle-size-dependent acoustophoretic contrast factor, highly relevant to acoustic separation of microparticles in gases, as well as to handling of nanoparticles in lab-on-a-chip systems.