I present the evolution of the stellar mass function (SMF) of galaxies from z=4.0 to z=1.3 measured from a sample constructed from the deep NIR MUSYC, the FIRES, and the GOODS-CDFS surveys, all having very high-quality optical to mid-infrared data. I provide, for the first time, a comprehensive analysis of random and systematic uncertainties affecting the derived SMFs. I find that the mass density evolves by a factor of ~17 since z=4.0, and a factor of ~4 from z=1.3. The evolution appears to be mostly driven by a change in the normalization Phi*, but I also find evidence for evolution in the shape of the SMF, with the low-mass end evolving more rapidly than the high-mass end. Taking these results at face value, I find that they are in conflict with the predictions of semi-analytic models of galaxy formation. I will conclude by presenting preliminary results on the evolution of the massive end of the SMF from z=4 from the NEWFIRM Medium-Band Survey, a large NOAO/Yale program which uses medium band-width filters in the near-IR to obtained well-sampled spectral energy distributions and high quality photometric redshifts over 1 square degree, returning redshifts accurate to 0.02 in Delta z / (1+z) over the redshift range 1.5