Hard physical labor? The Hunter - Gatherers were referred to as the original affluent society! They had much more free time than we do today. Not to mention that "hard physical labor" kept them in awesome shape. Looking at the bone density and attachment points on HG bones show they were stronger, fitter, and healthier than us. Yes, life expectancy was around 40, but that is an average. Meaning many more people died due to injury and disease b/c of lack of modern medicine. That does not mean everyone died at 40. Not to mention, post-agricultural revolution, the life expectancy dropped! What does that tell you? Looking at life expectancy in the middle ages, it was 18!! I'd much rather take my chanced as an HG any day of the week.
Seriously, check out The Primal Blueprint by Mark Sisson or read The Paleo Solution by Robb Wolf. Both excellent books that are highly researched and interesting reads.
Short term, long term, diet, lifestyle, doesn't matter. The hunter-gatherers on whom the so-called "paleo" diet is ostensibly modeled lived in conditions of food scarcity and had to do hard physical labor for what they could get. If you hope to mimic what they ate (which you really can't, but never mind that), based on the idea that it was somehow healthy for them, you also need to eat the same quantities and live the same kind of life. And then die at 40.