I do a lot of that type of thing, myself. I think that's one reason that Morihei Ueshiba put so much emphasis on "cultivating the land" at various points in his life.
The problem with that is that the actual work is more important than the body development and it doesn't affect the whole body equally. Anyone working like that tends to work much more on one side of the body than the other and that has a negative aspect. People who work hard become very strong in their youth, but they tend to break down with age because of the negative aspects, when the work being done is more important than the development of the body.
In my opinion.