So.. considering that I feel like your post is in fact stating my own views and training strategy-- do you think my video lines up (in a beginner-level way) with my words (ie, with your words)?
You are certainly on the right track...
You need to move your arms less and let your body do more. To do that you need to "go deeper" with the touch to his center. Ikeda Sensei will say "touch their spine". Actually, I like to set up cross body connections. So if he grabs with his right hand, I'll go in and connect to something on the left side of him, rather than simply just dead center.
Once you have that connection, the ideal is to experience NO CHANGE in pressure at the point of contact as you use your body to redirect the energy of that connection. If the pressure increases, in this case on his hand or arm, you are putting your attention on the point of contact and that's where your energy is going. Raso Hultgren Sensei would say that you need to put your energy "through" him rather than "at" him.
Look to maintain a perfect balance between what you are putting out to him and what you are bringing in to you. Usually, you bring his energy into your spine before you give it back to him. This would be tricky to explain rather than show, at least for me. You should feel someone do it. There's a balance of extension and draw between the fingers and the elbow along the forearm line. There's a balance between the elbow and the shoulder along the upper arm line. There's a balance of energy running up your spine and down out your tail bone. Lots of oppositional balances. More than I am currently aware of... I keep being exposed to more and more subtle lines of balance, cross body connections and tensions, etc. But you don't have to be super sophisticated to do the basic work.
Just remember, the arms do not create power. They can be used to connect with the partner's center, they help give direction to the energy of the connection, but they don't create power. At the beginning you can play with using a bit of hip movement to get horizontal movement and a bit of pelvic roll forward and back to get an up and down. Later there are ways to accomplish exectly the same thing internally and one won't see much movement outside at all. But it will work simply by letting you upper body sit on the pelvis and letting the pelvis sit on top of the hip joints and float there. Any movement back there will reflect instantly in the partner if one is already connected to that partner's center.
Basically, don't think too much about what you are doing since focusing on just yourself tends to put all your attention and energy on you and often you lose the outflow to the partner. Don't be afraid to look right at him and observe every movement he or she makes as you experiment with what you moved ion your side to produce that movement.
On the other hand, if no movement is occurring, you may be going out too much and running into their structure. In which case look at how you can correct your posture and re-balance all those forces we talked about. I hope this helps. It's easy to show in a hands on setting but not so easy to explain because s much is about "feel".