Your teachers statements are absolutely correct.
All of us learn at different rates, your pace is not really an issue for the other students. For instance, they take hi falls.... you do rolls, till the hi-falls feel right.
The sensei seems to be aware of your reservations, and will/should work to make the learning process as smooth as possible for all of the class. You too.
As long as the class isn't in a "holding pattern" while you are working on a technique for an innordinant amount of time, then they should be supportive of you learning. ( My class has been very patient as I struggled with the basics.Hell! I am still struggling with them.)
Just do it if you really want to.