i’ve been told this is a bad idea to train for punches. because it pulls your fist down instead of back
You can do it with light weights depending on your size. Usually 5-8lbs and 8lbs being for big guys, I fight either at 135-145lbs and my trainer only let me use 5’s because of exactly what you stated
Light weights are totally fine.