No. You can take them off by taking off the dust cover and the spindle nut. Then the whole drum/bearing assembly should come off in 1 peice. If you need to replace the drum or remove it from the hub, then the rivets do need to me removed. They will also require removal if you plan to intall aftermarket wheels, as the will interfere with the wheel properly seating to the hub.

Bill Burmeister