Advice on replacing an extractor

[Stewart]

Is it difficult to remove the extractor rivet from the bolt sleeve in order to replace the extractor or will it just come out with a punch and a light tap? It's called a "rivet" but is it actually riveted into the bolt sleeve? I wanted to get a little advice before I get out the hammer...