A trebuchet primarily transforms downward motion (of the counterweight) into forward motion, so it would actually work - the trebuchet doesn’t push the ship back as much as it pushes its load forward. This is particularly so if your trebuchet has wheels and you have room on your ship to accommodate it rocking back and forth when firing.
Edit: Thinking about it this technique would work even with something that does impart equal backwards force on the ship when firing a projectile, because there is considerably more force involved in winching the ship towards the anchor than what is involved in actually moving the anchor. You aren’t pulling against the inertia of a free floating anchor after all, you’re pulling against the ground the anchor has hooked into.
Indeed, I just realised that point - the force of propelling the anchor is tiny compared to what you can exert on the ship once the anchor is hooked.