I just got up from conversation with a couple of older black men, that I said “well I got to go back to work and start cracking the whip.” And it occurred to me then that it was probably a really insensitive stupid thing to say.
Sadly, it hadn’t occurred to me until it’s already said.
This is wrong. What OP said was very much offensive, comparing yourself to that of a slaver. It is a rare figure of speech, and while not intentional it is still offensive. However, the context of who you’re talking to doesn’t matter, it is an offensive statement regardless of the history of the person you’re saying it to.
For OP: it is okay to make a mistake, especially when you recognize it.