"Figuratively"

Consider this great T-shirt slogan (I have no financial ties to the seller, but if you click on it you can buy the shirt with it).

Everyone I've talked to agrees that statements like the one on the T-shirt are an example of literal language.  The wearer is claiming to literally be made figuratively insane.

But here is an oddity. If you say: "Misuse of 'literally' makes me insane", I can say: "Figuratively speaking, that is." My use of "figuratively" attributes figurativeness to your sentence, which sentence is figurative. But in the slogan on the T-shirt, what does "figuratively" attribute figurativeness to? Presumably, the word "insane"? So does the sentence, thus, contain figurative language after all? But the sentence seemed like a piece of literal language. The "insane" is only there in the scope of "figuratively". So does the "figuratively", perhaps, implicitly attribute figurativeness to a different sentence that hasn't actually been uttered, namely the sentence "Misuse of 'literally' makes me insane". Or, more precisely, maybe it attributes figurativeness to the word "insane" as found in that unsaid sentential context? If so, then analyzing actual sentence tokens requires thinking about sentence types or nonactual sentence tokens.