We should distinguish between two kinds of choices. Intrinsically free choices are free solely in virtue of the internal features of how the choice is made (if externalism about the mind is true, we may need to allow mental facts to count as "internal" for the purposes of distinguishing between chocies). Any exact duplicate of an intrinsically free choice will also be free. Extrinsically free choices are free in part in virtue of something occurring or not occurring outside the choice itself, typically facts about the history of the agent.
Paradigm libertarian-free choices are intrinsically free. If Curley is choosing whether to take a bribe or not, and he is causally and psychologically able to take it and causally and psychologically able not to take it, and he has non-overwhelming motivations in favor of taking the bribe (he desires money) and non-overwhelming motivations in favor of not taking the bribe (he desires to avoid moral degradation and he desires to avoid jail), then it does not matter whether he was brainwashed into having a desire for money and a desire to avoid moral degradation. His choice is intrinsically free. Note that, interestingly, Frankfurt examples trade on our intuition that at least some of our choices are intrinsically free—that's why the neurosurgeon's standing by does not affect the choice's freedom.
Libertarians can, of course, admit that there are extrinsically free choices. Thus, if my earlier free decisions have formed, through the right kinds of causal chains, a character that is unable to give my son a scorpion when he asks for bread, I may still act freely when I refrain from giving my son a scorpion when he asks for bread. However, this is an extrinsic freedom. For if I were brainwashed into having this sort of character, I would not be acting freely.
Compatibilists are committed, I think, to the claim that it is possible for all of one's free choices to be merely extrinsically free.
We can then argue for incompatibilism by arguing that the freedom of an extrinsically free choice always depends on one's being antecedently (to the choice) responsible for something outside the choice itself, and that anything one is responsible for depends on a free (intrinsically or extrinsically so) choice or on something else one is responsible for. For then, if all choices are extrinsically free, an infinite regress will be generated.
One might make a similar distinction about responsibility, and then try to argue that intrinsically free choices are the only thing one can be intrinsically responsible for. If one could do that, and then argue that anything one is extrinsically responsible for one is responsible for in part because one is responsible for something else, again a regress results if there is nothing that we are intrinsically responsible for.