According to the penal substitution theory of the atonement, Christ's sufferings satisfy justice in place of our being punished. That is, basically, the theory as found in Anselm's Cur Deus Homo.
Some contemporary Christians, mainly Protestant, add the claim that Christ was punished by the Father, and his punishment substitutes for our punishment. We can call the resulting theory punishment by punishment substitution (PBPS). PBPS isn't Anselm's theory, and as Mark Murphy has pointed out it may even be incoherent, since a part of punishing is the showing of disapproval at the person being punished, while God cannot show disapproval at an innocent person.
The Heidelberg Catechism explicitly only says that Christ satisfies for us. But it says in the answer to Question 14 that no mere creature can satisfy for us because "God will not punish any other creature for the sin which man has committed", which may implicate that satisfaction involves being punished. Still, it does not say that it does so in the case of Christ.
In any case, it seems to me that the biblical theory is not that the punishment of Christ substitutes for our punishment, but that the sacrifice of Christ substitutes for our punishment. Old Testament sacrifices for our sins were not punishments of the animals, except in the extended sense of the word as when we speak of "the punishing heat of Texas summer." It is central to the idea of sacrifice in the Old Testament that it is the best that is sacrificed. To sacrifice something is to treat it as the best that is available. But when someone is being punished, then he is far from being treated as the best—he is being treated as one of the worst. Thus, the biblical picture of Christ as sacrificed is in serious tension with PBPS.
That the sacrifice of Christ substitutes for our punishment isn't yet a theory of the atonement. To make it a theory of the atonement one would have to say how it does so.