Does gravely immoral activity by the clergy detract from the holiness of the Church as it is on earth, the Church militant? It sure seems so. But I want to argue, very speculatively (and I will withdraw the claim if it turns out that the Church teaches otherwise), that it directly detracts no more—or at least not significantly more—than equally gravely immoral activity by similar numbers of laity would.
Consider the three features we desire the Church as found on earth to have: doctrinal orthodoxy, liturgical integrity and holiness of life.
The doctrinal orthodoxy and liturgical integrity of the clergy does indeed especially contribute to the doctrinal orthodoxy and liturgical integrity of the Church. If a priest or especially a bishop is unorthodox, all other things being equal, that in itself detracts more from the orthodoxy of the Church on earth than when a lay person is unorthodox, simply because of the teaching role of the clergy. Similarly, if a priest or bishop engages in liturgical anarchy, say by changing some prayers at Mass, that detracts from the liturgical integrity of the Church more than if a lay person does so, all other things being equal.
But when a deacon, priest or bishop (including a pope) is wicked, that no more (and no less) directly detracts from the holiness of the Church than when a non-cleric person is wicked, when the degree of wickedness is the same. We can see this by considering the happier flip-side. Think of a non-cleric like St Teresa of Avila (she was a nun, of course, but a nun is a non-cleric[note 1]) and a priest like St John of the Cross. The holiness of their lives directly contributed to the holiness of the Church. But it would, I think, be mistaken to say that St John's holiness contributed more, or was a more central contribution, than St Teresa's just because St John was a priest and St Teresa was not. To say that would be to engage in clericalism, and is perhaps a species of the same error that leads to Donatism. The clergy's activity makes a special constitutive contribution to the Church's orthodoxy and liturgical integrity. But a layperson's holiness is just as constitutive of the holiness of the Church as the holiness of a deacon, priest or bishop. Mary makes a greater direct contribution to the holiness of the Church than any deacon, priest or bishop—not counting Christ the High Priest—ever did or would.
Of course, wickedness in a deacon, priest or bishop (and especially when the bishop is pope) typically has a greater negative effect on the Church's holiness, because it is more likely to scandalize others, leading them either to imitate the wickedness or abandon the faith. This indirectly detracts from the Church's holiness.
Suppose every single Catholic priest next year committed some particular grave and scandalous sin. That would be a terrible thing, would have a very unfortunate negative effect on the Church, and may God preserve us from this disaster. But it would no more directly detract from the Church's holiness than if some other group comprising 0.04% of the world's Catholic population committed an equally grave sin.
That said, a sin that is otherwise of the same sort may be graver when committed by a cleric, because (a) the cleric bears a responsibiity for avoiding the further negative effects and (b) is less likely to be excusable through ignorance. The above assumes we are dealing with sins of equal gravity.
Christ promised that the Church would be holy. The above argument shows that an argument based on clerical crimes that the Catholic Church cannot be Christ's Church because Christ's Church is holy is no stronger than argument based on equal numbers of non-clerical crimes would be. But an argument based on the crimes committed by non-clerics would fail: we do not expect the Church's holiness to imply the sinlessness of her members. The Church while holy as a body—the body of Christ—is yet a Church for sinners who need Christ's reforming grace. Thus the argument based on clerical crimes also fails.
And then, of course, there is always Boccaccio's argument.