Love contains three aspects: appreciation, benevolence and a striving for union. Suppose that we are supposed to love all human beings. In a secular context, each of the three aspects of love threatens to make the love in many cases rather anemic. It is only in a context like that of Judaeo-Christian theology that one can have a love that is both rich and universal. For let's consider the aspects severally.
Appreciation: Unless we have some picture of the human being as in the image and likeness of God, it is difficult to see that much to appreciate in a Mengele. This can perhaps be overcome if one has a robust enough notion of human nature, though perhaps that is just bringing in the image and likeness of God in a hidden way.
Benevolence: It is possible to will the good to all, as this involves a merely dispositional property. But a merely dispositional beneficence is an anemic sort of benevolence. In a religious context, however, the benevolence can act as a genuine beneficence through prayer and something like the communion of the saints.
Unitiveness: While appreciation and benevolence by themselves imply a kind of union, love's striving for union goes beyond these. But in a secular context one can't really go much beyond these in many cases. First, there is the problem of those who appear completely morally corrupt, with whom a further union would be morally problematic. In a religious context, however, the striving for union connects with eschatology. Every individual human on earth is someone with whom we can strive for eternal union in heaven (even if we believe that we won't achieve this union in every case). Second, and even more seriously, there is the problem of the billions of people with whom we simply cannot have a deeper union, because life is too short and their lives do not intersect our lives enough (for a more radical case, one might cite people in the distant past or distant future!) Again, this is overcome in a religious context, often by a potential for liturgical union—in liturgy, we are importantly united with people all over the world participating in the same liturgy—and always by a striving for a union in heaven that is prefigured by the liturgical union.
An interesting question is how the unitiveness will be realized in heaven between those who are in heaven and those who are in hell. I think here there is a kind of liturgical union, in that both those who are in heaven and those who are in hell are united in praise of God: those in heaven deliberately and explicitly so, while those in hell praise God by the value of their existence and the divine justice they exemplify. This is probably a hard saying.
If the above is right, then the duty of universal love can only fully come into its own in a religious context.