I agree that the flogging and banning would still be applicable, and that's why 24:5 says "من بعد ذلك". My reading of the exception in 24:5 is that it refers to "أولئك هم الفاسقون" not to the punishment. JMHO.
Ibn Abbaas, Judge Shurayh ibn Al-Haarith, Al-Hasan, Mugheera ibn Muqsim Adh-Dhabbi, Ibrahim An-Nakh`i, Muhammad ibn Katheer, Hammaad ibn Salama, Sa`eed ibn Al-Musayyib, Al-Haytham ibn Jameel, Shareek ibn Abdillah, Saalim ibn `Ajlaan Al-Aftas and Sa`eed ibn Jabeer all agree with you! Al-Qaasim ibn Salaam, in his book الناسخ والمنسوخ في القرآن والسنة, page 124, quotes them saying that 24:5 provided an exception to 24:4 in that if a false accuser repents, he is no longer labeled "deviant", but his testimony remains disallowed.
But he also quotes other narrations where Ibn Abbaas, Az-Zuhri, Al-Qaasim ibn Muhammad, Saalim ibn Abdillah, Ibrahim An-Nakh`i, Abdillah ibn `Utba Al-Huzhali, Yahya ibn Bakeer, Abdillah ibn Yasaar (Ibn Abi-Nujayh), `Ataa', Taawoos, Mujaahid and Maalik all said that repentance of the false accuser re-allows his testimony. Ibn Salaam agrees.
There is, however, a conflation of the accuser with his collaborating witnesses. Ibn Salaam's agreement is based on a narration by Sa`eed ibn Al-Musaayib that Umar ibn Al-Khattaab, may God have been pleased with him, offered Abu-Bakra, a false accuser to repent and recant his accusation. He refused and Umar never again accepted his testimony. Umar also offered repentance to the four witnesses and two did and Umar allowed their testimony afterward.
I think that 24:4 applies to the accuser only, because if two witnesses out of four recanted, then the accuser no longer has four witnesses! Hence, the conditional in 24:4 applies. As to what to do with witnesses who recant, Ibn Salaam said that the majority have said that their subsequent testimonies are allowed. Their argument is that a repenting adulterer's testimony is acceptable. I think that there is a confusion between whether God accepts repentance from sin, however big it may have been, and whether testimony of a repenting sinner should be acceptable. I may be harsh here, but I don't see how the testimony of someone who once deliberately lied under oath can ever be accepted again. If he's done it once before, he may do it again. As lawyers would say, "Goes to credibility, your honor."