Well, I decided that, since my Sim could not marry his grandniece, he would marry her niece. It turned out he could not have any romantic interactions with his great-grandniece either, even though the game did not recognise him as family of either Sims. Therefore, my conclusion is that Sims cannot have romantic interactions with any descendant of their sibling/any sibling of their ancestor.