I also had bad lag in Riverview, not 100% sure what caused that.
I've always had lag in Riverview, but it will come and go. I've also experienced a similar problem with lag in Barnacle Bay, only it's worse (it will eventually lead to the game freezing up completely). Every other town plays beautifully (I haven't tried Union Cove just yet).
One thing that those two towns have in common is they both generate "World Cache" files. I believe the problem is somehow related to them, although it doesn't fix the lag in Riverview if I delete them (like I said, the lag will eventually stop for a while). However, doing so prevents Barnacle Bay from freezing. I am only theorizing here, but to me it seems related.