Do you have any consignments running at the moment? There's a long standing glitch that causes what you describe that happens after consignments have been in the store for a while. After a certain amount of time, the items that haven't sold are supposed to be mailed back to you. What happens is that the counter reaches 0, but the consignments remain in the store and continues to count down as the days pass. This negative number causes confusion in the register, and makes it unusable. The easiest way to fix it is to "Cancel Consignments." This should give all of your items back and allow the register to work.
If you don't have any consignment items in the store, replacing the register or the entire consignment shop sometimes fixes the problem. However, there are times where the consignment system in a certain version of a town is simply broken. There isn't a whole lot that can be done in cases like this as far as I know.