When the game pushes inactives to go to community lots, all of them count. So if you have a lot of lounges, hangouts, etc. then that is your bakery's competition for visitors. To increase the visitors to my bakery, I ended up removing a lot of the lounges and dives and kept just one that I really liked. Then I started seeing a lot of visitors. Also, and I don't know why, but sales seem to be better when the shelves are only maybe half full.