To improve Kingdom security, choose Quests that reward multiple Shields (the security icon) rather than Hearts (well-being), Heads (knowledge) or Lyres (culture). When you select a quest, you can mouse over the reward icon, to see the improvement you earn for each level - bronze, silver, gold and platinum. It is worthwhile to strive to complete each quest at Platinum. The best quests will raise your Kingdom's security by 4, at platinum-level victory.
Unless you raise Kingdom security with quests, your buildings only increase potential. And as long as you have "low" security, you will have bandits and burglars, no matter how many you duel to the death or send to the pit. Raising Kingdom security is the only solution.