The disposal trick didn't work for me in the case of toxic chamomile.
I disposed of every chamomile plant I encountered, spent the night at the hermit's area, another half day in the lodging area, then went home for about a week.
Returning to Granite Falls and fishing for 8 hours, I've found that none of the chamomile plants have grown back.
This is my dynasty file. I've played 3-4 other files without this problem. Mysterious....