Thanks Mustawd
I had a quick go and wasn't immediately able to replicate the problem, which is annoying... I was hoping it would be something obvious!

I must have accidentally hidden a really weird trigger hidden somewhere on the map, or buried in a convo. I'll track through the triggers and find it, but it might take a little longer than I first thought.
Basically, once the Druid quest is triggered it should be impossible to climb back over the wall to interact with the bandit camp (once the vines are cut back, the wall should be a permanent barrier); and there shouldn't be any issues with leaving the area at any point. I just had a go and this all seemed to work fine... so I'm clearly missing something. On the case.