It’s a cliché answer but if it sounds good to you then you can do what you want. Switching scales is quite normal but it’s usually determined by the underlying chord progression and/or melody of the song.
For instance, think of a i iv v in Em. The chords are Em, Am, Bm. In this progression you could / would probably use the E natural minor scale to solo. Minor progressions like this however will often alter the v chord to the V7, in this case B7.
This means raising the minor third of B minor to a major third, or D to D#. D is the (b)7th degree of E natural minor so we raise it to a natural 7th. The result is E harmonic minor. Therefore with this progression you could use E natural minor of Em and Am, and then switch to E harmonic minor over the B7.
Whether working with scales or modes, the same idea applies.
There are probably hundreds of ways you could think about it. The bottom line is, does it fit the music?
You could look at it another way. Let’s say you are playing over a simple G drone. In this case the sky’s the limit because nothing else is going to clash. You could play around with anything in G and it wouldn’t necessarily be wrong, you’d just do what feels good and sounds right to your ears.
Hope that helps. It’s not a simple question because there are many possible ways to answer it but this is the general line of thinking.
