zooming and panning across a photo

Within a video I have a sequence of photos. When there is a row of people, I have managed to zoom in to the left most person's head and then pan from left to right across all the heads. The problem I have is when it wants to show the next photo. When it goes to the next photo the key frame (scales and position etc) in the envelope strip which had just been used is carried over so the new photo is misaligned. If I put in another key frame - which resets the parameters before the new photo comes along - then the one I was originally panning swerves around to give its original size and position. Is there a way of having just zoomed and panned to be able to see the next sequence of photos in their normal state?
I find it quite difficult sorting out the scales, centres etc.

Thanks