Harmonic phase (HARP) motion analysis is widely used in the analysis of tagged magnetic resonance images of the heart. HARP motion tracking can yield gross errors, however, when there is a large amount of motion between successive time frames. Methods that use spatial continuity of motion - so-called refinement methods - have previously been reported to reduce these errors. This paper describes a new refinement method based on shortest-path computations. The method uses a graph representation of the image and seeks an optimal tracking order from a specified seed to each point in the image by solving a single source shortest path problem. This minimizes the potential for path dependent solutions which are found in other refinement methods. Experiments on cardiac motion tracking shows that the proposed method can track the whole tissue more robustly and is also computationally efficient.