The human tongue muscles plays an important role in multiple vital human functions. Most tongue regions are extensively interdigitated with two orthogonal muscle fibers. Reconstruction of the tongue muscle fiber orientations can help understand the deformation of each muscle group and its function. High angular resolution diffusion imaging (HARDI), one of the diffusion weighted imaging techniques, has been used to resolve the crossing muscle fibers in the tongue. Most existing fiber reconstruction methods use HARDI data to estimate the fiber orientation distribution function (fODF), from which the distinct fiber orientations can be identified by a peak finding algorithm. The assignment of the primary and second fiber orientations can be inconsistent with neighboring voxels. In this paper, we propose a fiber matching algorithm to refine the display of the fiber orientations, which can be used as a post-processing step for fiber reconstruction. The fiber matching algorithm takes the fiber orientations that are reconstructed by a deep convolutional neural network as input, and computes the similarity between neighboring fibers under different assignments. The optimal assignments are achieved by solving a quadratic unconstrained binary optimization model. The proposed method was shown to greatly improve the fiber assignments on synthetic tongue fiber orientations. Application to post-mortem human tongue indicated that the proposed method can reconstruct the complex muscle fibers of the human tongue and improve the visualization of the fiber orientations.