C-arm fluoroscopy is ubiquitous in contemporary surgery, but it lacks the ability to accurately reconstruct three-dimensional information, attributable to the difficulty in obtaining the pose of X-ray images in 3D space. We propose a unified mathematical framework to address the issues of intra-operative pose estimation, correspondence and reconstruction, using simple elliptic curves. In contrast to other fiducial-based tracking methods, our method uses a single ellipse to constrain 5 out of 6 degrees of freedom of C-arm pose, along with randomly distributed unknown points in the imaging volume (either naturally present or induced by randomly placed beads or other markers in the image space) from two images/views to completely recover the C-arm pose. Preliminary phantom experiments indicate an average C-arm tracking accuracy of 0.51° and 0.12° STD. The method appears to be sufficiently accurate and appealing for many clinical applications, since it uses a simple elliptic fiducial coupled with patient information and has very minimal interference with the workspace.