In the general case, that is correct - impossible. A PDF in general contains only the text and its locations, with no structure information. Some PDF files are "tagged" and will identify things using a rather complex grammar. This makes your task considerably harder, but with less guesswork.
Probably can extract - No. You can turn the chicken into chicken soup, but it is much harder to start with chicken soup and end up with the chicken flying off.
You are making big assumptions, and need to read the PDF reference, especially the graphics chapter.