Not sure which APIs you are using, but in general we don't support Khmer shaping (at least 100%). We introduced Indic shaping in some parts of the Acrobat product in Acrobat XI, but I don't believe Khmer is part of that (though it may get some limited supported).
Your best bet is to use an existing shaper - be it MS Uniscribe, Apple's NSText, open source solutions, etc. - and then use the font/glyph selection and positioning to provide the information for use in the PDF content generation.