What would happen if I just left in the middle of the night?
He wouldn't remember you when he got older.
A price would need to be paid, but I don't know about him completely forgetting.
Personally, I think you should go.

He loped into the night, thinking and rubbing his too soft hands face, never quite sure if he had been slapped in the face.