If you're looking for actual physical damage, go ask a doctor, I think neurosurgeon. I don't think so, though. The liver rations out blood glucose accordingly and the brain is a relative priority. Brain damage from low blood glucose usually occurs pretty abruptly when the body just doesn't have any more to give and you'd already be hospitalized for it a la Terry Schiavo. Physically underdeveloped, on the other hand... possibly but I'm just going by overall physical development wherein stature really is hampered by a poor diet. Anyway, I suggest you just go out there and learn. You probably haven't maximized what you have now. When you think you've hit a wall and suspect it was because you fasted, then, consult a doctor.
I've fainted once when I was at my peak at eating disorder, when my weight was at its lowest, 100 lbs. That time, I did a game marathon straight for 2-3 days without sleeping. I only ate once and drank only a 500 bottle of water in those days. When I was walking home, I fell for the first time in my life. Did it have something to do with the blood glucose thing you were talking about? I've also read if the blood flow to the brain is interrupted even only for a brief of time, you'll be having a brain damage.