That's one of the things that happened to you while you lived there. The castle took something important from you. Maybe it was your ability to sing, or a memory of somepony special to you, or a really important item you never, ever would be without. It was called a "loss". Kinda a price for being kidnapped there.

Mine was the ability to travel alone.