Despite most people's belief that Castles were humongous impenetrable fortresses full of pain and death and misery, most of the time a castle was the exact opposite.
Castles were the epicentres of trade for the medieval ages and if they were the dark scary places most people think of them then the Knights, Lords and Warlords who owned them would have gone broke.
Back in the age of Castles there was a King, who owned the land, but did not control it indefinitely - in a matter of speaking. There were those who owned castles, and if you owned a castle you owned a town which lay outside its walls. These castles and towns were owned by one of three people, either a very successful knight (such as a knight of the round table), a wealthy man, known as a lord or a wealthy man out to kill and slaughter, known as a warlord.
None of these owners were nice, or kind (in most cases anyhow) and they levied massive taxes on the populace, just for living. Not to mention the taxes for owning a house, the even higher taxes for owning a farm, the rather high fee to enter the castle, the even higher fee to set up a stall, and finally the large commission on the goods sold within the castle walls (often up to 50%, depending on the owner!).
The knights, lords and warlords all paid taxes to the king, but they were very small compared to the taxes they levied on their followers. The rest of the money went to buying food to fatten them up and luxuries to make them lazy.
The only way to skip tax was to become part of the militia. It was a bad life, but usually better than life on your own. The militia was conscripted from the general populace, and if you were chosen you had the choice of joining, or death. You could also volunteer, but you got little special treatment as a result.
The only times castles became the impenetrable fortresses was during a siege. The enemy would surround the castle with men, forcing the drawbridge up and the portculli down, shutting the castle off from the outside. The castle couldn't get food and they had to surrender - until castle owners started building secret passages to combat that.
Then siege weapons were brought in and the Siege Attack started... but that's another story and I think I'll save it for my next write up.