Welcome from the Netherlands.
Personly I started quite small some years ago just as well when I joined the forums, just the stuff I had from my childhood. Didnt have a single building but slow and steady I've been expanding a bit each year, I do have a rough playmobil budget for each year. Mostly medieval and some pirates.
It helps to keep a look out for reissued old sets through direct service since they can get very expensive on Ebay and other auction sites.
I've been collecting all medieval houses released in the past years and now I have the blacksmith, the baker, the museum and a yellow house. Next up is the tailor once its available here.
If you dont have a castle yet thats probably the best start when going medieval. Especially the older castles of the Steck system are very versatile and give plenty of options to build them in different ways.