There is a question on Stack Overflow talking about that. They say that Windows always will free everything.
But in this tutorial, they say it won't, and everything will keep in memory even after the program has closed.
From this tutorial:
Why? Well, let's say it would be a bad thing to do otherwise. Basically, if you create Direct3D, but never close it, it will just keep on running in the background of the computer until your next reboot, even after the program itself closes. Bad. Especially bad if you have a lot of resources in your game. Releasing these two interfaces let's everything off the hook and allows Windows to take back it's memory.