Every time your team play with one of the 6 standard formations (4-3-3, 5-3-2, 3-5-2, 4-5-1, 3-4-3, 5-4-1), it will aquire some Formation experience with that formation. This is true for both league and friendly matches. The team always has the maximum amount of experience with 4-4-2. The team can't acquire experience with formations differing from the 6 standard alternatives, so if you want to use an extreme formation, you'll have to put your trust in your players' individual experience.
If you don't play with a certain formation for some time, the experience in that formation will decrease. A good idea is to always use an alternative, or switch between two alternatives. If you use more alternatives it is difficult to maintain experience, so the players may get confused. Also, anytime you sell a player, you will risk losing formation experience.