Every time (here, friendly matches are as effective as regular ones) your team plays with one of the 6 standard alternatives (4-3-3, 5-3-2, 3-5-2, 4-5-1, 3-4-3, 5-4-1), it will acquire experience with this formation. The team always has the maximum amount of experience with 4-4-2. The team can't acquire experience with formations differing from the 6 standard alternatives, so if you want to use an extreme formation, you'll have to put your trust in your players' individual experience.
If you don't play with a certain formation, the formations's experience will decrease over time. A good idea is to always use an alternative, or switch between two alternatives. If you use more, it is not surprising that the players will get confused. Also, anytime you sell a player, you will risk losing formation experience.