Every time your team plays a match with one of the 6 standard alternatives (4-3-3, 5-3-2, 3-5-2, 4-5-1, 3-4-3, 5-4-1, it will acquire formation experience. Formation experience for 4-4-2 is always excellent. The team can't acquire formation experience with formations differing from the 6 standard alternatives; so if you want to use extreme formations, you'll have to put your trust in your players' individual experience.
If you don't play with a certain formation, experience will decrease over time. A good idea is to always use an alternative, or switch between two alternatives. If you use more, it is not surprising that the players will get confused. Also, anytime you sell a player, you will risk losing formation experience.