Every time your team plays, it will aquire some formation experience with any formations used, in proportion to the number of minutes spent in the formations. This is true for both league and friendly matches.
If you don't play with a certain formation for some time, the experience in that formation will decrease. A good idea is to always use an alternative, or switch between two alternatives. If you use more alternatives it is difficult to maintain experience, so the players may get confused. Also, anytime you sell a player, you will risk losing formation experience.
Until season 38 (February 2009), National teams and National U20 teams would get formation experience even when using four or more inner midfielders or central defenders, but this was removed just a couple of weeks before the start of the World Cup.
Starting with season 42 (April 2010), it is no longer possible to play with more than three central defenders, inner midfielders, or forwards, and thus some types of extreme formations will no longer be possible to play. Concurrent with this change, the 4-4-2 no longer maintains excellent routine, and formation experience may be gained with any legal formation (and not just a subset of "standard" formations).