When you receive the Armor Pass and you haven’t reached the Wild Area, the game will notify you that you can access it after reaching the Wild Area. :P
While you can start the Isle of Armor very early on, you cannot finish it early on. You can do most of the Story of the Isle of Armor, up until the the part where you must challenge a tower, which requires around level 65 Kubfu or higher. :P
Source: Experience (I did a play through where I picked up Kubfu early recently)
Hope I helped!