As you can see, it's in the Three-Point Pass. To get there, go through the Roaring-Sea Caves, and the Frigid Sea. You can easily fly to the Pass again after you've reached the temple for the first time.
Source
Hope I helped!
