It's 'an' instead of 'a' because other begins with a vowel.
If you add a modifier, such as the word whole, you have two options:
- split other from its article and insert the modifier before other - That's a whole other story.
- add the modifier after the word another - That's another whole story.
That 'n' goes away. It does not belong to other.
That's a whole nother story is incorrect.
Nother is not a word.