You’re right, when exporting to e-books you should have the ‘never’ selected for page numbers.
If you’re exporting to mobi you don’t actually need to have a table of contents in your document. Kindle will create one automatically for you. For epub you should always include a Table of Contents.