Let's not play pretend: most people have done this at at some point.
The boat which takes you from Battery Park to Liberty Island also stops at Ellis Island as considered one its destinations, so it's on method back.