The Fisherman is not an evil man, that's why he didn't abandon the stowaways he found on his ship, but until his next stop he is stuck with them lest he lose track of what he's hunting. So, with no other real choice, he's having the kids earn their keep, or at least, he's trying to.