Yes, you should be able to. Other users on the plan won't be affected unless there's a bulk price for that family plan you are currently using. Don't end the subscription of your number yet, contact that carrier you plan to move to and they will help you do the moving.
What if I am not the account holder, my husband I am separated from is. Family plan with kids.