The closest banks are in Paihia, a 15 minute drive away. If you are going to be here for a while you may want to open an account with one of these banks and get an EFTPOS card (cash card). It will minimise the foreign transaction fees on your offshore credit card. Some businesses charge a 1.5% fee to use a credit card but no fee for an EFTPOS card.
You will most likely need an IRD (tax) number to open an account.