Contacting previous employers

I have an interview next week. They want me to fill out some paperwork and one of them states they need to contact my previous employer. Thing is... I don't want them to. The director and I had issues

I have an interview next week. They want me to fill out some paperwork and one of them states they need to contact my previous employer. Thing is... I don't want them to. The director and I had issues with each other and I dont want him ruining my chances of getting this job. I also don't want to talk bad about him in the interview as our problems were personal not professional. (We just didnt like each other) What can I tell this job that doesnt make it look like Im hiding something?