My girlfriend and her father are co-renters, can the father kick me out at his discretion?

Here’s the situation. It’s a bit lengthy, so I’ll try to break it up to make it easier to read.

This is purely a legal question. It’s irrelevant whether or not I remain on good terms with my girlfriend’s father. Anything that follows is just in case there are any applicable circumstances that affect the legality. Location: oneida county/new york state.

My girlfriend and her father are both on the lease for a subsidized apartment. I am a frequent guest, not a squatter. The father wants to kick me out and keep me out because I spend the night too often.

The reason I’m there so much is that I receive late night homework help from my girlfriend. Afterward, we’re both tired and fall asleep on her bed. Nothing nefarious.

It gets to be 3-5am and the father starts pounding on the door every ten minutes yelling at the top of his lungs for me to get out. He claims that he loses sleep because of how late I stay. He wakes the neighbors up with how loud he is while I don’t say a word to instigate it.

During the late hours of my stay, I’m asleep. I don’t make noise to wake him up or keep him up. He purposely keeps himself up in his bedroom with his light on to make sure he doesn’t fall asleep until he sees that my car is gone and that I’ve left. He blames me for keeping him up when he’s the one that makes that choice. Any sleep can be made up during his job-less day.

There have been numerous times that he’s accidentally fallen asleep, didn’t realized that I left, and pounded on his daughter’s door waking her up from deep sleep to get me out. So my presence really can’t directly be affecting him that much.

I don’t claim to live there. I have my own official residence. None of my possessions are at her apartment. Am I allowed visitation rights?

[optional]Here’s the background information:
-My girlfriend and her father have an agreement on how they split costs. She doesn’t technically pay rent, but rather half the time warner bill and mostly food.
-He hasn’t had a job in over a decade and has been milking his disability (I know, it speculative)
-She has a part-time job, goes to school full-time, owns a car in her name on his insurance and cleans up more than he does. He claims she is lazy while he spends all day watching court tv.
-He constantly borrows money from her to pay rent and his half of cable. He could afford it if he did not buy beer, cigarettes, and pot (though I don’t want him charged for possession), and constantly go to the casino in her car. He pays for luxuries first and comes short on rent all the time. [/optional]

I rarely talk to him. And for months during his loud, late-night complaints, I’ve never had a confrontation with him. He’s always yelled at his daughter about it. So I’m not sure what kind of complaint he can make to authorities if I don’t do anything other than exist.

Neither me or my girlfriend want me to leave on his terms. I want to spend time with her. What do I do? How much leverage does he have over her? Is it within his legal rights to kick out his roommate’s guest?
Does it matter at all that my girlfriend is also on the lease as an equal tenant?