Aline connecting (5, 6) to another point has an undefined slope. what is the closest point on this line with whole number coordinates below (5, 6)?

Lets do it like this:
As this is a line connecting to another point that has an undefined slope, X is 5 for all values of Y: 
Then 
P1(5,6) 
P2(5,7). 
P3(5,8).

I hope this can help you

First subtract 10 from both side
y^2+6y-16=0
(y+8)(y-2)=0
-8 and 2