A right triangle has legs with lengths of 6in and 8in. What is the length of the hypotenuse

You require the Pythagorean Theorem: in a right triangle the sum of the squares of the legs equals the square of the hypotenuse.
In this case 6^2 + 8^2 = 36 + 64 = 100 = 10^2, i.e. the hypotenuse is 10.
Please note that a so-called 3-4-5 right triangle is the smallest right triangle which has integer sides...and of course, a 6-8-10 triangle like this one is simply a magnification of the smaller, i.e. a so-called similar triangle.

The answer to your question is b