The total number of sides in two regular polygons is 13, and total number of diagonals is 25. how many sides are in each polygon?

The answer is 5 sides and 8 sides.

x - number of sides in one polygon
y - number of sides in another polygon
The total number of sides in two regular polygons is 13:
x + y = 13

The number of diagonals of a polygon with n sides is: (n - 3)*n/2 = (n²-3n)/2. 
Total number of diagonals is 25:
(x²-3x)/2 + (y²-3y)/2 = 25
(x² - 3x + y² - 3y)/2 = 25
x² - 3x + y² - 3y = 25 * 2
x² - 3x + y² - 3y = 50

We have system of two equations:
x + y = 13
x² - 3x + y² - 3y = 50

x = 13 - y
x² - 3x + y² - 3y = 50

Substitute x in the first equation:
(13 - y)² - 3(13 - y) + y² - 3y = 50
169 - 26y + y² - 39 + 3y + y² - 3y = 50
169 - 39 - 26y + 3y - 3y + y² + y² = 50
130 - 26y + 2y² = 50
2y² - 26y + 130 - 50 = 0
2y² - 26y + 80 = 0

Divide both sides by 2:
y² - 13y + 40 = 0
y² - 8y - 5y + 40 = 0
y*y - 8*y - 5 * y - 5 * (-8) = 0
y*(y-8) - 5(y-8) = 0
(y - 8)(y - 5) = 0
y - 8 = 0 or y - 5 = 0
y = 8 or y = 5

When y = 8, x = 13 - 8 = 5
When y = 5, x = 13 - 5 = 8