hi TARAJS,

There's a restriction on posting pictures until you've become a member (automatic at post 10).  This can be a nuisance for genuine posters but it is there to help prevent unsuitable posts from people who aren't really interested in math.  Hope you understand.

I've put my diagram below with the triangle starting in the corner and another position shown dotted.   

I'll try to justify why the 'solid' line triangle is the maximum.

To get the biggest area you would want the sides to be as long as possible.  The triangle GHI must have shorter sides than ACE.  You get the longest sides by moving the top vertex into a corner of the square and then stretching down towards the opposite sides as far as you can (AC and AE)

As both shapes are symmetrical the triangle must 'sit' equally either side of the diagonal AD.

ABD = 90 and CAE = 60, so the symmetry property means BAC = 15.

So you can calculate AC by using trig.

Uploaded Images

I tried to calculate the side of an equilateral triangle inscribed in a square in the general case. I refer to your figure, assume BD=a,  BG=d, BH=x and DI=y.  Pythagore theorem applied to GH, GI and HI as hypothenuses gives the two equations :
x^2+d^2 = (a-d)^2+y^2       (1)
x^2+d^2 = a^2+(y-x)^2       (2)
from which I should calculate x and y when GHI is equilateral. I thought that was imposssible, which explains why I found no refernce to this problem on Internet. Later I took a different approach to the problem and found very easily the solution
x= (2*a-d)/sqr(3) and y= (a+d)/sqr(3)
I wonder how we can get these values directly from equations (1) and (2)

Hi;

I have fixed it for you.