I don't think it calculates the determinant; after all, it does not even take into account all entries of the matrix.

I believe it does this: Let's devide the matrix into the three vectors a = F(:,1), b=F(:,2) and c=F(:,3). Depending on the variable "face", it calculates the length of the cross product for a pair of these vectors. For example, if face.eq.BOTTOM_FACE, the norm |a x b| is calculated (where x is the cross product). Just observe that you can rewrite (|a x b|)^2 as a^2*b^2-(a*b)^2, where a*b is the dot product of a and b.

It then calculates the vector a x b itself: nor = a x b. This, of course, makes the first part of the code somewhat redundant as you could as well just calculate the norm of nor.