Let's clarify a few things here:
1) Everybody seems to have worked out that the longest hole is a diagonal. So far so good.
2) Most have realized that the volume of wood removed is less than a cylinder of that length because of the pointy ends - and it's those ends that are the problem.
3) The ends are three-sided pyramids, not four-sided. Calculating their volumes could be done by relatively straightforward calculus, that is by integrating an infinite number of triangular laminae. Alternatively, just use the 'third the volume of the enclosing prism' rule.
But here's the fly in the ointment --
4) Those pyramids do not have flat bases.
Their lower edges are curves which extend downwards below their corners until they meet the cylinder below. This is where the trouble starts because you are now into the business of integrating triangles with their corners cut off, not by straight lines but by circular arcs.
5) To solve this one you will first have to derive a formula for the area of such a truncated triangle, which won't be easy. (Hint: Divide the thing into three smaller triangles interleaved with three circular sectors.) Once you have your formula, you then have to integrate it - and the best of British luck on that one!
