Continuity mistake: In the scene where Brooks Hatlen is committing suicide you see him engraving "Brooks was here" into the wood panel near the ceiling. The table he is standing on puts his face opposite the vertical wooden bars underneath the engraving (we see this in a shot looking up at him through these bars). Then when he rocks to tip the table over to hang himself he doesn't seem to fall at all, maybe a few inches if anything, yet when we see the pan out he has fallen at least 2 feet.
Continuity mistake: The "A" that Andy carves initially is different from the "A" shown near the end of the movie showing how he escaped.
Continuity mistake: When Hank Williams is singing "Lovesick Blues" the MGM label is from 1968. This is meant to be 1963.
Continuity mistake: The warden pulls out a box of Remington .38 Special cartridges to load his revolver prior to his suicide. The warden empties the cartridges on to his desk and loads the revolver with what are Winchester/Western cartridges.
Chosen answer: In a nutshell, tax evasion. He's running the prison like a company doing building work with, as its described by the builder in the movie, a pool of slave labour to allow him undercut other contractors. He is skimming profit for his own pockets, along with taking bribes, etc. that Andy's creative book keeping is hiding from the IRS.