In what lies does Nick catch Gatsby?

To catch someone in a lie means that the listener knows that the speaker is lying the moment he or she does so and then either challenges the utterance or chooses to ignore it. The listener must have prior knowledge of what the speaker is talking about to know that it is a lie. There is only one occasion, in chapter 5, that this actually happens. When Jay tells Nick that it had taken him only three years to earn the money to build his enormous house, Nick questions him and says, "I thought you inherited your money." Jay is quick to reply:

“I did, old sport,” he said automatically, “but I lost most of it in the big panic — the panic of the war.”

Although Nick does not pursue the matter any further, he does ask Jay what business he is involved in. Jay answers abruptly that it is his affair and later provides more detail when he realizes that his original response was inappropriate.

The above incident relates to a conversation between Nick and Jay in chapter 4. During what Nick calls their "disconcerting ride" in chapter 4, Jay divulges some details about his past to Nick. He tells Nick that his enormous wealth comes from an inheritance which he received when all the members of his family died. He also tells him about how he lived like a young rajah in all the capitals of Europe and that he collected jewels, went big game hunting, and painted.

Nick is skeptical about the information but does not declare outright that it is a lie. He suggests his skepticism by using phrases such as "I knew why Jordan Baker had believed he was lying" and "I wondered if there wasn’t something a little sinister about him, after all." He confirms his suspicion when he later mentions, "With an effort I managed to restrain my incredulous laughter." It is evident, though, that Nick is not entirely sure what to make of Jay's revelations. He states, for example,

My incredulity was submerged in fascination now; it was like skimming hastily through a dozen magazines.

Nick is finally persuaded that Jay is telling the truth when the latter shows him an old photograph of himself posing with a few other young men in Trinity Quad at Oxford. Nick declares his conviction by stating:

Then it was all true. I saw the skins of tigers flaming in his palace on the Grand Canal; I saw him opening a chest of rubies to ease, with their crimson-lighted depths, the gnawings of his broken heart.

In chapter 6, Nick makes it clear that Jay had divulged the truth about his past to him. Nick mentions that Jay had confessed that his parents were "shiftless and unsuccessful farm people" and that he had restlessly sought to find his true destiny. He had told Nick about Dan Cody who employed him for five years and the fact that when Cody died, he inherited twenty-five thousand dollars which he did not get because of some legal dispute. Nick provides two reasons for providing this information: 

...with the idea of exploding those first wild rumors about his antecedents, which weren’t even faintly true.

and

Moreover he told it to me at a time of confusion, when I had reached the point of believing everything and nothing about him. So I take advantage of this short halt, while Gatsby, so to speak, caught his breath, to clear this set of misconceptions away.

Nick, however, never learns exactly how Jay makes his money and how he, within five short years, has become so enormously wealthy. All he can surmise is that Jay has been, and is, caught up in some criminal activity with Meyer Wolfsheim and other shady characters involving bootlegging and dealing in fake bonds and securities.