How is the title related to the story?

The title of the story is an allusion to Psalm 137 from the Bible.  

By the waters, the waters of BabylonWe sat down and wept,and wept, for thee, Zion.

At the time, the Israelites were weeping at their capture and removal from the promised holy land and Jerusalem.  At the time, Jerusalem was one of the main centers of civilization.  It, and its people were mighty, knowledgeable, and great.  But the greatness...

The title of the story is an allusion to Psalm 137 from the Bible.  

By the waters, the waters of Babylon
We sat down and wept,
and wept, for thee, Zion.

At the time, the Israelites were weeping at their capture and removal from the promised holy land and Jerusalem.  At the time, Jerusalem was one of the main centers of civilization.  It, and its people were mighty, knowledgeable, and great.  But the greatness was lost when the Babylonians came in and conquered.  Then it was the Babylonian empire that was the center of civilization and learning.  But like all great historical empires, it too fell from glory.  

New York, the Place of the Gods, is like the former Babylonian empire.  It was once great and fell despite all of its advanced technology.  When John is standing by the Hudson river, he is on the banks of a former amazing civilization.  That's very similar to what the people of Israel went through.  They are by the waters of the amazing Babylonians and lamenting the loss of their own former greatness.