From the New York Sun:
When the rich families moved away, Union Square became the fancy shopping district. Such stores as Tiffany's and Brentano's Literary Emporium, and restaurants like Delmonico's and Lüchow's, were there. In Edith Wharton's "The Age of Innocence," which is set in the 1870s, society ritually congregates at the Academy of Music, on the northeast corner of 14th Street and Irving Place. (It was torn down for the Consolidated Edison Building, and shouldn't be confused with the much later movie theater called the Academy of Music on the other side of 14th Street.) Theaters abounded around Union Square in the late 19th century, making it the forerunner to today's Times Square theater district.