1754 The sugar house on Crown (now Liberty) Street in Manhattan was a six-story stone building that had been built in 1754 by the Livingston family as a refinery with very low floors.

1776 According to Revolutionary War veteran Levi Hanford, who was captured in March 1777, the cramped conditions initially housed about 40 to 50 prisoners. The population soon swelled to between 400 and 500, though attrition was constant due to those succumbing to illness. Rations consisted of pork and sea biscuits, which were often moldy from sea water and infested with worms. Nevertheless, the starving prisoners seldom refused the food, which was made consumable by placing it in a kettle of water and skimming off the parasites. Supplies for sick prisoners were provided by the fledgling American government, as Hanford stated that "the British furnished nothing." Deceased prisoners were sewn up into their blankets and placed in a corner of the yard for pickup by a dead cart in the morning; as many as fifteen bodies once accumulated in the period of one day. Prisoner exchanges were organized with the oldest prisoners having priority.

1846 The structure was later demolished in 1846. The site is now occupied by buildings numbered 34 and 36.