Agassiz refused to sell his property. He and other local landowners did not want a fog bell sounding "at their very doors." In 1886 a lighthouse was proposed along with the fog bell at Castle Hill.
Agassiz finally sold a portion of his land to the Government, but then refused to grant the Government the right to pass through his property to the lighthouse site. Castle Hill is steep and rocky, and landing by boat is difficult. In 1888 Agassiz finally granted the right-of-way and construction began the following year.