James Francis Baby and his children stayed in the area through the remainder of the nineteenth century. His son Raymond Baby’s home was located on the north side of what is now Langmuir Crescent near Orchard Crest (the plateau became his orchard). Raymond Avenue was the lane that led to his home.
About 1880, the family moved into a new home at 173 Jane Street (between Hanely and Montye). That home was demolished in the 1950s (about that time, Mrs. Baby, an elderly lady with flaming red hair, recollected that Annette Street had been named for one of the women of the family, and Jane Street for her best friend. That way, the two friends would always ‘meet’).
By 1901, the surviving grandsons of the Hon. James Baby had effectively lost control of the property. A complicated title situation was resolved when the Federal Government purchased the property in 1909, intending to use it as a new Garrison common to replace the one adjacent to old Fort York, but quickly realized there was insufficient distance from a residential area for rifle ranges. After purchasing an alternate site, the land was sold to developer Robert Home Smith. Home Smith submitted a Plan of Subdivision to the City of Toronto in April 1911, retaining the name of the man who first purchased the land – Baby Point.