In August 1881 Aboobaker Amod Jhaveri together with Hajee Mahomed Hajee Dada purchased a site in Grey Street (now known as Doctor Yusuf Dadoo Street) from K Moonsamy for one hundred and fifteen pounds with the intention of building a mosque there. A tiny brick and mortar structure which was situated on the site was converted into the mosque. In 1884 the two founders had the brick and mortar structure rebuilt, making it bigger. The Juma Masjid was the first mosque constructed in Natal, and in February of 1884 Aboobaker's estate bought land next to the mosque to enable its further expansion.

In 1889 Hajee Mahomed Dada, in his capacity as the only surviving trustee of the Grey Street (now known as Doctor Yusuf Dadoo Street) Mosque purchased more adjoining land due to the swift rise in the number of worshippers. The first of the two minarets on the Grey Street (now known as Doctor Yusuf Dadoo Street) Mosque was constructed in 1904. At the same time, two shops were built alongside the mosque providing an income for its maintenance. A second minaret was added to the mosque in 1905 and several rooms, toilets and shower facilities were also added at the rear of the mosque for travelers to use. Rooms were also built for the mu'adhdhin, who calls worshippers to prayer. All these dwellings had to be removed when the Juma Masjid Girls School was built adjacent to the mosque. These minarets were at time two of the highest structures in the city of Durban.The mosque was rebuilt in 1927 following the design of architects Payne and Payne. The building is a unique blend of Islamic decorations and strong Union period vernacular style. Further extensions and alterations were made to the Grey Street (now known as Doctor Yusuf Dadoo Street) Mosque in 1943.

Today the mosque building is a large plastered structure which features a mixture of various styles. A bridge extends from the neighbouring girls' school to the roof of the mosque. The flat roof, which is used for prayer during festivals is used as a playground during school days as the school is not equipped with one. The style of the Grey Street (now known as Doctor Yusuf Dadoo Street) Mosque is essentially geometrical. The windows and interleading doors and the arched doorways all stress this geometrical design. It's gilt-domed minarets protrude above the bustling commercial area, but inside the marbled worship hall is peaceful and boasts a simple elegance. The Grey Street (now known as Doctor Yusuf Dadoo Street) Mosque is said to be the oldest mosque in the southern hemisphere and until the late 1970s also enjoyed the status of being the largest mosque in the southern hemisphere.