Note: I am only listing the berries that you cannot get from berry patches, these berries may be available through catching wild Pokemon, contests.But I will include where you can get them if not from berry patches.
Cornn Berry | Berry master or Little Girl in Sootopolis City
Magost Berry | Berry master or Little Girl in Sootopolis City
Rabuta Berry | Berry master or Little Girl in Sootopolis City
Nomel Berry | Berry master or Little Girl in Sootopolis City
Spelon Berry | Tell the berry master's wife the special saying or beating hyper rank of a Contest and getting it from a fan
Pamtre Berry | Tell the berry master's wife the special saying or beating hyper rank of a Contest and getting it from a fan
Watmel Berry | Tell the berry master's wife the special saying or beating hyper rank of a Contest and getting it from a fan
Durin Berry | Tell the berry master's wife the special saying or beating hyper rank of a Contest and getting it from a fan
Belue Berry | Tell the berry master's wife the special saying or beating hyper rank of a Contest and getting it from a fan
Liechi Berry | Secret Pal who has the Berry Farmer skill
Ganlon Berry | Secret Pal who has the Berry Farmer skill
Salac Berry | ?
Petaya Berry | Secret Pal who has the Berry Farmer skill
Apicot Berry | Secret Pal who has the Berry Farmer skill
Lansat berry | Win 100 super battles in a row
Starf Berry | Win 200 super battles in a row
Enigma Berry | Special events
Micle Berry | Special events
Custap berry | Special events
Jaboca Berry | Special events
Rowap Berry | ?
Kee Berry | Secret Pal who has the Berry Farmer skill
Maranga Berry | Secret Pal who has the Berry Farmer skill
So, I would say the rarest ones are the special event only berries or the ? berries.
Hope I helped!
Source1
Source2
Source3