Test n1: no terrain
Test n2: terrain
So, after agility, Dragapult has 698 speed, and Raichu has 350, so Diancie is still slower. However in Electric Terrain Raichu has 700 speed, meaning it outspeeds dragapult. However it still moved before Diancie, so it seems like ability modifiers don't affect Speed Swap at all.
Edit: The showdown replays are dead :(
Here's a quote from Speed Swap's Bulbapedia page as a new source to this answer
Speed Swap exchanges the Speed stats of the user and the target. This move does not swap in-battle speed modifiers, such as Abilities and stat stages.