When the wielder consumes an item or has one removed from it (for example via Knock Off, Bug Bite, Fling, or Natural Gift) its Speed is doubled until it switches out. This boost is not a change to the wielder's Speed stage, so if it were to use Agility after Unburden activates, it would have four times its original Speed. The Unburden boost cannot be Baton Passed, and losing items after the first does not provide additional boosts. Unburden will not activate if the wielder simply has no item; an item must be removed for it to activate.
So yes it would considering the gem is used. :O Scarf speed however is lost.
Sorry for edit, question was edited after I answered xD