You can use cardano-cli to convert an extended signing key to a Shelley-format key (which is what you need in order to sign a transaction).
cardano-cli key convert-cardano-address-key --shelley-payment-key --signing-key-file address.xsk --out-file address.skey
I believe this is the missing piece of information you are after. The steps needed to get address.xsk are the same as those needed to derive the extended verification keys. The answer to this question covers that.
Later edit : Here are all the steps for getting a signing key usable with cardano-cli from a seed phrase
./cardano-address key from-recovery-phrase Shelley < phrase.prv > root.xsk
# Derive the address signing key
./cardano-address key child 1852H/1815H/0H/0/2 < root.xsk > address.xsk
# Convert the extended signing key to a Shelley-format key
cardano-cli key convert-cardano-address-key --shelley-payment-key --signing-key-file address.xsk --out-file address.skey
In my case the address holding funds had this derivation path 1852H/1815H/0H/0/2. In Yoroi you can see the derivation path for an address when you click the ✓ Verify address button.
You can derive any path you need following the steps above.