European and U.S. lenders such as Credit Suisse and Deutsche Bank have set aside money to cover the cost of any losses arising from the dispute with the Federal Housing Finance Agency but estimates vary widely.
Shares in Royal Bank of Scotland, which had risen by a quarter since July 3 having slumped following the ousting of chief executive Stephen Hester in June, dropped over three percent on Friday after the UBS settlement was revealed.
The FHFA said late on Thursday UBS will pay $415 million and $470 million respectively to government-sponsored housing enterprises Fannie Mae and Freddie Mac to resolve claims related to securities sold to the companies between 2004 and 2007.
UBS is just one of 18 banks the FHFA pursued in 2011 for allegedly misrepresenting the quality of the collateral backing securities during the run-up to the financial crisis.
The Swiss bank is the third to settle, after Citigroup and General Electric did so for undisclosed sums. UBS said on Monday that its second-quarter profit beat forecasts even after the settlement, which it said then had been agreed in principle without specifying the exact amount involved.
The FHFA said it "remains committed to satisfactorily resolving the remaining suits as well" and the deal may lay down a marker for how much it could cost rival banks.
Fears it will face a hefty settlement added to uncertainty around RBS, which is striving to attract a replacement for Hester while the government conducts a review into whether it should be broken up.
The bank has already paid out $612 million to settle separate allegations that it manipulated benchmark interest rates and the government is anxious that the lender gets back on track so it can start to offload its 80 percent shareholding.
Ronnie Chopra, head of strategy at TradeNext, said fears RBS could face a multibillion-dollar payment in the U.S. "puts more negativity on the bank and highlights concerns regarding the finances of the behemoth".
Analysts at Credit Suisse earlier this year said European banks could take an $11 billion hit from a raft of mortgage-related litigation costs in the United States.
They estimated RBS alone could face an FHFA litigation loss of $1.6 billion, Barclays a $1.1 billion loss and HSBC could take a $900 million loss.
But another London-based analyst, Joseph Dickerson at investment bank Jefferies, said he expected RBS’s losses to be "sub-$1 billion".
Other banks have acknowledged they could incur losses from the suits but few have said how much it could cost.
Barclays said in its last annual report if it lost the cases against the FHFA and other civil actions it could incur a loss of up to the outstanding amount of the RMBS at the time of judgment and some additional interest and costs, less the market value of the RMBS.
It said the outstanding amount was $2.7 billion at the end of 2012, and estimated the market value was $1.6 billion.
Deutsche Bank has set aside 2.4 billion euros for litigation costs after topping that up in March by an additional 600 million euros, mainly related to lawsuits over its role in selling bonds backed by U.S. sub-prime mortgages.
HSBC said in its annual report it was unable to estimate reliably the financial effect of any action or litigation, but any claims "could be significant."