The Singapore-registered MV X-Press Pearl, carrying 1,486 containers, including 25 tonnes of nitric acid along with other chemicals and cosmetics, was anchored offshore when a fire erupted after an explosion on May 20.
Flaming containers filled with billions of plastic pellets tumbled into the sea from the ship's deck as emergency crews sought to contain the blaze over the next two weeks.
In Lellama, a fishing town north of the capital Colombo, more than 100,000 people are partly or wholly dependent on the 2,000 or so boats that normally operate from there.
"We're told not to go to sea: that has caused a lot of problems," said 67-year-old fisherman Antony Sebastian.
"Fishermen are very helpless. We can't do our work because of this ship for more than a week now. We are just hanging around at home."
The disaster is an additional blow to the fishing community, already badly affected by repeated lockdowns due to the coronavirus.
The X-Press Pearl, which was only four months old, began to sink early on Wednesday. A salvage crew tried to tow the vessel to deeper water, away from the coast, but the attempt was abandoned after the rear of the ship touched the sea bed.
Now there are fears some 300 tonnes of fuel oil remaining onboard could leak from the ship.
The ship's operators, X-Press Feeders, said on Friday there was no sign of an oil spill.
The firm has previously said that much of the remaining toxic cargo had been incinerated in the fire.
But experts say the damage to the marine habitat may already have been done.
Charitha Pattiaratchi, a professor of coastal oceanography at the University of Western Australia, said the plastic pellets, that are expected to travel as far as Indonesia and Somalia, could act as a breeding ground for bacteria, putting marine life at risk.
Nirashan Fernando, a 44-year-old fisherman, said some boats had been able to evade the navy, which has been turning back those still trying to work, but the outlook wasn't good.