It looks like your query (which you should show us) has something like
ORDER BY modifieddate LIMIT 100
It thinks it can walk the index in order, then stop once it collects 100 qualifying rows. But since almost all rows are removed by the join conditions, it ends up walking a large chunk of the index before finding 100 of them which qualify.
You could try forcing it out of this plan by doing:
ORDER BY modifieddate + interval '0 second' LIMIT 100