The Thresher Maw's rough size of 60-90 meters in length gives it a severe size disadvantage to the sandworm, which is reportedly 400 meters long when full-grown. However, the sandworms are not built for speed or maneuverability, preferring to dig around while the thresher maw can move very quickly and with great force if it's in a location that allows it to burrow. Since the sandworm would be less-than maneuverable in a place consisting of solid and impassable material, and the fight would be very boring as the two creatures flail and coil rather than actually engaging in combat, the fight would need to take place in a location where they could both burrow.

Since the sandworm's main method of prey location involves detecting and following vibration, it would be safe to say that the thresher maw's movement could be easily followed, due to its aggressive movement and almost seismic method of locomotion. Thus, most of the battle would take place underground, where the thresher maw has the advantage of speed, but loses its preferred attack method of thrusting itself aboveground and using its body mass against the opponent. Still, its belowground speed and physical power essentially makes up for that, and it would be just as able to find its opponent as the sandworm would, since it possesses a similar vibration-based detection system, so the sandworm wouldn't be able to change directions and come at it from a different angle without it noticing.

However, the sandworm is very difficult to entirely destroy. The only ways to kill them permanently involve attacking the spread-out vital segments(The wiki says that the humans killed them by applying electricity to each segment at once, so sandworms may be able to change which segments are most vital, or regenerate, or something of the sort. It wasn't very clear.), which the thresher maw may not have the capacity for, nucleic weaponry, which would kill the thresher maw as well, or introducing it to water. I have no idea how the water kills it, as the wikis only say something along the lines of "The water, upon entering the sandworm's body, fatally accelerates its metabolism", so I'm not sure whether it would be an easy kill for the thresher to just aim in the sandworm's open mouth. Possibly not, because of the pH difference between water and any sort of fast-acting acid, but it bears consideration nonetheless.
They balance this out with a rather small offensive capability. They have their teeth, yes, but since their mouths are structured in a trifoil formation with inward-facing jawlines, they aren't exactly able to bite anything; their mouths are more suited to swallowing objects whole and letting their furnace-like digestive system deal with the details, which won't exactly work on the threshers. Not only would the thresher maw be able to see, or rather feel, it coming, the sandworms' mouths aren't large enough to swallow a full-grown thresher, especially if it's fighting to get away. In fact, even if it did manage to swallow the thresher, blood or other circulatory fluid is close enough to water that it can give the worms trouble, and be just as fatal in quantities above a few random humans that get caught in a truck the worm swallowed when it surface. The thresher would have enough blood to kill the sandworm outright, assuming it's chemically akin to blood on Earth, which usually stays at around 7.4 pH. If the blood isn't similar enough, however, the thresher's body would be so lodged in the worm's system that it would very likely end up strangling its digestive system the creature, which would end the fight in a draw.

Overall, however, I'm giving the thresher maw the win here. The sandworm would be hard to kill, but it could still be incapacitated and eventually worn down, while the sandworm simply doesn't have the right kind of offensive capacity to take out a full-grown thresher. A thresher maw, on the other hand, is enormously powerful offensively, and, while not as defensively powerful as the sandworms, it would still be able to take a lot more punishment than the sandworm can give it. It would be a very long fight, since it's a battle between two resilient behemoths instead of smaller and quicker beings, but I still say the thresher wins out at the end.

Admittedly, I'm not terribly proficient in either fandom; I got the information here from wiki-trawling and from reading a few of the Dune books a while back, so I could easily have missed something important to tip the scales one way or the other. Anyone have any corrections to make?

You do make some good points, admittedly, but I think you are missing some key things.

1) The Rancor has long, thin fingers attached to its claws. Easy to grip. Easy to break.

2) The Cave Troll knows how to use tools (aka weapons). It has been known to use its chain, a large club, and a spear.

3) The Rancor is not a fast creature. The Cave Troll is much faster.

4) The Cave Troll has a thick hide. Arrows and swords weren't able to effectively damage it. The Rancor's claws are likely not as sharp as an arrow or sword.



Advantage: Cave Troll

-Phobos