Turns out that Altium takes the middle of the part to be the middle of the three pads, not the middle of the physical part. You can set the part reference to a specified location, and Altium pick and place files have both the mid and ref coordinates. Here is the Altium pick and place file header:
"Designator","Footprint","Mid X","Mid Y","Ref X","Ref Y","Pad X","Pad Y","Layer","Rotation","Comment"
The problem is that LitePlacer uses mid coordinates. I'm hesitant to change that, as it would likely break it for other users. I could change it so, that if both md and ref are found, use ref. Even this might break it for some Altium users, so I'm not going to do that either, at least for now.
There are (at least) two ways to fix this:
1: Use reference coordinates.
If you manually change the header line in the csv file from
"Designator","Footprint","Mid X","Mid Y","Ref X","Ref Y","Pad X","Pad Y","Layer","Rotation","Comment"
to
"Designator","Footprint","mx","my","Mid X","Mid Y","Pad X","Pad Y","Layer","Rotation","Comment"
it makes the software to use the reference column. The downside is the manual step needed and potential problems if the reference is not the same
as the center in all parts; in other words, your system behavior changes
2: Modify the part itself.
This might be safer, as any potential issues will be related to the problematic part. The issue was solved by putting a small pad in the copper pour in a right position, which fools Altium to have the center where you want. The downside is the free pad (designator 0) in your part with no pin assignment, but this is no different than a mounting hole in a mechanical component.
You might want to check your files for parts where mid and ref coordinates are different to flag any potential issues. I know I need start doing this; easy enough in Excel, but an additional manual step. Maybe someone fluent in Altium scripts can build a check to be run on a full design rules check?