Hey I changed my mind I'll add English sub instead of Chinese.
I just went through about 10+ duplicates in ABP folder and it seems to have a pattern. The smaller file is cut off at the end about 10-20 lines, so it is incomplete. The larger file is the complete file. The translation are the same, so you should delete the smaller file. Would be frustrating for the user every time to have subtitles stop near the end of movie.
One exception is ABP-004, the larger file have partial jibberish even though it's complete. The smaller file is good translation but incomplete. So keep them both. I have not checked other folders.
While I appreciate the feedback, I'm not sure how you're expecting me to do that for the approximate 4K+ subtitles that are alternate versions for the same movie. This is an automated process, so I can't just arbitrarily pick the larger version, as you pointed out, since it might be worse than the smaller one. And of course, I could only really judge the ~500 English files in this category.
There are also some examples of people releasing part A & B subtitles. Unfortunately, this impossible to detect and distinguish between incomplete versions, and semantically, it was too difficult to preserve that. So, for scenarios like this, I think the safest policy is to delete only exact duplicates. I understand that people may need to look through multiple files, and there's no guarantee of quality or completeness, but I just don't have the time or ability to go though and clean up the entire collection by hand.


