File chunks reported as Roff

[DecimalTurn]

I was looking at Programming Language Trends on Languish and saw that Roff had a major uptick:

Turns out it's related to files like this one that are a binary file chunk.

Original hypothesis for the cause

It seems the reason why these files are not seen as binary is because although the start of the file looks like text, the binary data appears too far into the file to trigger charlock_holmes' binary detection:

[...]

Perhaps, the heuristics for Roff could be edited to avoid matching this type of file? The rule that matches is this one: ^\.(?:[A-Za-z]{2}(?:\s|$)|\\") at the end.

Matching here:

[!NOTE]
It's not possible that the heuristic is matching this far in the file since the strategy only considers the first 50 kb of the file.

[DecimalTurn]

Any thoughts on this @Alhadis? I was thinking we could use a regex trick to limit the search to a certain number of rows (similar to #6965).

For example, we could limit the search to the first 100 rows by prepending this: \A(?:.*(?:\r?\n|\r)){0,99} to ^\.(?:[A-Za-z]{2}(?:\s|$)|\\"). Is it reasonable to expect the heuristic to match with real Roff files with such limit in place?

[DecimalTurn]

I tried the regex above and there was apparently issues with non-fixed length. We'd have to make a seperate negative lookbehinds instead. Perhaps a multiline version would be cleaner:

      MANPAGE_EXTS = /
        (?:
          (?<!\.pdf)
          (?<!\.mp3)
          (?<!\.mp4)
          (?<!\.avi)
        )
        \.(?:[1-9](?![0-9])[a-z_0-9]*|0p|n|man|mdoc)(?:\.in)?$
      /ix

Or maybe we don't have to go this way in case the Manpage strategy should avoid matching all files that have another prepended extension like :

• filename.something.1
• filename.something.1.in

If that's the case maybe a good solution would be to ensure we don't have any prepended extension by matching anything except a . starting from the start:

MANPAGE_EXTS = /^[^\.]*\.(?:[1-9](?![0-9])[a-z_0-9]*|0p|n|man|mdoc)(?:\.in)?$/i

[DecimalTurn]

Wouldn't that break RE2 compatibility and effectively contradict several of your PRs on making Linguist RE2 compatible?

@lildude The regex already includes a negative lookahead ((?![0-9])), so it's already not RE2 compatible. I'm just adding a little lookbehind to balance things out 😅

[DecimalTurn]

I was going to say that go-enry doesn't use this strat, but it actually does and they've made it RE2 compatible: https://github.com/go-enry/go-enry/blob/4a80f7a1d1c73b102bbde484a8f0fa4317d428c1/common.go#L415

So, perhaps to keep things simple, we just use the last solution I mentioned with "no prepended extension".

If only charlock_holmes could actually detect this blob as binary!

[Alhadis]

Oh, good to know! In this context, what do you think of the idea to add a negative lookbehind at the start of the MANPAGE_EXTS regex that would prevent matching with extensions such as .pdf.1, .mp4.1, etc. ?

For a start, this would affect legitimate man pages whose names happen to end in .pdf (though rare, it's still valid and something that needs to be considered). Secondly, how would you determine the list of file extensions to check for? We can't logically include a list of every known binary file-format in the regex.

I'd like to point you back to the issue we're trying to solve here, and that being a handful of users who (ostensibly) downloaded a file with wget(1) and didn't bother to correct the numeric suffix appended to filenames that risk overwriting an existing file in the download directory. We shouldn't complicate Linguist with needless complexity to solve what's ultimately an issue with users. If they take issue with .pdf.1 being interpreted as Roff files, the the onus is on them to fix the file extension (e.g., to .1.pdf or whatever).

[DecimalTurn]

I'll first address the second part of your message, because perhaps the motivation and context around this discussion were not clearly expressed by me.

I'd like to point you back to the issue we're trying to solve here, and that being a handful of users who (ostensibly) downloaded a file with wget(1)

The problem I'm raising here has little to do with wget(1) and everything to do with people trying to work around GitHub's upload limit by splitting large files in multiple chunks.

That might not be a lot of people, but when their repo ends up being the most starred Roff repo when searching for Roff on GitHub, I think that this issue is worth fixing.

For a start, this would affect legitimate man pages whose names happen to end in .pdf

Considering that there's only 2 files and they're both html, I'm not sure how much weight this consideration should have in the discussion, but I'm open to other ideas.

Secondly, how would you determine the list of file extensions to check for? We can't logically include a list of every known binary file-format in the regex.

I think we could limit this to .pdf, .mp4 and .mp3 for now since those files can be large and are the one that need to be seperated in chunks. We can always adjust if we see the need to.

[DecimalTurn]

The more I think about this the more it seems that we just need to add the fallback to Text in the heuristic rules for single digit extensions in order to get the desired (and expected) outcome.

- extensions: ['.1', '.2', '.3', '.4', '.5', '.6', '.7', '.8', '.9']
  rules:
  - language: Roff Manpage
    and:
    - named_pattern: mdoc-date
    - named_pattern: mdoc-title
    - named_pattern: mdoc-heading
  - language: Roff Manpage
    and:
    - named_pattern: man-title
    - named_pattern: man-heading
  - language: Roff
    pattern: '^\.(?:[A-Za-z]{2}(?:\s|$)|\\")'
  - language: Text # <------ Adding this

Explanation:

This will make sure that even if the Manpage strategy selects Roff Manpage and Roff as candidates, they'd still have to match with a heuristic rule in order for the file to be identified as Roff. This would match what is expected of a file extension that is part of our generic file extensions and it avoids having to consider various prepended extension types.

This solves the issue for the file chunk mentioned in the OP, but also for files like these ones:

• https://github.com/pyenv/pyenv/blob/master/plugins/python-build/share/python-build/2.1.3 (MIT)
• https://github.com/GangZhuo/BaiduPCS/blob/master/autom4te.cache/output.2 (MIT)
• https://github.com/ophub/amlogic-s9xxx-armbian/blob/main/compile-kernel/tools/config/config-6.1