Currently only SysV init scripts are checked using shellcheck and a few
other rules (e.g. variable naming, file naming).
Extend the check using shellcheck to all shell scripts in the tree.
This is actually limited to the list of directories that check-package
knows that can check, but that list can be expanded later.
In order to apply the check to all shell scripts, use python3-magic to
determine the file type. Unfortunately, there are two different python
modules called "magic". Support both by detecting which one is installed
and defining get_filetype accordingly.
Keep testing first for name pattern, and only in the case there is no
match, check the file type. This ensures, for instance, that SysV
init scripts follow specific rules.
Apply these checks for shell scripts:
- shellcheck;
- trailing space;
- consecutive empty lines;
- empty last line on file;
- newline at end of file.
Update the list of ignored warnings.
Do not add unit tests since no function was added, they were just
reused.
But expand the runtime test for check-package using as fixture a file
that generates a shellcheck warning.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: support both variants of the "magic" module]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
Commit 4a6f9d2516 removed
package/urandom-scripts/S20urandom but didn't remove it from
.checkpackageignore. Do so now.
The commit actually renamed it to S20seedrng, but it also fixed the
Variables errors so it no longer needs to be ignored.
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
... just like check-flake8 already does.
When a new check_function is added to check-package, often there are
files in the tree that would generate warnings.
An example is the Sob check_function for patch files:
| $ ./utils/check-package --i Sob $(git ls-files) >/dev/null
| 369301 lines processed
| 46 warnings generated
Currently these warnings are listed when calling check-package directly,
and also at the output of pkg-stats, but the check_function does not run
on 'make check-package' (that is used to catch regressions on GitLab CI
'check-package' job) until all warnings in the tree are fixed.
This (theoretically) allows new .patch files be added without SoB,
without the GitLab CI catching it.
Since now check-package has an ignore file to list all warnings in the
tree, that will eventually be fixed, there is no need to filter the
files passed to check-package.
So test all files in the tree when 'make check-package' is called.
It brings following advantages;
- any new check_function added to check-package takes place immediately
for new files;
- adding new check_functions is less traumatic to the developer doing
this, since he/she does not need anymore to fix all warnings in the
tree before the new check_function takes effect;
- prevent regressions, e.g. ANY new .patch file must have SoB;
- as a side-effect, print a single statistics line as output of
'make ckeck-package'.
But just enabling the check would generate many warnings when
'make check-package' is called, so update the ignore file by using:
$ ./utils/docker-run make .checkpackageignore
Notice: in order to ensure reproducible results, one should run 'make
check-package' and 'make .checkpackageignore' inside the docker image,
otherwise a variation in shellcheck version (installed in the host) can
produce different results.
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
When a new check_function is added to check-package, often there are
files in the tree that would generate warnings.
An example is the Sob check_function for patch files:
| $ ./utils/check-package --i Sob $(git ls-files) >/dev/null
| 369301 lines processed
| 46 warnings generated
Currently these warnings are listed when calling check-package directly,
and also at the output of pkg-stats, but the check_function does not run
on 'make check-package' (that is used to catch regressions on GitLab CI
'check-package' job) until all warnings in the tree are fixed.
This (theoretically) allows new .patch files be added without SoB,
without the GitLab CI catching it.
So add a way to check-package itself ignore current warnings, while
still catching new files that do not follow that new check_function.
Add a file named .checkpackageignore to the buildroot topdir.
It contains the list of check_functions that are expected to fail for
each given intree file tested by check-package.
Each entries is in the format:
<filename> <check_function> [<check_function> ...]
These are 2 examples of possible entries:
package/initscripts/init.d/rcK ConsecutiveEmptyLines EmptyLastLine Shellcheck
utils/test-pkg Shellcheck
Keeping such a list allows us to have fine-grained control over which
warning to ignore.
In order to avoid this list to grow indefinitely, containing entries for
files that are already fixed, make each entry an 'expected to fail'
instead of just an 'ignore', and generate a warning if a check_function
that was expect to fail for a given files does not generate that
warning.
Unfortunately one case that do not generate warning is an entry for a
file that is deleted in a later commit.
By default, all checks are applied. The --ignore-list option allows to
specify a file that contains the list of warnings that should be
ignored.
The paths in the ignore file must be relative to the location of the
ignore file itself, which means:
- in the main Buildroot tree, the paths in the ignore file are
relative to the root of the main Buildroot tree
- in a BR2_EXTERNAL tree, if the ignore file is at the root of the
BR2_EXTERNAL, the paths it contains must be relative to that root
of the BR2_EXTERNAL
This is one more step towards standardizing the use of just 'make
check-package' before submitting patches to the list.
Cc: Sen Hastings <sen@phobosdpl.com>
Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
