kumquat-buildroot/support/scripts/check-uniq-files

50 lines
1.7 KiB
Plaintext
Raw Normal View History

core: check files are not touched by more than one package Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconciliating all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Peter Seiderer <ps.report@gmx.net> [Thomas: rename script without .py extension.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-28 17:30:59 +02:00
#!/usr/bin/env python
import sys
import csv
import argparse
from collections import defaultdict
warn = 'Warning: {0} file "{1}" is touched by more than one package: {2}\n'
core: check files are not touched by more than one package Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconciliating all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Peter Seiderer <ps.report@gmx.net> [Thomas: rename script without .py extension.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-28 17:30:59 +02:00
core: check files are not touched by more than one package Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconciliating all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Peter Seiderer <ps.report@gmx.net> [Thomas: rename script without .py extension.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-28 17:30:59 +02:00
def main():
parser = argparse.ArgumentParser()
parser.add_argument('packages_file_list', nargs='*',
help='The packages-file-list to check from')
core: check files are not touched by more than one package Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconciliating all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Peter Seiderer <ps.report@gmx.net> [Thomas: rename script without .py extension.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-28 17:30:59 +02:00
parser.add_argument('-t', '--type', metavar="TYPE",
help='Report as a TYPE file (TYPE is either target, staging, or host)')
args = parser.parse_args()
if not len(args.packages_file_list) == 1:
sys.stderr.write('No packages-file-list was provided.\n')
return False
if args.type is None:
sys.stderr.write('No type was provided\n')
return False
file_to_pkg = defaultdict(list)
with open(args.packages_file_list[0], 'rb') as pkg_file_list:
for line in pkg_file_list.readlines():
pkg, _, file = line.rstrip(b'\n').partition(b',')
core: check files are not touched by more than one package Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconciliating all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Peter Seiderer <ps.report@gmx.net> [Thomas: rename script without .py extension.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-28 17:30:59 +02:00
file_to_pkg[file].append(pkg)
for file in file_to_pkg:
if len(file_to_pkg[file]) > 1:
# If possible, try to decode the binary strings with
# the default user's locale
try:
sys.stderr.write(warn.format(args.type, file.decode(),
[p.decode() for p in file_to_pkg[file]]))
except UnicodeDecodeError:
# ... but fallback to just dumping them raw if they
# contain non-representable chars
sys.stderr.write(warn.format(args.type, file,
file_to_pkg[file]))
core: check files are not touched by more than one package Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconciliating all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Peter Seiderer <ps.report@gmx.net> [Thomas: rename script without .py extension.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-28 17:30:59 +02:00
core: check files are not touched by more than one package Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. However, when we later introduce top-level parallel build, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconciliating all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Arnout Vandecappelle <arnout@mind.be> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Peter Korsgaard <peter@korsgaard.com> Cc: Baruch Siach <baruch@tkos.co.il> Cc: Peter Seiderer <ps.report@gmx.net> [Thomas: rename script without .py extension.] Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
2017-10-28 17:30:59 +02:00
if __name__ == "__main__":
sys.exit(main())