294fc3218c
The major bottleneck in pkg-stats is the time spent waiting for answers from remote servers. Two functions involve such communication with remote servers: - 'check_package_urls' which checks that each package upstream website is up, it is efficient due to the use of process-pools thanks to Matt Weber. - 'check_package_latest_version' which fetches the latest package version from release-monitoring, it uses a http-pool but runs sequentially. This patch extends the use of process-pools to 'check_latest_version'. Due to some limitations of multiprocess callbacks, this patch loses the overall progress of packages in favour of just the current package name. Runtimes for this function are ~3m vs ~25m for the linear version. Tested on an i7 7500U (2/4 cores/threads @3.5GHz) with 15ms ping. Note: There have already been work trying to parallelize this function using threads but there were a failure on some configurations [1]. This implementation rely on a dedicated module already in use on this script, so it's unlikely to see failure with this version. [1] http://lists.busybox.net/pipermail/buildroot/2018-March/215368.html Signed-off-by: Victor Huesca <victor.huesca@bootlin.com> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> |
||
|---|---|---|
| .. | ||
| config-fragments | ||
| dependencies | ||
| docker | ||
| download | ||
| gnuconfig | ||
| kconfig | ||
| legal-info | ||
| libtool | ||
| misc | ||
| scripts | ||
| testing | ||