commit eac78a4b0b7da4de2c0a297f4d528ca9cc6256a3
author Helge Deller <deller@gmx.de> Mon Jul 17 12:39:38 2023 +0200
committer Helge Deller <deller@gmx.de> Tue Jul 18 20:42:05 2023 +0200
tree 31c66f7598c03a387372051a5192f2caa7b83595
parent dfe49864afb06e7e452a4366051697bc4fcfc1a5

linux-user: Fix signed math overflow in brk() syscall

Fix the math overflow when calculating the new_malloc_size.

new_host_brk_page and brk_page are unsigned integers. If userspace
reduces the heap, new_host_brk_page is lower than brk_page which results
in a huge positive number (but should actually be negative).

Fix it by adding a proper check and as such make the code more readable.

Signed-off-by: Helge Deller <deller@gmx.de>
Tested-by: "Markus F.X.J. Oberhumer" <markus@oberhumer.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Fixes: 86f04735ac ("linux-user: Fix brk() to release pages")
Cc: qemu-stable@nongnu.org
Buglink: https://github.com/upx/upx/issues/683