UPSTREAM: mm/mremap: hold the rmap lock in write mode when moving page table entries.

commit 97113eb39fa7972722ff490b947d8af023e1f6a2 upstream.

To avoid a race between rmap walk and mremap, mremap does
take_rmap_locks().  The lock was taken to ensure that rmap walk don't miss
a page table entry due to PTE moves via move_pagetables().  The kernel
does further optimization of this lock such that if we are going to find
the newly added vma after the old vma, the rmap lock is not taken.  This
is because rmap walk would find the vmas in the same order and if we don't
find the page table attached to older vma we would find it with the new
vma which we would iterate later.

As explained in commit eb66ae030829 ("mremap: properly flush TLB before
releasing the page") mremap is special in that it doesn't take ownership
of the page.  The optimized version for PUD/PMD aligned mremap also
doesn't hold the ptl lock.  This can result in stale TLB entries as show

This patch updates the rmap locking requirement in mremap to handle the race condition
explained below with optimized mremap::

Optmized PMD move

    CPU 1                           CPU 2                                   CPU 3

    mremap(old_addr, new_addr)      page_shrinker/try_to_unmap_one


                                    addr = old_addr
    pmd = *old_pmd

    *new_pmd = pmd
                                                                            *new_addr = 10; and fills
                                                                            TLB with new addr
                                                                            and old pfn

                                    old pfn is free.
                                                                            Stale TLB entry

Optimized PUD move also suffers from a similar race.  Both the above race
condition can be fixed if we force mremap path to take rmap lock.

Bug: 248354871
Fixes: 2c91bd4a4e2e ("mm: speed up mremap by 20x on large regions")
Fixes: c49dd3401802 ("mm: speedup mremap on 1GB or larger regions")
Signed-off-by: Aneesh Kumar K.V <>
Acked-by: Hugh Dickins <>
Acked-by: Kirill A. Shutemov <>
Cc: Christophe Leroy <>
Cc: Joel Fernandes <>
Cc: Kalesh Singh <>
Cc: Kirill A. Shutemov <>
Cc: Michael Ellerman <>
Cc: Nicholas Piggin <>
Cc: Stephen Rothwell <>
Cc: <>
Signed-off-by: Andrew Morton <>
Signed-off-by: Linus Torvalds <>
[patch rewritten for backport since the code was refactored since]
Signed-off-by: Jann Horn <>
Signed-off-by: Greg Kroah-Hartman <>
Signed-off-by: Lee Jones <>
Change-Id: I046ea082f78b4c35c364dca305953a3f9ed28f1d
diff --git a/mm/mremap.c b/mm/mremap.c
index 8ce1b76..f6b8c00 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -293,12 +293,10 @@
 			bool moved;
-			if (need_rmap_locks)
-				take_rmap_locks(vma);
+			take_rmap_locks(vma);
 			moved = move_normal_pmd(vma, old_addr, new_addr,
 					old_end, old_pmd, new_pmd);
-			if (need_rmap_locks)
-				drop_rmap_locks(vma);
+			drop_rmap_locks(vma);
 			if (moved)