gdb: move displaced stepping logic to gdbarch, allow starting concurrent displaced steps

Today, GDB only allows a single displaced stepping operation to happen
per inferior at a time.  There is a single displaced stepping buffer per
inferior, whose address is fixed (obtained with
gdbarch_displaced_step_location), managed by infrun.c.

In the case of the AMD ROCm target [1] (in the context of which this
work has been done), it is typical to have thousands of threads (or
waves, in SMT terminology) executing the same code, hitting the same
breakpoint (possibly conditional) and needing to to displaced step it at
the same time.  The limitation of only one displaced step executing at a
any given time becomes a real bottleneck.

To fix this bottleneck, we want to make it possible for threads of a
same inferior to execute multiple displaced steps in parallel.  This
patch builds the foundation for that.

In essence, this patch moves the task of preparing a displaced step and
cleaning up after to gdbarch functions.  This allows using different
schemes for allocating and managing displaced stepping buffers for
different platforms.  The gdbarch decides how to assign a buffer to a
thread that needs to execute a displaced step.

On the ROCm target, we are able to allocate one displaced stepping
buffer per thread, so a thread will never have to wait to execute a
displaced step.

On Linux, the entry point of the executable if used as the displaced
stepping buffer, since we assume that this code won't get used after
startup.  From what I saw (I checked with a binary generated against
glibc and musl), on AMD64 we have enough space there to fit two
displaced stepping buffers.  A subsequent patch makes AMD64/Linux use
two buffers.

In addition to having multiple displaced stepping buffers, there is also
the idea of sharing displaced stepping buffers between threads.  Two
threads doing displaced steps for the same PC could use the same buffer
at the same time.  Two threads stepping over the same instruction (same
opcode) at two different PCs may also be able to share a displaced
stepping buffer.  This is an idea for future patches, but the
architecture built by this patch is made to allow this.

Now, the implementation details.  The main part of this patch is moving
the responsibility of preparing and finishing a displaced step to the
gdbarch.  Before this patch, preparing a displaced step is driven by the
displaced_step_prepare_throw function.  It does some calls to the
gdbarch to do some low-level operations, but the high-level logic is
there.  The steps are roughly:

- Ask the gdbarch for the displaced step buffer location
- Save the existing bytes in the displaced step buffer
- Ask the gdbarch to copy the instruction into the displaced step buffer
- Set the pc of the thread to the beginning of the displaced step buffer

Similarly, the "fixup" phase, executed after the instruction was
successfully single-stepped, is driven by the infrun code (function
displaced_step_finish).  The steps are roughly:

- Restore the original bytes in the displaced stepping buffer
- Ask the gdbarch to fixup the instruction result (adjust the target's
  registers or memory to do as if the instruction had been executed in
  its original location)

The displaced_step_inferior_state::step_thread field indicates which
thread (if any) is currently using the displaced stepping buffer, so it
is used by displaced_step_prepare_throw to check if the displaced
stepping buffer is free to use or not.

This patch defers the whole task of preparing and cleaning up after a
displaced step to the gdbarch.  Two new main gdbarch methods are added,
with the following semantics:

  - gdbarch_displaced_step_prepare: Prepare for the given thread to
    execute a displaced step of the instruction located at its current PC.
    Upon return, everything should be ready for GDB to resume the thread
    (with either a single step or continue, as indicated by
    gdbarch_displaced_step_hw_singlestep) to make it displaced step the
    instruction.

  - gdbarch_displaced_step_finish: Called when the thread stopped after
    having started a displaced step.  Verify if the instruction was
    executed, if so apply any fixup required to compensate for the fact
    that the instruction was executed at a different place than its
    original pc.  Release any resources that were allocated for this
    displaced step.  Upon return, everything should be ready for GDB to
    resume the thread in its "normal" code path.

The displaced_step_prepare_throw function now pretty much just offloads
to gdbarch_displaced_step_prepare and the displaced_step_finish function
offloads to gdbarch_displaced_step_finish.

The gdbarch_displaced_step_location method is now unnecessary, so is
removed.  Indeed, the core of GDB doesn't know how many displaced step
buffers there are nor where they are.

To keep the existing behavior for existing architectures, the logic that
was previously implemented in infrun.c for preparing and finishing a
displaced step is moved to displaced-stepping.c, to the
displaced_step_buffer class.  Architectures are modified to implement
the new gdbarch methods using this class.  The behavior is not expected
to change.

The other important change (which arises from the above) is that the
core of GDB no longer prevents concurrent displaced steps.  Before this
patch, start_step_over walks the global step over chain and tries to
initiate a step over (whether it is in-line or displaced).  It follows
these rules:

  - if an in-line step is in progress (in any inferior), don't start any
    other step over
  - if a displaced step is in progress for an inferior, don't start
    another displaced step for that inferior

After starting a displaced step for a given inferior, it won't start
another displaced step for that inferior.

In the new code, start_step_over simply tries to initiate step overs for
all the threads in the list.  But because threads may be added back to
the global list as it iterates the global list, trying to initiate step
overs, start_step_over now starts by stealing the global queue into a
local queue and iterates on the local queue.  In the typical case, each
thread will either:

  - have initiated a displaced step and be resumed
  - have been added back by the global step over queue by
    displaced_step_prepare_throw, because the gdbarch will have returned
    that there aren't enough resources (i.e. buffers) to initiate a
    displaced step for that thread

Lastly, if start_step_over initiates an in-line step, it stops
iterating, and moves back whatever remaining threads it had in its local
step over queue to the global step over queue.

Two other gdbarch methods are added, to handle some slightly annoying
corner cases.  They feel awkwardly specific to these cases, but I don't
see any way around them:

  - gdbarch_displaced_step_copy_insn_closure_by_addr: in
    arm_pc_is_thumb, arm-tdep.c wants to get the closure for a given
    buffer address.

  - gdbarch_displaced_step_restore_all_in_ptid: when a process forks
    (at least on Linux), the address space is copied.  If some displaced
    step buffers were in use at the time of the fork, we need to restore
    the original bytes in the child's address space.

These two adjustments are also made in infrun.c:

  - prepare_for_detach: there may be multiple threads doing displaced
    steps when we detach, so wait until all of them are done

  - handle_inferior_event: when we handle a fork event for a given
    thread, it's possible that other threads are doing a displaced step at
    the same time.  Make sure to restore the displaced step buffer
    contents in the child for them.

[1] https://github.com/ROCm-Developer-Tools/ROCgdb

gdb/ChangeLog:

	* displaced-stepping.h (struct
	displaced_step_copy_insn_closure): Adjust comments.
	(struct displaced_step_inferior_state) <step_thread,
	step_gdbarch, step_closure, step_original, step_copy,
	step_saved_copy>: Remove fields.
	(struct displaced_step_thread_state): New.
	(struct displaced_step_buffer): New.
	* displaced-stepping.c (displaced_step_buffer::prepare): New.
	(write_memory_ptid): Move from infrun.c.
	(displaced_step_instruction_executed_successfully): New,
	factored out of displaced_step_finish.
	(displaced_step_buffer::finish): New.
	(displaced_step_buffer::copy_insn_closure_by_addr): New.
	(displaced_step_buffer::restore_in_ptid): New.
	* gdbarch.sh (displaced_step_location): Remove.
	(displaced_step_prepare, displaced_step_finish,
	displaced_step_copy_insn_closure_by_addr,
	displaced_step_restore_all_in_ptid): New.
	* gdbarch.c: Re-generate.
	* gdbarch.h: Re-generate.
	* gdbthread.h (class thread_info) <displaced_step_state>: New
	field.
	(thread_step_over_chain_remove): New declaration.
	(thread_step_over_chain_next): New declaration.
	(thread_step_over_chain_length): New declaration.
	* thread.c (thread_step_over_chain_remove): Make non-static.
	(thread_step_over_chain_next): New.
	(global_thread_step_over_chain_next): Use
	thread_step_over_chain_next.
	(thread_step_over_chain_length): New.
	(global_thread_step_over_chain_enqueue): Add debug print.
	(global_thread_step_over_chain_remove): Add debug print.
	* infrun.h (get_displaced_step_copy_insn_closure_by_addr):
	Remove.
	* infrun.c (get_displaced_stepping_state): New.
	(displaced_step_in_progress_any_inferior): Remove.
	(displaced_step_in_progress_thread): Adjust.
	(displaced_step_in_progress): Adjust.
	(displaced_step_in_progress_any_thread): New.
	(get_displaced_step_copy_insn_closure_by_addr): Remove.
	(gdbarch_supports_displaced_stepping): Use
	gdbarch_displaced_step_prepare_p.
	(displaced_step_reset): Change parameter from inferior to
	thread.
	(displaced_step_prepare_throw): Implement using
	gdbarch_displaced_step_prepare.
	(write_memory_ptid): Move to displaced-step.c.
	(displaced_step_restore): Remove.
	(displaced_step_finish): Implement using
	gdbarch_displaced_step_finish.
	(start_step_over): Allow starting more than one displaced step.
	(prepare_for_detach): Handle possibly multiple threads doing
	displaced steps.
	(handle_inferior_event): Handle possibility that fork event
	happens while another thread displaced steps.
	* linux-tdep.h (linux_displaced_step_prepare): New.
	(linux_displaced_step_finish): New.
	(linux_displaced_step_copy_insn_closure_by_addr): New.
	(linux_displaced_step_restore_all_in_ptid): New.
	(linux_init_abi): Add supports_displaced_step parameter.
	* linux-tdep.c (struct linux_info) <disp_step_buf>: New field.
	(linux_displaced_step_prepare): New.
	(linux_displaced_step_finish): New.
	(linux_displaced_step_copy_insn_closure_by_addr): New.
	(linux_displaced_step_restore_all_in_ptid): New.
	(linux_init_abi): Add supports_displaced_step parameter,
	register displaced step methods if true.
	(_initialize_linux_tdep): Register inferior_execd observer.
	* amd64-linux-tdep.c (amd64_linux_init_abi_common): Add
	supports_displaced_step parameter, adjust call to
	linux_init_abi.  Remove call to
	set_gdbarch_displaced_step_location.
	(amd64_linux_init_abi): Adjust call to
	amd64_linux_init_abi_common.
	(amd64_x32_linux_init_abi): Likewise.
	* aarch64-linux-tdep.c (aarch64_linux_init_abi): Adjust call to
	linux_init_abi.  Remove call to
	set_gdbarch_displaced_step_location.
	* arm-linux-tdep.c (arm_linux_init_abi): Likewise.
	* i386-linux-tdep.c (i386_linux_init_abi): Likewise.
	* alpha-linux-tdep.c (alpha_linux_init_abi): Adjust call to
	linux_init_abi.
	* arc-linux-tdep.c (arc_linux_init_osabi): Likewise.
	* bfin-linux-tdep.c (bfin_linux_init_abi): Likewise.
	* cris-linux-tdep.c (cris_linux_init_abi): Likewise.
	* csky-linux-tdep.c (csky_linux_init_abi): Likewise.
	* frv-linux-tdep.c (frv_linux_init_abi): Likewise.
	* hppa-linux-tdep.c (hppa_linux_init_abi): Likewise.
	* ia64-linux-tdep.c (ia64_linux_init_abi): Likewise.
	* m32r-linux-tdep.c (m32r_linux_init_abi): Likewise.
	* m68k-linux-tdep.c (m68k_linux_init_abi): Likewise.
	* microblaze-linux-tdep.c (microblaze_linux_init_abi): Likewise.
	* mips-linux-tdep.c (mips_linux_init_abi): Likewise.
	* mn10300-linux-tdep.c (am33_linux_init_osabi): Likewise.
	* nios2-linux-tdep.c (nios2_linux_init_abi): Likewise.
	* or1k-linux-tdep.c (or1k_linux_init_abi): Likewise.
	* riscv-linux-tdep.c (riscv_linux_init_abi): Likewise.
	* s390-linux-tdep.c (s390_linux_init_abi_any): Likewise.
	* sh-linux-tdep.c (sh_linux_init_abi): Likewise.
	* sparc-linux-tdep.c (sparc32_linux_init_abi): Likewise.
	* sparc64-linux-tdep.c (sparc64_linux_init_abi): Likewise.
	* tic6x-linux-tdep.c (tic6x_uclinux_init_abi): Likewise.
	* tilegx-linux-tdep.c (tilegx_linux_init_abi): Likewise.
	* xtensa-linux-tdep.c (xtensa_linux_init_abi): Likewise.
	* ppc-linux-tdep.c (ppc_linux_init_abi): Adjust call to
	linux_init_abi.  Remove call to
	set_gdbarch_displaced_step_location.
	* arm-tdep.c (arm_pc_is_thumb): Call
	gdbarch_displaced_step_copy_insn_closure_by_addr instead of
	get_displaced_step_copy_insn_closure_by_addr.
	* rs6000-aix-tdep.c (rs6000_aix_init_osabi): Adjust calls to
	clear gdbarch methods.
	* rs6000-tdep.c (struct ppc_inferior_data): New structure.
	(get_ppc_per_inferior): New function.
	(ppc_displaced_step_prepare): New function.
	(ppc_displaced_step_finish): New function.
	(ppc_displaced_step_restore_all_in_ptid): New function.
	(rs6000_gdbarch_init): Register new gdbarch methods.
	* s390-tdep.c (s390_gdbarch_init): Don't call
	set_gdbarch_displaced_step_location, set new gdbarch methods.

gdb/testsuite/ChangeLog:

	* gdb.arch/amd64-disp-step-avx.exp: Adjust pattern.
	* gdb.threads/forking-threads-plus-breakpoint.exp: Likewise.
	* gdb.threads/non-stop-fair-events.exp: Likewise.

Change-Id: I387cd235a442d0620ec43608fd3dc0097fcbf8c8

48 files changed, 1010 insertions, 307 deletions

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index 8512cb449e9..f39730fa585 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,5 +1,128 @@
 2020-12-04  Simon Marchi  <simon.marchi@efficios.com>
 
+	* displaced-stepping.h (struct
+	displaced_step_copy_insn_closure): Adjust comments.
+	(struct displaced_step_inferior_state) <step_thread,
+	step_gdbarch, step_closure, step_original, step_copy,
+	step_saved_copy>: Remove fields.
+	(struct displaced_step_thread_state): New.
+	(struct displaced_step_buffer): New.
+	* displaced-stepping.c (displaced_step_buffer::prepare): New.
+	(write_memory_ptid): Move from infrun.c.
+	(displaced_step_instruction_executed_successfully): New,
+	factored out of displaced_step_finish.
+	(displaced_step_buffer::finish): New.
+	(displaced_step_buffer::copy_insn_closure_by_addr): New.
+	(displaced_step_buffer::restore_in_ptid): New.
+	* gdbarch.sh (displaced_step_location): Remove.
+	(displaced_step_prepare, displaced_step_finish,
+	displaced_step_copy_insn_closure_by_addr,
+	displaced_step_restore_all_in_ptid): New.
+	* gdbarch.c: Re-generate.
+	* gdbarch.h: Re-generate.
+	* gdbthread.h (class thread_info) <displaced_step_state>: New
+	field.
+	(thread_step_over_chain_remove): New declaration.
+	(thread_step_over_chain_next): New declaration.
+	(thread_step_over_chain_length): New declaration.
+	* thread.c (thread_step_over_chain_remove): Make non-static.
+	(thread_step_over_chain_next): New.
+	(global_thread_step_over_chain_next): Use
+	thread_step_over_chain_next.
+	(thread_step_over_chain_length): New.
+	(global_thread_step_over_chain_enqueue): Add debug print.
+	(global_thread_step_over_chain_remove): Add debug print.
+	* infrun.h (get_displaced_step_copy_insn_closure_by_addr):
+	Remove.
+	* infrun.c (get_displaced_stepping_state): New.
+	(displaced_step_in_progress_any_inferior): Remove.
+	(displaced_step_in_progress_thread): Adjust.
+	(displaced_step_in_progress): Adjust.
+	(displaced_step_in_progress_any_thread): New.
+	(get_displaced_step_copy_insn_closure_by_addr): Remove.
+	(gdbarch_supports_displaced_stepping): Use
+	gdbarch_displaced_step_prepare_p.
+	(displaced_step_reset): Change parameter from inferior to
+	thread.
+	(displaced_step_prepare_throw): Implement using
+	gdbarch_displaced_step_prepare.
+	(write_memory_ptid): Move to displaced-step.c.
+	(displaced_step_restore): Remove.
+	(displaced_step_finish): Implement using
+	gdbarch_displaced_step_finish.
+	(start_step_over): Allow starting more than one displaced step.
+	(prepare_for_detach): Handle possibly multiple threads doing
+	displaced steps.
+	(handle_inferior_event): Handle possibility that fork event
+	happens while another thread displaced steps.
+	* linux-tdep.h (linux_displaced_step_prepare): New.
+	(linux_displaced_step_finish): New.
+	(linux_displaced_step_copy_insn_closure_by_addr): New.
+	(linux_displaced_step_restore_all_in_ptid): New.
+	(linux_init_abi): Add supports_displaced_step parameter.
+	* linux-tdep.c (struct linux_info) <disp_step_buf>: New field.
+	(linux_displaced_step_prepare): New.
+	(linux_displaced_step_finish): New.
+	(linux_displaced_step_copy_insn_closure_by_addr): New.
+	(linux_displaced_step_restore_all_in_ptid): New.
+	(linux_init_abi): Add supports_displaced_step parameter,
+	register displaced step methods if true.
+	(_initialize_linux_tdep): Register inferior_execd observer.
+	* amd64-linux-tdep.c (amd64_linux_init_abi_common): Add
+	supports_displaced_step parameter, adjust call to
+	linux_init_abi.  Remove call to
+	set_gdbarch_displaced_step_location.
+	(amd64_linux_init_abi): Adjust call to
+	amd64_linux_init_abi_common.
+	(amd64_x32_linux_init_abi): Likewise.
+	* aarch64-linux-tdep.c (aarch64_linux_init_abi): Adjust call to
+	linux_init_abi.  Remove call to
+	set_gdbarch_displaced_step_location.
+	* arm-linux-tdep.c (arm_linux_init_abi): Likewise.
+	* i386-linux-tdep.c (i386_linux_init_abi): Likewise.
+	* alpha-linux-tdep.c (alpha_linux_init_abi): Adjust call to
+	linux_init_abi.
+	* arc-linux-tdep.c (arc_linux_init_osabi): Likewise.
+	* bfin-linux-tdep.c (bfin_linux_init_abi): Likewise.
+	* cris-linux-tdep.c (cris_linux_init_abi): Likewise.
+	* csky-linux-tdep.c (csky_linux_init_abi): Likewise.
+	* frv-linux-tdep.c (frv_linux_init_abi): Likewise.
+	* hppa-linux-tdep.c (hppa_linux_init_abi): Likewise.
+	* ia64-linux-tdep.c (ia64_linux_init_abi): Likewise.
+	* m32r-linux-tdep.c (m32r_linux_init_abi): Likewise.
+	* m68k-linux-tdep.c (m68k_linux_init_abi): Likewise.
+	* microblaze-linux-tdep.c (microblaze_linux_init_abi): Likewise.
+	* mips-linux-tdep.c (mips_linux_init_abi): Likewise.
+	* mn10300-linux-tdep.c (am33_linux_init_osabi): Likewise.
+	* nios2-linux-tdep.c (nios2_linux_init_abi): Likewise.
+	* or1k-linux-tdep.c (or1k_linux_init_abi): Likewise.
+	* riscv-linux-tdep.c (riscv_linux_init_abi): Likewise.
+	* s390-linux-tdep.c (s390_linux_init_abi_any): Likewise.
+	* sh-linux-tdep.c (sh_linux_init_abi): Likewise.
+	* sparc-linux-tdep.c (sparc32_linux_init_abi): Likewise.
+	* sparc64-linux-tdep.c (sparc64_linux_init_abi): Likewise.
+	* tic6x-linux-tdep.c (tic6x_uclinux_init_abi): Likewise.
+	* tilegx-linux-tdep.c (tilegx_linux_init_abi): Likewise.
+	* xtensa-linux-tdep.c (xtensa_linux_init_abi): Likewise.
+	* ppc-linux-tdep.c (ppc_linux_init_abi): Adjust call to
+	linux_init_abi.  Remove call to
+	set_gdbarch_displaced_step_location.
+	* arm-tdep.c (arm_pc_is_thumb): Call
+	gdbarch_displaced_step_copy_insn_closure_by_addr instead of
+	get_displaced_step_copy_insn_closure_by_addr.
+	* rs6000-aix-tdep.c (rs6000_aix_init_osabi): Adjust calls to
+	clear gdbarch methods.
+	* rs6000-tdep.c (struct ppc_inferior_data): New structure.
+	(get_ppc_per_inferior): New function.
+	(ppc_displaced_step_prepare): New function.
+	(ppc_displaced_step_finish): New function.
+	(ppc_displaced_step_restore_all_in_ptid): New function.
+	(rs6000_gdbarch_init): Register new gdbarch methods.
+	* s390-tdep.c (s390_gdbarch_init): Don't call
+	set_gdbarch_displaced_step_location, set new gdbarch methods.
+
+2020-12-04  Simon Marchi  <simon.marchi@efficios.com>
+
 	* Makefile.in (COMMON_SFILES): Add displaced-stepping.c.
 	* aarch64-tdep.h: Include displaced-stepping.h.
 	* displaced-stepping.h (struct displaced_step_copy_insn_closure):
diff --git a/gdb/aarch64-linux-tdep.c b/gdb/aarch64-linux-tdep.c
index c9898bdafda..4fe7babe59c 100644
--- a/gdb/aarch64-linux-tdep.c
+++ b/gdb/aarch64-linux-tdep.c
@@ -1445,7 +1445,7 @@ aarch64_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 
   tdep->lowest_pc = 0x8000;
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, true);
 
   set_solib_svr4_fetch_link_map_offsets (gdbarch,
 					 svr4_lp64_fetch_link_map_offsets);
@@ -1658,7 +1658,6 @@ aarch64_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
   set_gdbarch_displaced_step_copy_insn (gdbarch,
 					aarch64_displaced_step_copy_insn);
   set_gdbarch_displaced_step_fixup (gdbarch, aarch64_displaced_step_fixup);
-  set_gdbarch_displaced_step_location (gdbarch, linux_displaced_step_location);
   set_gdbarch_displaced_step_hw_singlestep (gdbarch,
 					    aarch64_displaced_step_hw_singlestep);
 
diff --git a/gdb/alpha-linux-tdep.c b/gdb/alpha-linux-tdep.c
index 70ac5a87766..a6d6b15e9fd 100644
--- a/gdb/alpha-linux-tdep.c
+++ b/gdb/alpha-linux-tdep.c
@@ -356,7 +356,7 @@ alpha_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   struct gdbarch_tdep *tdep;
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Hook into the DWARF CFI frame unwinder.  */
   alpha_dwarf2_init_abi (info, gdbarch);
diff --git a/gdb/amd64-linux-tdep.c b/gdb/amd64-linux-tdep.c
index d484b1a1c59..a81bb9039df 100644
--- a/gdb/amd64-linux-tdep.c
+++ b/gdb/amd64-linux-tdep.c
@@ -1795,11 +1795,12 @@ amd64_dtrace_parse_probe_argument (struct gdbarch *gdbarch,
 }
 
 static void
-amd64_linux_init_abi_common(struct gdbarch_info info, struct gdbarch *gdbarch)
+amd64_linux_init_abi_common(struct gdbarch_info info, struct gdbarch *gdbarch,
+			    bool supports_displaced_step)
 {
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, supports_displaced_step);
 
   tdep->sigtramp_p = amd64_linux_sigtramp_p;
   tdep->sigcontext_addr = amd64_linux_sigcontext_addr;
@@ -1839,8 +1840,6 @@ amd64_linux_init_abi_common(struct gdbarch_info info, struct gdbarch *gdbarch)
   set_gdbarch_displaced_step_copy_insn (gdbarch,
 					amd64_displaced_step_copy_insn);
   set_gdbarch_displaced_step_fixup (gdbarch, amd64_displaced_step_fixup);
-  set_gdbarch_displaced_step_location (gdbarch,
-				       linux_displaced_step_location);
 
   set_gdbarch_process_record (gdbarch, i386_process_record);
   set_gdbarch_process_record_signal (gdbarch, amd64_linux_record_signal);
@@ -1881,7 +1880,7 @@ amd64_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
   if (!valid_p)
     return;
 
-  amd64_linux_init_abi_common (info, gdbarch);
+  amd64_linux_init_abi_common (info, gdbarch, true);
 
   /* Initialize the amd64_linux_record_tdep.  */
   /* These values are the size of the type that will be used in a system
@@ -2096,7 +2095,7 @@ amd64_x32_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
   if (!valid_p)
     return;
 
-  amd64_linux_init_abi_common (info, gdbarch);
+  amd64_linux_init_abi_common (info, gdbarch, false);
 
   /* Initialize the amd64_x32_linux_record_tdep.  */
   /* These values are the size of the type that will be used in a system
diff --git a/gdb/arc-linux-tdep.c b/gdb/arc-linux-tdep.c
index ef459bf596f..b919882177e 100644
--- a/gdb/arc-linux-tdep.c
+++ b/gdb/arc-linux-tdep.c
@@ -439,7 +439,7 @@ arc_linux_init_osabi (struct gdbarch_info info, struct gdbarch *gdbarch)
    */
   tdep->jb_pc = 15;
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Set up target dependent GDB architecture entries.  */
   set_gdbarch_cannot_fetch_register (gdbarch, arc_linux_cannot_fetch_register);
diff --git a/gdb/arm-linux-tdep.c b/gdb/arm-linux-tdep.c
index 9caae06adfe..d164cff3dff 100644
--- a/gdb/arm-linux-tdep.c
+++ b/gdb/arm-linux-tdep.c
@@ -1721,7 +1721,7 @@ arm_linux_init_abi (struct gdbarch_info info,
 								    NULL };
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, true);
 
   tdep->lowest_pc = 0x8000;
   if (info.byte_order_for_code == BFD_ENDIAN_BIG)
@@ -1807,7 +1807,6 @@ arm_linux_init_abi (struct gdbarch_info info,
   set_gdbarch_displaced_step_copy_insn (gdbarch,
 					arm_linux_displaced_step_copy_insn);
   set_gdbarch_displaced_step_fixup (gdbarch, arm_displaced_step_fixup);
-  set_gdbarch_displaced_step_location (gdbarch, linux_displaced_step_location);
 
   /* Reversible debugging, process record.  */
   set_gdbarch_process_record (gdbarch, arm_process_record);
diff --git a/gdb/arm-tdep.c b/gdb/arm-tdep.c
index bc086e18a57..7d96884695c 100644
--- a/gdb/arm-tdep.c
+++ b/gdb/arm-tdep.c
@@ -433,9 +433,12 @@ arm_pc_is_thumb (struct gdbarch *gdbarch, CORE_ADDR memaddr)
 {
   struct bound_minimal_symbol sym;
   char type;
-  arm_displaced_step_copy_insn_closure *dsc
-    = ((arm_displaced_step_copy_insn_closure * )
-	get_displaced_step_copy_insn_closure_by_addr (memaddr));
+  arm_displaced_step_copy_insn_closure *dsc = nullptr;
+
+  if (gdbarch_displaced_step_copy_insn_closure_by_addr_p (gdbarch))
+    dsc = ((arm_displaced_step_copy_insn_closure * )
+	   gdbarch_displaced_step_copy_insn_closure_by_addr
+	     (gdbarch, current_inferior (), memaddr));
 
   /* If checking the mode of displaced instruction in copy area, the mode
      should be determined by instruction on the original address.  */
diff --git a/gdb/bfin-linux-tdep.c b/gdb/bfin-linux-tdep.c
index 16a0a97eaf4..fc2f1d9ac65 100644
--- a/gdb/bfin-linux-tdep.c
+++ b/gdb/bfin-linux-tdep.c
@@ -150,7 +150,7 @@ bfin_linux_get_syscall_number (struct gdbarch *gdbarch,
 static void
 bfin_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Set the sigtramp frame sniffer.  */
   tramp_frame_prepend_unwinder (gdbarch, &bfin_linux_sigframe);
diff --git a/gdb/cris-linux-tdep.c b/gdb/cris-linux-tdep.c
index 535dc4a1fce..85cbf4cc093 100644
--- a/gdb/cris-linux-tdep.c
+++ b/gdb/cris-linux-tdep.c
@@ -35,7 +35,7 @@ cris_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   if (tdep->cris_version == 32)
     /* Threaded debugging is only supported on CRISv32 for now.  */
diff --git a/gdb/csky-linux-tdep.c b/gdb/csky-linux-tdep.c
index fb431020446..184fa5ffb23 100644
--- a/gdb/csky-linux-tdep.c
+++ b/gdb/csky-linux-tdep.c
@@ -233,7 +233,7 @@ csky_linux_rt_sigreturn_tramp_frame = {
 static void
 csky_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Shared library handling.  */
   set_gdbarch_skip_trampoline_code (gdbarch, find_solib_trampoline_target);
diff --git a/gdb/displaced-stepping.c b/gdb/displaced-stepping.c
index 5ae280fac39..fb5d23f92c1 100644
--- a/gdb/displaced-stepping.c
+++ b/gdb/displaced-stepping.c
@@ -19,9 +19,15 @@
 
 #include "defs.h"
 #include "displaced-stepping.h"
-#include "cli/cli-cmds.h"
 
+#include "cli/cli-cmds.h"
 #include "command.h"
+#include "gdbarch.h"
+#include "gdbcore.h"
+#include "gdbthread.h"
+#include "inferior.h"
+#include "regcache.h"
+#include "target/target.h"
 
 /* Default destructor for displaced_step_copy_insn_closure.  */
 
@@ -37,6 +43,193 @@ show_debug_displaced (struct ui_file *file, int from_tty,
   fprintf_filtered (file, _("Displace stepping debugging is %s.\n"), value);
 }
 
+displaced_step_prepare_status
+displaced_step_buffer::prepare (thread_info *thread, CORE_ADDR &displaced_pc)
+{
+  gdb_assert (!thread->displaced_step_state.in_progress ());
+
+  /* Is a thread currently using the buffer?  */
+  if (m_current_thread != nullptr)
+    {
+      /* If so, it better not be this thread.  */
+      gdb_assert (thread != m_current_thread);
+      return DISPLACED_STEP_PREPARE_STATUS_UNAVAILABLE;
+    }
+
+  regcache *regcache = get_thread_regcache (thread);
+  const address_space *aspace = regcache->aspace ();
+  gdbarch *arch = regcache->arch ();
+  ULONGEST len = gdbarch_max_insn_length (arch);
+
+  if (breakpoint_in_range_p (aspace, m_addr, len))
+    {
+      /* There's a breakpoint set in the scratch pad location range
+	 (which is usually around the entry point).  We'd either
+	 install it before resuming, which would overwrite/corrupt the
+	 scratch pad, or if it was already inserted, this displaced
+	 step would overwrite it.  The latter is OK in the sense that
+	 we already assume that no thread is going to execute the code
+	 in the scratch pad range (after initial startup) anyway, but
+	 the former is unacceptable.  Simply punt and fallback to
+	 stepping over this breakpoint in-line.  */
+      displaced_debug_printf ("breakpoint set in scratch pad.  "
+			      "Stepping over breakpoint in-line instead.");
+
+      return DISPLACED_STEP_PREPARE_STATUS_CANT;
+    }
+
+  m_original_pc = regcache_read_pc (regcache);
+  displaced_pc = m_addr;
+
+  /* Save the original contents of the displaced stepping buffer.  */
+  m_saved_copy.resize (len);
+
+  int status = target_read_memory (m_addr, m_saved_copy.data (), len);
+  if (status != 0)
+    throw_error (MEMORY_ERROR,
+		 _("Error accessing memory address %s (%s) for "
+		   "displaced-stepping scratch space."),
+		 paddress (arch, m_addr), safe_strerror (status));
+
+  displaced_debug_printf ("saved %s: %s",
+			  paddress (arch, m_addr),
+			  displaced_step_dump_bytes
+			    (m_saved_copy.data (), len).c_str ());
+
+  /* Save this in a local variable first, so it's released if code below
+     throws.  */
+  displaced_step_copy_insn_closure_up copy_insn_closure
+    = gdbarch_displaced_step_copy_insn (arch, m_original_pc, m_addr, regcache);
+
+  if (copy_insn_closure == nullptr)
+    {
+      /* The architecture doesn't know how or want to displaced step
+        this instruction or instruction sequence.  Fallback to
+        stepping over the breakpoint in-line.  */
+      return DISPLACED_STEP_PREPARE_STATUS_CANT;
+    }
+
+  /* Resume execution at the copy.  */
+  regcache_write_pc (regcache, m_addr);
+
+  /* This marks the buffer as being in use.  */
+  m_current_thread = thread;
+
+  /* Save this, now that we know everything went fine.  */
+  m_copy_insn_closure = std::move (copy_insn_closure);
+
+  /* Tell infrun not to try preparing a displaced step again for this inferior.  */
+  thread->inf->displaced_step_state.unavailable = true;
+
+  return DISPLACED_STEP_PREPARE_STATUS_OK;
+}
+
+static void
+write_memory_ptid (ptid_t ptid, CORE_ADDR memaddr,
+		   const gdb_byte *myaddr, int len)
+{
+  scoped_restore save_inferior_ptid = make_scoped_restore (&inferior_ptid);
+
+  inferior_ptid = ptid;
+  write_memory (memaddr, myaddr, len);
+}
+
+static bool
+displaced_step_instruction_executed_successfully (gdbarch *arch,
+						  gdb_signal signal)
+{
+  if (signal != GDB_SIGNAL_TRAP)
+    return false;
+
+  if (target_stopped_by_watchpoint ())
+    {
+      if (gdbarch_have_nonsteppable_watchpoint (arch)
+	  || target_have_steppable_watchpoint ())
+	return false;
+    }
+
+  return true;
+}
+
+displaced_step_finish_status
+displaced_step_buffer::finish (gdbarch *arch, thread_info *thread,
+			       gdb_signal sig)
+{
+  gdb_assert (thread->displaced_step_state.in_progress ());
+  gdb_assert (thread == m_current_thread);
+
+  /* Move this to a local variable so it's released in case something goes
+     wrong.  */
+  displaced_step_copy_insn_closure_up copy_insn_closure
+    = std::move (m_copy_insn_closure);
+  gdb_assert (copy_insn_closure != nullptr);
+
+  /* Reset M_CURRENT_THREAD immediately to mark the buffer as available, in case
+     something goes wrong below.  */
+  m_current_thread = nullptr;
+
+  /* Now that a buffer gets freed, tell infrun it can ask us to prepare a displaced
+     step again for this inferior.  Do that here in case something goes wrong
+     below.  */
+  thread->inf->displaced_step_state.unavailable = false;
+
+  ULONGEST len = gdbarch_max_insn_length (arch);
+
+  write_memory_ptid (thread->ptid, m_addr,
+		     m_saved_copy.data (), len);
+
+  displaced_debug_printf ("restored %s %s",
+			  target_pid_to_str (thread->ptid).c_str (),
+			  paddress (arch, m_addr));
+
+  regcache *rc = get_thread_regcache (thread);
+
+  bool instruction_executed_successfully
+    = displaced_step_instruction_executed_successfully (arch, sig);
+
+  if (instruction_executed_successfully)
+    {
+      gdbarch_displaced_step_fixup (arch, copy_insn_closure.get (), m_original_pc,
+				    m_addr, rc);
+      return DISPLACED_STEP_FINISH_STATUS_OK;
+    }
+  else
+    {
+      /* Since the instruction didn't complete, all we can do is relocate the
+	 PC.  */
+      CORE_ADDR pc = regcache_read_pc (rc);
+      pc = m_original_pc + (pc - m_addr);
+      regcache_write_pc (rc, pc);
+      return DISPLACED_STEP_FINISH_STATUS_NOT_EXECUTED;
+    }
+}
+
+const displaced_step_copy_insn_closure *
+displaced_step_buffer::copy_insn_closure_by_addr (CORE_ADDR addr)
+{
+  if (addr == m_addr)
+    return m_copy_insn_closure.get ();
+  else
+    return nullptr;
+}
+
+void
+displaced_step_buffer::restore_in_ptid (ptid_t ptid)
+{
+  if (m_current_thread != nullptr)
+    {
+      regcache *regcache = get_thread_regcache (m_current_thread);
+      gdbarch *arch = regcache->arch ();
+      ULONGEST len = gdbarch_max_insn_length (arch);
+
+      write_memory_ptid (ptid, m_addr, m_saved_copy.data (), len);
+
+      displaced_debug_printf ("restored in ptid %s %s",
+			      target_pid_to_str (ptid).c_str (),
+			      paddress (arch, m_addr));
+    }
+}
+
 void _initialize_displaced_stepping ();
 void
 _initialize_displaced_stepping ()
diff --git a/gdb/displaced-stepping.h b/gdb/displaced-stepping.h
index 928bfc0ccc9..a2d80aeda4f 100644
--- a/gdb/displaced-stepping.h
+++ b/gdb/displaced-stepping.h
@@ -22,6 +22,7 @@
 
 #include "gdbsupport/byte-vector.h"
 
+struct gdbarch;
 struct thread_info;
 
 /* True if we are debugging displaced stepping.  */
@@ -62,7 +63,8 @@ enum displaced_step_finish_status
   DISPLACED_STEP_FINISH_STATUS_NOT_EXECUTED,
 };
 
-/* Base class for displaced stepping closures (the arch-specific data).  */
+/* Data returned by a gdbarch displaced_step_copy_insn method, to be passed to
+   the matching displaced_step_fixup method.  */
 
 struct displaced_step_copy_insn_closure
 {
@@ -80,6 +82,9 @@ struct buf_displaced_step_copy_insn_closure : displaced_step_copy_insn_closure
   : buf (buf_size)
   {}
 
+  /* The content of this buffer is up to the user of the class, but typically
+     original instruction bytes, used during fixup to determine what needs to
+     be fixed up.  */
   gdb::byte_vector buf;
 };
 
@@ -95,37 +100,95 @@ struct displaced_step_inferior_state
   /* Put this object back in its original state.  */
   void reset ()
   {
-    failed_before = 0;
-    step_thread = nullptr;
-    step_gdbarch = nullptr;
-    step_closure.reset ();
-    step_original = 0;
-    step_copy = 0;
-    step_saved_copy.clear ();
+    failed_before = false;
+    in_progress_count = 0;
+    unavailable = false;
   }
 
   /* True if preparing a displaced step ever failed.  If so, we won't
      try displaced stepping for this inferior again.  */
-  int failed_before;
+  bool failed_before;
 
-  /* If this is not nullptr, this is the thread carrying out a
-     displaced single-step in process PID.  This thread's state will
-     require fixing up once it has completed its step.  */
-  thread_info *step_thread;
+  /* Number of displaced steps in progress for this inferior.  */
+  unsigned int in_progress_count;
 
-  /* The architecture the thread had when we stepped it.  */
-  gdbarch *step_gdbarch;
+  /* If true, this tells GDB that it's not worth asking the gdbarch displaced
+     stepping implementation to prepare a displaced step, because it would
+     return UNAVAILABLE.  This is set and reset by the gdbarch in the
+     displaced_step_prepare and displaced_step_finish methods.  */
+  bool unavailable;
+};
 
-  /* The closure provided gdbarch_displaced_step_copy_insn, to be used
-     for post-step cleanup.  */
-  displaced_step_copy_insn_closure_up step_closure;
+/* Per-thread displaced stepping state.  */
 
-  /* The address of the original instruction, and the copy we
-     made.  */
-  CORE_ADDR step_original, step_copy;
+struct displaced_step_thread_state
+{
+  /* Return true if this thread is currently executing a displaced step.  */
+  bool in_progress () const
+  {
+    return m_original_gdbarch != nullptr;
+  }
+
+  /* Return the gdbarch of the thread prior to the step.  */
+  gdbarch *get_original_gdbarch () const
+  {
+    return m_original_gdbarch;
+  }
+
+  /* Mark this thread as currently executing a displaced step.
+
+     ORIGINAL_GDBARCH is the current gdbarch of the thread (before the step
+     is executed).  */
+  void set (gdbarch *original_gdbarch)
+  {
+    m_original_gdbarch = original_gdbarch;
+  }
+
+  /* Mark this thread as no longer executing a displaced step.  */
+  void reset ()
+  {
+    m_original_gdbarch = nullptr;
+  }
+
+private:
+  gdbarch *m_original_gdbarch = nullptr;
+};
+
+/* Manage access to a single displaced stepping buffer.  */
+
+struct displaced_step_buffer
+{
+  explicit displaced_step_buffer (CORE_ADDR buffer_addr)
+    : m_addr (buffer_addr)
+  {}
+
+  displaced_step_prepare_status prepare (thread_info *thread,
+					 CORE_ADDR &displaced_pc);
+
+  displaced_step_finish_status finish (gdbarch *arch, thread_info *thread,
+				       gdb_signal sig);
+
+  const displaced_step_copy_insn_closure *
+    copy_insn_closure_by_addr (CORE_ADDR addr);
+
+  void restore_in_ptid (ptid_t ptid);
+
+private:
+  /* Original PC of the instruction being displaced-stepped in this buffer.  */
+  CORE_ADDR m_original_pc = 0;
+
+  /* Address of the buffer.  */
+  const CORE_ADDR m_addr;
+
+  /* If set, the thread currently using the buffer.  */
+  thread_info *m_current_thread = nullptr;
 
   /* Saved contents of copy area.  */
-  gdb::byte_vector step_saved_copy;
+  gdb::byte_vector m_saved_copy;
+
+  /* The closure provided gdbarch_displaced_step_copy_insn, to be used
+     for post-step cleanup.  */
+  displaced_step_copy_insn_closure_up m_copy_insn_closure;
 };
 
 #endif /* DISPLACED_STEPPING_H */
diff --git a/gdb/frv-linux-tdep.c b/gdb/frv-linux-tdep.c
index 2dc18206975..c5ae4212ab4 100644
--- a/gdb/frv-linux-tdep.c
+++ b/gdb/frv-linux-tdep.c
@@ -456,7 +456,7 @@ frv_linux_iterate_over_regset_sections (struct gdbarch *gdbarch,
 static void
 frv_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Set the sigtramp frame sniffer.  */
   frame_unwind_append_unwinder (gdbarch, &frv_linux_sigtramp_frame_unwind); 
diff --git a/gdb/gdbarch.c b/gdb/gdbarch.c
index 0ec626c6f8c..bf8d8bc211f 100644
--- a/gdb/gdbarch.c
+++ b/gdb/gdbarch.c
@@ -290,7 +290,10 @@ struct gdbarch
   gdbarch_displaced_step_copy_insn_ftype *displaced_step_copy_insn;
   gdbarch_displaced_step_hw_singlestep_ftype *displaced_step_hw_singlestep;
   gdbarch_displaced_step_fixup_ftype *displaced_step_fixup;
-  gdbarch_displaced_step_location_ftype *displaced_step_location;
+  gdbarch_displaced_step_prepare_ftype *displaced_step_prepare;
+  gdbarch_displaced_step_finish_ftype *displaced_step_finish;
+  gdbarch_displaced_step_copy_insn_closure_by_addr_ftype *displaced_step_copy_insn_closure_by_addr;
+  gdbarch_displaced_step_restore_all_in_ptid_ftype *displaced_step_restore_all_in_ptid;
   gdbarch_relocate_instruction_ftype *relocate_instruction;
   gdbarch_overlay_update_ftype *overlay_update;
   gdbarch_core_read_description_ftype *core_read_description;
@@ -445,7 +448,7 @@ gdbarch_alloc (const struct gdbarch_info *info,
   gdbarch->skip_permanent_breakpoint = default_skip_permanent_breakpoint;
   gdbarch->displaced_step_hw_singlestep = default_displaced_step_hw_singlestep;
   gdbarch->displaced_step_fixup = NULL;
-  gdbarch->displaced_step_location = NULL;
+  gdbarch->displaced_step_finish = NULL;
   gdbarch->relocate_instruction = NULL;
   gdbarch->has_shared_address_space = default_has_shared_address_space;
   gdbarch->fast_tracepoint_valid_at = default_fast_tracepoint_valid_at;
@@ -660,8 +663,10 @@ verify_gdbarch (struct gdbarch *gdbarch)
   /* Skip verify of displaced_step_copy_insn, has predicate.  */
   /* Skip verify of displaced_step_hw_singlestep, invalid_p == 0 */
   /* Skip verify of displaced_step_fixup, has predicate.  */
-  if ((! gdbarch->displaced_step_location) != (! gdbarch->displaced_step_copy_insn))
-    log.puts ("\n\tdisplaced_step_location");
+  /* Skip verify of displaced_step_prepare, has predicate.  */
+  if ((! gdbarch->displaced_step_finish) != (! gdbarch->displaced_step_prepare))
+    log.puts ("\n\tdisplaced_step_finish");
+  /* Skip verify of displaced_step_copy_insn_closure_by_addr, has predicate.  */
   /* Skip verify of relocate_instruction, has predicate.  */
   /* Skip verify of overlay_update, has predicate.  */
   /* Skip verify of core_read_description, has predicate.  */
@@ -925,6 +930,15 @@ gdbarch_dump (struct gdbarch *gdbarch, struct ui_file *file)
                       "gdbarch_dump: displaced_step_copy_insn = <%s>\n",
                       host_address_to_string (gdbarch->displaced_step_copy_insn));
   fprintf_unfiltered (file,
+                      "gdbarch_dump: gdbarch_displaced_step_copy_insn_closure_by_addr_p() = %d\n",
+                      gdbarch_displaced_step_copy_insn_closure_by_addr_p (gdbarch));
+  fprintf_unfiltered (file,
+                      "gdbarch_dump: displaced_step_copy_insn_closure_by_addr = <%s>\n",
+                      host_address_to_string (gdbarch->displaced_step_copy_insn_closure_by_addr));
+  fprintf_unfiltered (file,
+                      "gdbarch_dump: displaced_step_finish = <%s>\n",
+                      host_address_to_string (gdbarch->displaced_step_finish));
+  fprintf_unfiltered (file,
                       "gdbarch_dump: gdbarch_displaced_step_fixup_p() = %d\n",
                       gdbarch_displaced_step_fixup_p (gdbarch));
   fprintf_unfiltered (file,
@@ -934,8 +948,14 @@ gdbarch_dump (struct gdbarch *gdbarch, struct ui_file *file)
                       "gdbarch_dump: displaced_step_hw_singlestep = <%s>\n",
                       host_address_to_string (gdbarch->displaced_step_hw_singlestep));
   fprintf_unfiltered (file,
-                      "gdbarch_dump: displaced_step_location = <%s>\n",
-                      host_address_to_string (gdbarch->displaced_step_location));
+                      "gdbarch_dump: gdbarch_displaced_step_prepare_p() = %d\n",
+                      gdbarch_displaced_step_prepare_p (gdbarch));
+  fprintf_unfiltered (file,
+                      "gdbarch_dump: displaced_step_prepare = <%s>\n",
+                      host_address_to_string (gdbarch->displaced_step_prepare));
+  fprintf_unfiltered (file,
+                      "gdbarch_dump: displaced_step_restore_all_in_ptid = <%s>\n",
+                      host_address_to_string (gdbarch->displaced_step_restore_all_in_ptid));
   fprintf_unfiltered (file,
                       "gdbarch_dump: double_bit = %s\n",
                       plongest (gdbarch->double_bit));
@@ -4028,21 +4048,86 @@ set_gdbarch_displaced_step_fixup (struct gdbarch *gdbarch,
   gdbarch->displaced_step_fixup = displaced_step_fixup;
 }
 
-CORE_ADDR
-gdbarch_displaced_step_location (struct gdbarch *gdbarch)
+bool
+gdbarch_displaced_step_prepare_p (struct gdbarch *gdbarch)
+{
+  gdb_assert (gdbarch != NULL);
+  return gdbarch->displaced_step_prepare != NULL;
+}
+
+displaced_step_prepare_status
+gdbarch_displaced_step_prepare (struct gdbarch *gdbarch, thread_info *thread, CORE_ADDR &displaced_pc)
+{
+  gdb_assert (gdbarch != NULL);
+  gdb_assert (gdbarch->displaced_step_prepare != NULL);
+  if (gdbarch_debug >= 2)
+    fprintf_unfiltered (gdb_stdlog, "gdbarch_displaced_step_prepare called\n");
+  return gdbarch->displaced_step_prepare (gdbarch, thread, displaced_pc);
+}
+
+void
+set_gdbarch_displaced_step_prepare (struct gdbarch *gdbarch,
+                                    gdbarch_displaced_step_prepare_ftype displaced_step_prepare)
+{
+  gdbarch->displaced_step_prepare = displaced_step_prepare;
+}
+
+displaced_step_finish_status
+gdbarch_displaced_step_finish (struct gdbarch *gdbarch, thread_info *thread, gdb_signal sig)
+{
+  gdb_assert (gdbarch != NULL);
+  gdb_assert (gdbarch->displaced_step_finish != NULL);
+  if (gdbarch_debug >= 2)
+    fprintf_unfiltered (gdb_stdlog, "gdbarch_displaced_step_finish called\n");
+  return gdbarch->displaced_step_finish (gdbarch, thread, sig);
+}
+
+void
+set_gdbarch_displaced_step_finish (struct gdbarch *gdbarch,
+                                   gdbarch_displaced_step_finish_ftype displaced_step_finish)
+{
+  gdbarch->displaced_step_finish = displaced_step_finish;
+}
+
+bool
+gdbarch_displaced_step_copy_insn_closure_by_addr_p (struct gdbarch *gdbarch)
+{
+  gdb_assert (gdbarch != NULL);
+  return gdbarch->displaced_step_copy_insn_closure_by_addr != NULL;
+}
+
+const displaced_step_copy_insn_closure *
+gdbarch_displaced_step_copy_insn_closure_by_addr (struct gdbarch *gdbarch, inferior *inf, CORE_ADDR addr)
+{
+  gdb_assert (gdbarch != NULL);
+  gdb_assert (gdbarch->displaced_step_copy_insn_closure_by_addr != NULL);
+  if (gdbarch_debug >= 2)
+    fprintf_unfiltered (gdb_stdlog, "gdbarch_displaced_step_copy_insn_closure_by_addr called\n");
+  return gdbarch->displaced_step_copy_insn_closure_by_addr (inf, addr);
+}
+
+void
+set_gdbarch_displaced_step_copy_insn_closure_by_addr (struct gdbarch *gdbarch,
+                                                      gdbarch_displaced_step_copy_insn_closure_by_addr_ftype displaced_step_copy_insn_closure_by_addr)
+{
+  gdbarch->displaced_step_copy_insn_closure_by_addr = displaced_step_copy_insn_closure_by_addr;
+}
+
+void
+gdbarch_displaced_step_restore_all_in_ptid (struct gdbarch *gdbarch, inferior *parent_inf, ptid_t child_ptid)
 {
   gdb_assert (gdbarch != NULL);
-  gdb_assert (gdbarch->displaced_step_location != NULL);
+  gdb_assert (gdbarch->displaced_step_restore_all_in_ptid != NULL);
   if (gdbarch_debug >= 2)
-    fprintf_unfiltered (gdb_stdlog, "gdbarch_displaced_step_location called\n");
-  return gdbarch->displaced_step_location (gdbarch);
+    fprintf_unfiltered (gdb_stdlog, "gdbarch_displaced_step_restore_all_in_ptid called\n");
+  gdbarch->displaced_step_restore_all_in_ptid (parent_inf, child_ptid);
 }
 
 void
-set_gdbarch_displaced_step_location (struct gdbarch *gdbarch,
-                                     gdbarch_displaced_step_location_ftype displaced_step_location)
+set_gdbarch_displaced_step_restore_all_in_ptid (struct gdbarch *gdbarch,
+                                                gdbarch_displaced_step_restore_all_in_ptid_ftype displaced_step_restore_all_in_ptid)
 {
-  gdbarch->displaced_step_location = displaced_step_location;
+  gdbarch->displaced_step_restore_all_in_ptid = displaced_step_restore_all_in_ptid;
 }
 
 bool
diff --git a/gdb/gdbarch.h b/gdb/gdbarch.h
index 2b6876a1916..002628d8988 100644
--- a/gdb/gdbarch.h
+++ b/gdb/gdbarch.h
@@ -58,6 +58,7 @@ struct mem_range;
 struct syscalls_info;
 struct thread_info;
 struct ui_out;
+struct inferior;
 
 #include "regcache.h"
 
@@ -1071,17 +1072,36 @@ typedef void (gdbarch_displaced_step_fixup_ftype) (struct gdbarch *gdbarch, stru
 extern void gdbarch_displaced_step_fixup (struct gdbarch *gdbarch, struct displaced_step_copy_insn_closure *closure, CORE_ADDR from, CORE_ADDR to, struct regcache *regs);
 extern void set_gdbarch_displaced_step_fixup (struct gdbarch *gdbarch, gdbarch_displaced_step_fixup_ftype *displaced_step_fixup);
 
-/* Return the address of an appropriate place to put displaced
-   instructions while we step over them.  There need only be one such
-   place, since we're only stepping one thread over a breakpoint at a
-   time.
+/* Prepare THREAD for it to displaced step the instruction at its current PC.
   
-   For a general explanation of displaced stepping and how GDB uses it,
-   see the comments in infrun.c. */
+   Throw an exception if any unexpected error happens. */
+
+extern bool gdbarch_displaced_step_prepare_p (struct gdbarch *gdbarch);
+
+typedef displaced_step_prepare_status (gdbarch_displaced_step_prepare_ftype) (struct gdbarch *gdbarch, thread_info *thread, CORE_ADDR &displaced_pc);
+extern displaced_step_prepare_status gdbarch_displaced_step_prepare (struct gdbarch *gdbarch, thread_info *thread, CORE_ADDR &displaced_pc);
+extern void set_gdbarch_displaced_step_prepare (struct gdbarch *gdbarch, gdbarch_displaced_step_prepare_ftype *displaced_step_prepare);
+
+/* Clean up after a displaced step of THREAD. */
+
+typedef displaced_step_finish_status (gdbarch_displaced_step_finish_ftype) (struct gdbarch *gdbarch, thread_info *thread, gdb_signal sig);
+extern displaced_step_finish_status gdbarch_displaced_step_finish (struct gdbarch *gdbarch, thread_info *thread, gdb_signal sig);
+extern void set_gdbarch_displaced_step_finish (struct gdbarch *gdbarch, gdbarch_displaced_step_finish_ftype *displaced_step_finish);
+
+/* Return the closure associated to the displaced step buffer that is at ADDR. */
+
+extern bool gdbarch_displaced_step_copy_insn_closure_by_addr_p (struct gdbarch *gdbarch);
+
+typedef const displaced_step_copy_insn_closure * (gdbarch_displaced_step_copy_insn_closure_by_addr_ftype) (inferior *inf, CORE_ADDR addr);
+extern const displaced_step_copy_insn_closure * gdbarch_displaced_step_copy_insn_closure_by_addr (struct gdbarch *gdbarch, inferior *inf, CORE_ADDR addr);
+extern void set_gdbarch_displaced_step_copy_insn_closure_by_addr (struct gdbarch *gdbarch, gdbarch_displaced_step_copy_insn_closure_by_addr_ftype *displaced_step_copy_insn_closure_by_addr);
+
+/* PARENT_INF has forked and CHILD_PTID is the ptid of the child.  Restore the
+   contents of all displaced step buffers in the child's address space. */
 
-typedef CORE_ADDR (gdbarch_displaced_step_location_ftype) (struct gdbarch *gdbarch);
-extern CORE_ADDR gdbarch_displaced_step_location (struct gdbarch *gdbarch);
-extern void set_gdbarch_displaced_step_location (struct gdbarch *gdbarch, gdbarch_displaced_step_location_ftype *displaced_step_location);
+typedef void (gdbarch_displaced_step_restore_all_in_ptid_ftype) (inferior *parent_inf, ptid_t child_ptid);
+extern void gdbarch_displaced_step_restore_all_in_ptid (struct gdbarch *gdbarch, inferior *parent_inf, ptid_t child_ptid);
+extern void set_gdbarch_displaced_step_restore_all_in_ptid (struct gdbarch *gdbarch, gdbarch_displaced_step_restore_all_in_ptid_ftype *displaced_step_restore_all_in_ptid);
 
 /* Relocate an instruction to execute at a different address.  OLDLOC
    is the address in the inferior memory where the instruction to
diff --git a/gdb/gdbarch.sh b/gdb/gdbarch.sh
index 76ce2ff5efb..e90654dc5c1 100755
--- a/gdb/gdbarch.sh
+++ b/gdb/gdbarch.sh
@@ -813,14 +813,20 @@ m;bool;displaced_step_hw_singlestep;void;;;default_displaced_step_hw_singlestep;
 # see the comments in infrun.c.
 M;void;displaced_step_fixup;struct displaced_step_copy_insn_closure *closure, CORE_ADDR from, CORE_ADDR to, struct regcache *regs;closure, from, to, regs;;NULL
 
-# Return the address of an appropriate place to put displaced
-# instructions while we step over them.  There need only be one such
-# place, since we're only stepping one thread over a breakpoint at a
-# time.
+# Prepare THREAD for it to displaced step the instruction at its current PC.
 #
-# For a general explanation of displaced stepping and how GDB uses it,
-# see the comments in infrun.c.
-m;CORE_ADDR;displaced_step_location;void;;;NULL;;(! gdbarch->displaced_step_location) != (! gdbarch->displaced_step_copy_insn)
+# Throw an exception if any unexpected error happens.
+M;displaced_step_prepare_status;displaced_step_prepare;thread_info *thread, CORE_ADDR &displaced_pc;thread, displaced_pc
+
+# Clean up after a displaced step of THREAD.
+m;displaced_step_finish_status;displaced_step_finish;thread_info *thread, gdb_signal sig;thread, sig;;NULL;;(! gdbarch->displaced_step_finish) != (! gdbarch->displaced_step_prepare)
+
+# Return the closure associated to the displaced step buffer that is at ADDR.
+F;const displaced_step_copy_insn_closure *;displaced_step_copy_insn_closure_by_addr;inferior *inf, CORE_ADDR addr;inf, addr
+
+# PARENT_INF has forked and CHILD_PTID is the ptid of the child.  Restore the
+# contents of all displaced step buffers in the child's address space.
+f;void;displaced_step_restore_all_in_ptid;inferior *parent_inf, ptid_t child_ptid;parent_inf, child_ptid
 
 # Relocate an instruction to execute at a different address.  OLDLOC
 # is the address in the inferior memory where the instruction to
@@ -1297,6 +1303,7 @@ struct mem_range;
 struct syscalls_info;
 struct thread_info;
 struct ui_out;
+struct inferior;
 
 #include "regcache.h"
 
diff --git a/gdb/gdbthread.h b/gdb/gdbthread.h
index 1eecb989a79..e5484ac54ca 100644
--- a/gdb/gdbthread.h
+++ b/gdb/gdbthread.h
@@ -32,6 +32,7 @@ struct symtab;
 #include "gdbsupport/refcounted-object.h"
 #include "gdbsupport/common-gdbthread.h"
 #include "gdbsupport/forward-scope-exit.h"
+#include "displaced-stepping.h"
 
 struct inferior;
 struct process_stratum_target;
@@ -388,6 +389,9 @@ public:
      fields point to self.  */
   struct thread_info *step_over_prev = NULL;
   struct thread_info *step_over_next = NULL;
+
+  /* Displaced-step state for this thread.  */
+  displaced_step_thread_state displaced_step_state;
 };
 
 /* A gdb::ref_ptr pointer to a thread_info.  */
@@ -745,10 +749,27 @@ extern bool value_in_thread_stack_temporaries (struct value *,
 
 extern void global_thread_step_over_chain_enqueue (thread_info *tp);
 
+/* Append the thread step over chain CHAIN_HEAD to the global thread step over
+   chain. */
+
+extern void global_thread_step_over_chain_enqueue_chain
+  (thread_info *chain_head);
+
+/* Remove TP from step-over chain LIST_P.  */
+
+extern void thread_step_over_chain_remove (thread_info **list_p,
+					   thread_info *tp);
+
 /* Remove TP from the global pending step-over chain.  */
 
 extern void global_thread_step_over_chain_remove (thread_info *tp);
 
+/* Return the thread following TP in the step-over chain whose head is
+   CHAIN_HEAD.  Return NULL if TP is the last entry in the chain.  */
+
+extern thread_info *thread_step_over_chain_next (thread_info *chain_head,
+						 thread_info *tp);
+
 /* Return the thread following TP in the global step-over chain, or NULL if TP
    is the last entry in the chain.  */
 
@@ -758,6 +779,14 @@ extern thread_info *global_thread_step_over_chain_next (thread_info *tp);
 
 extern int thread_is_in_step_over_chain (struct thread_info *tp);
 
+/* Return the length of the the step over chain TP is in.
+
+   If TP is non-nullptr, the thread must be in a step over chain.
+   TP may be nullptr, in which case it denotes an empty list, so a length of
+   0.  */
+
+extern int thread_step_over_chain_length (thread_info *tp);
+
 /* Cancel any ongoing execution command.  */
 
 extern void thread_cancel_execution_command (struct thread_info *thr);
diff --git a/gdb/hppa-linux-tdep.c b/gdb/hppa-linux-tdep.c
index 080379992ae..a171e582e59 100644
--- a/gdb/hppa-linux-tdep.c
+++ b/gdb/hppa-linux-tdep.c
@@ -489,7 +489,7 @@ hppa_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* GNU/Linux is always ELF.  */
   tdep->is_elf = 1;
diff --git a/gdb/i386-linux-tdep.c b/gdb/i386-linux-tdep.c
index ab7d23611f8..1b209fd3eff 100644
--- a/gdb/i386-linux-tdep.c
+++ b/gdb/i386-linux-tdep.c
@@ -832,7 +832,7 @@ i386_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 
   gdb_assert (tdesc_data);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, true);
 
   /* GNU/Linux uses ELF.  */
   i386_elf_init_abi (info, gdbarch);
@@ -1065,8 +1065,6 @@ i386_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
   set_gdbarch_displaced_step_copy_insn (gdbarch,
 					i386_linux_displaced_step_copy_insn);
   set_gdbarch_displaced_step_fixup (gdbarch, i386_displaced_step_fixup);
-  set_gdbarch_displaced_step_location (gdbarch,
-				       linux_displaced_step_location);
 
   /* Functions for 'catch syscall'.  */
   set_xml_syscall_file_name (gdbarch, XML_SYSCALL_FILENAME_I386);
diff --git a/gdb/ia64-linux-tdep.c b/gdb/ia64-linux-tdep.c
index 587a455467d..d6d581ab3dd 100644
--- a/gdb/ia64-linux-tdep.c
+++ b/gdb/ia64-linux-tdep.c
@@ -223,7 +223,7 @@ ia64_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
   static const char *const stap_register_indirection_suffixes[] = { "]",
 								    NULL };
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Set the method of obtaining the sigcontext addresses at which
      registers are saved.  */
diff --git a/gdb/infrun.c b/gdb/infrun.c
index 9ac75eff4d3..e7b69ae5a5c 100644
--- a/gdb/infrun.c
+++ b/gdb/infrun.c
@@ -1452,21 +1452,6 @@ step_over_info_valid_p (void)
    displaced step operation on it.  See displaced_step_prepare and
    displaced_step_finish for details.  */
 
-/* Returns true if any inferior has a thread doing a displaced
-   step.  */
-
-static bool
-displaced_step_in_progress_any_inferior ()
-{
-  for (inferior *i : all_inferiors ())
-    {
-      if (i->displaced_step_state.step_thread != nullptr)
-	return true;
-    }
-
-  return false;
-}
-
 /* Return true if THREAD is doing a displaced step.  */
 
 static bool
@@ -1474,7 +1459,7 @@ displaced_step_in_progress_thread (thread_info *thread)
 {
   gdb_assert (thread != NULL);
 
-  return thread->inf->displaced_step_state.step_thread == thread;
+  return thread->displaced_step_state.in_progress ();
 }
 
 /* Return true if INF has a thread doing a displaced step.  */
@@ -1482,25 +1467,21 @@ displaced_step_in_progress_thread (thread_info *thread)
 static bool
 displaced_step_in_progress (inferior *inf)
 {
-  return inf->displaced_step_state.step_thread != nullptr;
+  return inf->displaced_step_state.in_progress_count > 0;
 }
 
-/* If inferior is in displaced stepping, and ADDR equals to starting address
-   of copy area, return corresponding displaced_step_copy_insn_closure.
-   Otherwise, return NULL.  */
+/* Return true if any thread is doing a displaced step.  */
 
-displaced_step_copy_insn_closure *
-get_displaced_step_copy_insn_closure_by_addr (CORE_ADDR addr)
+static bool
+displaced_step_in_progress_any_thread ()
 {
-  displaced_step_inferior_state &displaced
-    = current_inferior ()->displaced_step_state;
-
-  /* If checking the mode of displaced instruction in copy area.  */
-  if (displaced.step_thread != nullptr
-      && displaced.step_copy == addr)
-    return displaced.step_closure.get ();
+  for (inferior *inf : all_non_exited_inferiors ())
+    {
+      if (displaced_step_in_progress (inf))
+	return true;
+    }
 
-  return NULL;
+  return false;
 }
 
 static void
@@ -1512,12 +1493,15 @@ infrun_inferior_exit (struct inferior *inf)
 static void
 infrun_inferior_execd (inferior *inf)
 {
-  /* If a thread was doing a displaced step in this inferior at the moment of
-     the exec, it no longer exists.  Even if the exec'ing thread was the one
+  /* If some threads where was doing a displaced step in this inferior at the
+     moment of the exec, they no longer exist.  Even if the exec'ing thread
      doing a displaced step, we don't want to to any fixup nor restore displaced
      stepping buffer bytes.  */
   inf->displaced_step_state.reset ();
 
+  for (thread_info *thread : inf->threads ())
+    thread->displaced_step_state.reset ();
+
   /* Since an in-line step is done with everything else stopped, if there was
      one in progress at the time of the exec, it must have been the exec'ing
      thread.  */
@@ -1555,9 +1539,9 @@ show_can_use_displaced_stepping (struct ui_file *file, int from_tty,
 static bool
 gdbarch_supports_displaced_stepping (gdbarch *arch)
 {
-  /* Only check for the presence of step_copy_insn.  Other required methods
-     are checked by the gdbarch validation.  */
-  return gdbarch_displaced_step_copy_insn_p (arch);
+  /* Only check for the presence of `prepare`.  The gdbarch verification ensures
+     that if `prepare` is provided, so is `finish`.  */
+  return gdbarch_displaced_step_prepare_p (arch);
 }
 
 /* Return non-zero if displaced stepping can/should be used to step
@@ -1595,10 +1579,10 @@ use_displaced_stepping (thread_info *tp)
   return true;
 }
 
-/* Simple function wrapper around displaced_step_inferior_state::reset.  */
+/* Simple function wrapper around displaced_step_thread_state::reset.  */
 
 static void
-displaced_step_reset (displaced_step_inferior_state *displaced)
+displaced_step_reset (displaced_step_thread_state *displaced)
 {
   displaced->reset ();
 }
@@ -1649,10 +1633,8 @@ displaced_step_prepare_throw (thread_info *tp)
 {
   regcache *regcache = get_thread_regcache (tp);
   struct gdbarch *gdbarch = regcache->arch ();
-  const address_space *aspace = regcache->aspace ();
-  CORE_ADDR original, copy;
-  ULONGEST len;
-  int status;
+  displaced_step_thread_state &disp_step_thread_state
+    = tp->displaced_step_state;
 
   /* We should never reach this function if the architecture does not
      support displaced stepping.  */
@@ -1667,15 +1649,14 @@ displaced_step_prepare_throw (thread_info *tp)
      jump/branch).  */
   tp->control.may_range_step = 0;
 
-  /* We have to displaced step one thread at a time, as we only have
-     access to a single scratch space per inferior.  */
-
-  displaced_step_inferior_state *displaced = &tp->inf->displaced_step_state;
+  /* We are about to start a displaced step for this thread.  If one is already
+     in progress, something's wrong.  */
+  gdb_assert (!disp_step_thread_state.in_progress ());
 
-  if (displaced->step_thread != nullptr)
+  if (tp->inf->displaced_step_state.unavailable)
     {
-      /* Already waiting for a displaced step to finish.  Defer this
-	 request and place in queue.  */
+      /* The gdbarch tells us it's not worth asking to try a prepare because
+	 it is likely that it will return unavailable, so don't bother asking.  */
 
       displaced_debug_printf ("deferring step of %s",
 			      target_pid_to_str (tp->ptid).c_str ());
@@ -1683,79 +1664,54 @@ displaced_step_prepare_throw (thread_info *tp)
       global_thread_step_over_chain_enqueue (tp);
       return DISPLACED_STEP_PREPARE_STATUS_UNAVAILABLE;
     }
-  else
-    displaced_debug_printf ("stepping %s now",
-			    target_pid_to_str (tp->ptid).c_str ());
 
-  displaced_step_reset (displaced);
+  displaced_debug_printf ("displaced-stepping %s now",
+			  target_pid_to_str (tp->ptid).c_str ());
 
   scoped_restore_current_thread restore_thread;
 
   switch_to_thread (tp);
 
-  original = regcache_read_pc (regcache);
+  CORE_ADDR original_pc = regcache_read_pc (regcache);
+  CORE_ADDR displaced_pc;
 
-  copy = gdbarch_displaced_step_location (gdbarch);
-  len = gdbarch_max_insn_length (gdbarch);
+  displaced_step_prepare_status status
+    = gdbarch_displaced_step_prepare (gdbarch, tp, displaced_pc);
 
-  if (breakpoint_in_range_p (aspace, copy, len))
+  if (status == DISPLACED_STEP_PREPARE_STATUS_CANT)
     {
-      /* There's a breakpoint set in the scratch pad location range
-	 (which is usually around the entry point).  We'd either
-	 install it before resuming, which would overwrite/corrupt the
-	 scratch pad, or if it was already inserted, this displaced
-	 step would overwrite it.  The latter is OK in the sense that
-	 we already assume that no thread is going to execute the code
-	 in the scratch pad range (after initial startup) anyway, but
-	 the former is unacceptable.  Simply punt and fallback to
-	 stepping over this breakpoint in-line.  */
-      displaced_debug_printf ("breakpoint set in scratch pad.  "
-			      "Stepping over breakpoint in-line instead.");
+      displaced_debug_printf ("failed to prepare (%s)",
+			      target_pid_to_str (tp->ptid).c_str ());
 
       return DISPLACED_STEP_PREPARE_STATUS_CANT;
     }
+  else if (status == DISPLACED_STEP_PREPARE_STATUS_UNAVAILABLE)
+    {
+      /* Not enough displaced stepping resources available, defer this
+	 request by placing it the queue.  */
 
-  /* Save the original contents of the copy area.  */
-  displaced->step_saved_copy.resize (len);
-  status = target_read_memory (copy, displaced->step_saved_copy.data (), len);
-  if (status != 0)
-    throw_error (MEMORY_ERROR,
-		 _("Error accessing memory address %s (%s) for "
-		   "displaced-stepping scratch space."),
-		 paddress (gdbarch, copy), safe_strerror (status));
+      displaced_debug_printf ("not enough resources available, "
+			      "deferring step of %s",
+			      target_pid_to_str (tp->ptid).c_str ());
 
-  displaced_debug_printf ("saved %s: %s",
-			  paddress (gdbarch, copy),
-			  displaced_step_dump_bytes
-			    (displaced->step_saved_copy.data (), len).c_str ());
+      global_thread_step_over_chain_enqueue (tp);
 
-  displaced->step_closure
-    = gdbarch_displaced_step_copy_insn (gdbarch, original, copy, regcache);
-  if (displaced->step_closure == NULL)
-    {
-      /* The architecture doesn't know how or want to displaced step
-	 this instruction or instruction sequence.  Fallback to
-	 stepping over the breakpoint in-line.  */
-      return DISPLACED_STEP_PREPARE_STATUS_CANT;
+      return DISPLACED_STEP_PREPARE_STATUS_UNAVAILABLE;
     }
 
+  gdb_assert (status == DISPLACED_STEP_PREPARE_STATUS_OK);
+
   /* Save the information we need to fix things up if the step
      succeeds.  */
-  displaced->step_thread = tp;
-  displaced->step_gdbarch = gdbarch;
-  displaced->step_original = original;
-  displaced->step_copy = copy;
-
-  {
-    displaced_step_reset_cleanup cleanup (displaced);
+  disp_step_thread_state.set (gdbarch);
 
-    /* Resume execution at the copy.  */
-    regcache_write_pc (regcache, copy);
+  tp->inf->displaced_step_state.in_progress_count++;
 
-    cleanup.release ();
-  }
-
-  displaced_debug_printf ("displaced pc to %s", paddress (gdbarch, copy));
+  displaced_debug_printf ("prepared successfully thread=%s, "
+			  "original_pc=%s, displaced_pc=%s",
+			  target_pid_to_str (tp->ptid).c_str (),
+			  paddress (gdbarch, original_pc),
+			  paddress (gdbarch, displaced_pc));
 
   return DISPLACED_STEP_PREPARE_STATUS_OK;
 }
@@ -1797,33 +1753,6 @@ displaced_step_prepare (thread_info *thread)
   return status;
 }
 
-static void
-write_memory_ptid (ptid_t ptid, CORE_ADDR memaddr,
-		   const gdb_byte *myaddr, int len)
-{
-  scoped_restore save_inferior_ptid = make_scoped_restore (&inferior_ptid);
-
-  inferior_ptid = ptid;
-  write_memory (memaddr, myaddr, len);
-}
-
-/* Restore the contents of the copy area for thread PTID.  */
-
-static void
-displaced_step_restore (struct displaced_step_inferior_state *displaced,
-			ptid_t ptid)
-{
-  ULONGEST len = gdbarch_max_insn_length (displaced->step_gdbarch);
-
-  write_memory_ptid (ptid, displaced->step_copy,
-		     displaced->step_saved_copy.data (), len);
-
-  displaced_debug_printf ("restored %s %s",
-			  target_pid_to_str (ptid).c_str (),
-			  paddress (displaced->step_gdbarch,
-				    displaced->step_copy));
-}
-
 /* If we displaced stepped an instruction successfully, adjust registers and
    memory to yield the same effect the instruction would have had if we had
    executed it at its original address, and return
@@ -1836,13 +1765,15 @@ displaced_step_restore (struct displaced_step_inferior_state *displaced,
 static displaced_step_finish_status
 displaced_step_finish (thread_info *event_thread, enum gdb_signal signal)
 {
-  displaced_step_inferior_state *displaced
-    = &event_thread->inf->displaced_step_state;
+  displaced_step_thread_state *displaced = &event_thread->displaced_step_state;
 
-  /* Was this event for the thread we displaced?  */
-  if (displaced->step_thread != event_thread)
+  /* Was this thread performing a displaced step?  */
+  if (!displaced->in_progress ())
     return DISPLACED_STEP_FINISH_STATUS_OK;
 
+  gdb_assert (event_thread->inf->displaced_step_state.in_progress_count > 0);
+  event_thread->inf->displaced_step_state.in_progress_count--;
+
   /* Fixup may need to read memory/registers.  Switch to the thread
      that we're fixing up.  Also, target_stopped_by_watchpoint checks
      the current thread, and displaced_step_restore performs ptid-dependent
@@ -1851,35 +1782,10 @@ displaced_step_finish (thread_info *event_thread, enum gdb_signal signal)
 
   displaced_step_reset_cleanup cleanup (displaced);
 
-  displaced_step_restore (displaced, displaced->step_thread->ptid);
-
-  /* Did the instruction complete successfully?  */
-  if (signal == GDB_SIGNAL_TRAP
-      && !(target_stopped_by_watchpoint ()
-	   && (gdbarch_have_nonsteppable_watchpoint (displaced->step_gdbarch)
-	       || target_have_steppable_watchpoint ())))
-    {
-      /* Fix up the resulting state.  */
-      gdbarch_displaced_step_fixup (displaced->step_gdbarch,
-				    displaced->step_closure.get (),
-				    displaced->step_original,
-				    displaced->step_copy,
-				    get_thread_regcache (displaced->step_thread));
-
-      return DISPLACED_STEP_FINISH_STATUS_OK;
-    }
-  else
-    {
-      /* Since the instruction didn't complete, all we can do is
-	 relocate the PC.  */
-      struct regcache *regcache = get_thread_regcache (event_thread);
-      CORE_ADDR pc = regcache_read_pc (regcache);
-
-      pc = displaced->step_original + (pc - displaced->step_copy);
-      regcache_write_pc (regcache, pc);
-
-      return DISPLACED_STEP_FINISH_STATUS_NOT_EXECUTED;
-    }
+  /* Do the fixup, and release the resources acquired to do the displaced
+     step. */
+  return gdbarch_displaced_step_finish (displaced->get_original_gdbarch (),
+					event_thread, signal);
 }
 
 /* Data to be passed around while handling an event.  This data is
@@ -1927,14 +1833,42 @@ static step_over_what thread_still_needs_step_over (struct thread_info *tp);
 static bool
 start_step_over (void)
 {
-  struct thread_info *tp, *next;
+  thread_info *next;
 
   /* Don't start a new step-over if we already have an in-line
      step-over operation ongoing.  */
   if (step_over_info_valid_p ())
     return false;
 
-  for (tp = global_thread_step_over_chain_head; tp != NULL; tp = next)
+  /* Steal the global thread step over chain.  As we try to initiate displaced
+     steps, threads will be enqueued in the global chain if no buffers are
+     available.  If we iterated on the global chain directly, we might iterate
+     indefinitely.  */
+  thread_info *threads_to_step = global_thread_step_over_chain_head;
+  global_thread_step_over_chain_head = NULL;
+
+  infrun_debug_printf ("stealing global queue of threads to step, length = %d",
+		       thread_step_over_chain_length (threads_to_step));
+
+  bool started = false;
+
+  /* On scope exit (whatever the reason, return or exception), if there are
+     threads left in the THREADS_TO_STEP chain, put back these threads in the
+     global list.  */
+  SCOPE_EXIT
+    {
+      if (threads_to_step == nullptr)
+	infrun_debug_printf ("step-over queue now empty");
+      else
+	{
+	  infrun_debug_printf ("putting back %d threads to step in global queue",
+			       thread_step_over_chain_length (threads_to_step));
+
+	  global_thread_step_over_chain_enqueue_chain (threads_to_step);
+	}
+    };
+
+  for (thread_info *tp = threads_to_step; tp != NULL; tp = next)
     {
       struct execution_control_state ecss;
       struct execution_control_state *ecs = &ecss;
@@ -1943,12 +1877,23 @@ start_step_over (void)
 
       gdb_assert (!tp->stop_requested);
 
-      next = global_thread_step_over_chain_next (tp);
+      next = thread_step_over_chain_next (threads_to_step, tp);
 
-      /* If this inferior already has a displaced step in process,
-	 don't start a new one.  */
-      if (displaced_step_in_progress (tp->inf))
-	continue;
+      if (tp->inf->displaced_step_state.unavailable)
+	{
+	  /* The arch told us to not even try preparing another displaced step
+	     for this inferior.  Just leave the thread in THREADS_TO_STEP, it
+	     will get moved to the global chain on scope exit.  */
+	  continue;
+	}
+
+      /* Remove thread from the THREADS_TO_STEP chain.  If anything goes wrong
+	 while we try to prepare the displaced step, we don't add it back to
+	 the global step over chain.  This is to avoid a thread staying in the
+	 step over chain indefinitely if something goes wrong when resuming it
+	 If the error is intermittent and it still needs a step over, it will
+	 get enqueued again when we try to resume it normally.  */
+      thread_step_over_chain_remove (&threads_to_step, tp);
 
       step_what = thread_still_needs_step_over (tp);
       must_be_in_line = ((step_what & STEP_OVER_WATCHPOINT)
@@ -1958,13 +1903,11 @@ start_step_over (void)
       /* We currently stop all threads of all processes to step-over
 	 in-line.  If we need to start a new in-line step-over, let
 	 any pending displaced steps finish first.  */
-      if (must_be_in_line && displaced_step_in_progress_any_inferior ())
-	return false;
-
-      global_thread_step_over_chain_remove (tp);
-
-      if (global_thread_step_over_chain_head == NULL)
-	infrun_debug_printf ("step-over queue now empty");
+      if (must_be_in_line && displaced_step_in_progress_any_thread ())
+	{
+	  global_thread_step_over_chain_enqueue (tp);
+	  continue;
+	}
 
       if (tp->control.trap_expected
 	  || tp->resumed
@@ -1998,13 +1941,27 @@ start_step_over (void)
       if (!ecs->wait_some_more)
 	error (_("Command aborted."));
 
-      gdb_assert (tp->resumed);
+      /* If the thread's step over could not be initiated because no buffers
+	 were available, it was re-added to the global step over chain.  */
+      if (tp->resumed)
+	{
+	  infrun_debug_printf ("[%s] was resumed.",
+			       target_pid_to_str (tp->ptid).c_str ());
+	  gdb_assert (!thread_is_in_step_over_chain (tp));
+	}
+      else
+	{
+	  infrun_debug_printf ("[%s] was NOT resumed.",
+			       target_pid_to_str (tp->ptid).c_str ());
+	  gdb_assert (thread_is_in_step_over_chain (tp));
+	}
 
       /* If we started a new in-line step-over, we're done.  */
       if (step_over_info_valid_p ())
 	{
 	  gdb_assert (tp->control.trap_expected);
-	  return true;
+	  started = true;
+	  break;
 	}
 
       if (!target_is_non_stop_p ())
@@ -2017,7 +1974,8 @@ start_step_over (void)
 	  /* With remote targets (at least), in all-stop, we can't
 	     issue any further remote commands until the program stops
 	     again.  */
-	  return true;
+	  started = true;
+	  break;
 	}
 
       /* Either the thread no longer needed a step-over, or a new
@@ -2026,7 +1984,7 @@ start_step_over (void)
 	 displaced step on a thread of other process. */
     }
 
-  return false;
+  return started;
 }
 
 /* Update global variables holding ptids to hold NEW_PTID if they were
@@ -3611,18 +3569,16 @@ prepare_for_detach (void)
   struct inferior *inf = current_inferior ();
   ptid_t pid_ptid = ptid_t (inf->pid);
 
-  displaced_step_inferior_state *displaced = &inf->displaced_step_state;
-
   /* Is any thread of this process displaced stepping?  If not,
      there's nothing else to do.  */
-  if (displaced->step_thread == nullptr)
+  if (displaced_step_in_progress (inf))
     return;
 
   infrun_debug_printf ("displaced-stepping in-process while detaching");
 
   scoped_restore restore_detaching = make_scoped_restore (&inf->detaching, true);
 
-  while (displaced->step_thread != nullptr)
+  while (displaced_step_in_progress (inf))
     {
       struct execution_control_state ecss;
       struct execution_control_state *ecs;
@@ -5288,17 +5244,12 @@ handle_inferior_event (struct execution_control_state *ecs)
 	struct gdbarch *gdbarch = regcache->arch ();
 	inferior *parent_inf = find_inferior_ptid (ecs->target, ecs->ptid);
 
-	/* If this is a fork (child gets its own address space copy) and the
-	   displaced step buffer was in use at the time of the fork, restore
-	   displaced step buffer bytes in the child process.  */
+	/* If this is a fork (child gets its own address space copy) and some
+	   displaced step buffers were in use at the time of the fork, restore
+	   the displaced step buffer bytes in the child process.  */
 	if (ecs->ws.kind == TARGET_WAITKIND_FORKED)
-	  {
-	    displaced_step_inferior_state *displaced
-	      = &parent_inf->displaced_step_state;
-
-	    if (displaced->step_thread != nullptr)
-	      displaced_step_restore (displaced, ecs->ws.value.related_pid);
-	  }
+	  gdbarch_displaced_step_restore_all_in_ptid
+	    (gdbarch, parent_inf, ecs->ws.value.related_pid);
 
 	/* If displaced stepping is supported, and thread ecs->ptid is
 	   displaced stepping.  */
diff --git a/gdb/infrun.h b/gdb/infrun.h
index c83cb333083..d5e6d279f1a 100644
--- a/gdb/infrun.h
+++ b/gdb/infrun.h
@@ -226,9 +226,6 @@ extern void clear_exit_convenience_vars (void);
 /* Dump LEN bytes at BUF in hex to a string and return it.  */
 extern std::string displaced_step_dump_bytes (const gdb_byte *buf, size_t len);
 
-extern struct displaced_step_copy_insn_closure *
-  get_displaced_step_copy_insn_closure_by_addr (CORE_ADDR addr);
-
 extern void update_observer_mode (void);
 
 extern void signal_catch_update (const unsigned int *);
diff --git a/gdb/linux-tdep.c b/gdb/linux-tdep.c
index 655a2c33737..70e74d51097 100644
--- a/gdb/linux-tdep.c
+++ b/gdb/linux-tdep.c
@@ -199,6 +199,9 @@ struct linux_info
      yet.  Positive if we tried looking it up, and found it.  Negative
      if we tried looking it up but failed.  */
   int vsyscall_range_p = 0;
+
+  /* Inferior's displaced step buffer.  */
+  gdb::optional<displaced_step_buffer> disp_step_buf;
 };
 
 /* Per-inferior data key.  */
@@ -2531,6 +2534,65 @@ linux_displaced_step_location (struct gdbarch *gdbarch)
 
 /* See linux-tdep.h.  */
 
+displaced_step_prepare_status
+linux_displaced_step_prepare (gdbarch *arch, thread_info *thread,
+			      CORE_ADDR &displaced_pc)
+{
+  linux_info *per_inferior = get_linux_inferior_data (thread->inf);
+
+  if (!per_inferior->disp_step_buf.has_value ())
+    {
+      CORE_ADDR disp_step_buf_addr
+	= linux_displaced_step_location (thread->inf->gdbarch);
+
+      per_inferior->disp_step_buf.emplace (disp_step_buf_addr);
+    }
+
+  return per_inferior->disp_step_buf->prepare (thread, displaced_pc);
+}
+
+/* See linux-tdep.h.  */
+
+displaced_step_finish_status
+linux_displaced_step_finish (gdbarch *arch, thread_info *thread, gdb_signal sig)
+{
+  linux_info *per_inferior = get_linux_inferior_data (thread->inf);
+
+  gdb_assert (per_inferior->disp_step_buf.has_value ());
+
+  return per_inferior->disp_step_buf->finish (arch, thread, sig);
+}
+
+/* See linux-tdep.h.  */
+
+const displaced_step_copy_insn_closure *
+linux_displaced_step_copy_insn_closure_by_addr (inferior *inf, CORE_ADDR addr)
+{
+  linux_info *per_inferior = linux_inferior_data.get (inf);
+
+  if (per_inferior == nullptr
+      || !per_inferior->disp_step_buf.has_value ())
+    return nullptr;
+
+  return per_inferior->disp_step_buf->copy_insn_closure_by_addr (addr);
+}
+
+/* See linux-tdep.h.  */
+
+void
+linux_displaced_step_restore_all_in_ptid (inferior *parent_inf, ptid_t ptid)
+{
+  linux_info *per_inferior = linux_inferior_data.get (parent_inf);
+
+  if (per_inferior == nullptr
+      || !per_inferior->disp_step_buf.has_value ())
+    return;
+
+  per_inferior->disp_step_buf->restore_in_ptid (ptid);
+}
+
+/* See linux-tdep.h.  */
+
 CORE_ADDR
 linux_get_hwcap (struct target_ops *target)
 {
@@ -2577,8 +2639,19 @@ show_dump_excluded_mappings (struct ui_file *file, int from_tty,
    various GNU/Linux architectures and machine types.  */
 
 void
-linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
+linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch,
+		bool supports_displaced_step)
 {
+  if (supports_displaced_step)
+    {
+      set_gdbarch_displaced_step_prepare (gdbarch, linux_displaced_step_prepare);
+      set_gdbarch_displaced_step_finish (gdbarch, linux_displaced_step_finish);
+      set_gdbarch_displaced_step_copy_insn_closure_by_addr
+	(gdbarch, linux_displaced_step_copy_insn_closure_by_addr);
+      set_gdbarch_displaced_step_restore_all_in_ptid
+	(gdbarch, linux_displaced_step_restore_all_in_ptid);
+    }
+
   set_gdbarch_core_pid_to_str (gdbarch, linux_core_pid_to_str);
   set_gdbarch_info_proc (gdbarch, linux_info_proc);
   set_gdbarch_core_info_proc (gdbarch, linux_core_info_proc);
@@ -2608,6 +2681,7 @@ _initialize_linux_tdep ()
   /* Observers used to invalidate the cache when needed.  */
   gdb::observers::inferior_exit.attach (invalidate_linux_cache_inf);
   gdb::observers::inferior_appeared.attach (invalidate_linux_cache_inf);
+  gdb::observers::inferior_execd.attach (invalidate_linux_cache_inf);
 
   add_setshow_boolean_cmd ("use-coredump-filter", class_files,
 			   &use_coredump_filter, _("\
diff --git a/gdb/linux-tdep.h b/gdb/linux-tdep.h
index 91c28738f52..0f83dc3c781 100644
--- a/gdb/linux-tdep.h
+++ b/gdb/linux-tdep.h
@@ -21,7 +21,9 @@
 #define LINUX_TDEP_H
 
 #include "bfd.h"
+#include "displaced-stepping.h"
 
+struct inferior;
 struct regcache;
 
 /* Enum used to define the extra fields of the siginfo type used by an
@@ -57,7 +59,30 @@ extern int linux_gdb_signal_to_target (struct gdbarch *gdbarch,
    the target auxiliary vector.  */
 extern CORE_ADDR linux_displaced_step_location (struct gdbarch *gdbarch);
 
-extern void linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch);
+
+/* Implementation of gdbarch_displaced_step_prepare.  */
+
+extern displaced_step_prepare_status linux_displaced_step_prepare
+  (gdbarch *arch, thread_info *thread, CORE_ADDR &displaced_pc);
+
+/* Implementation of gdbarch_displaced_step_finish.  */
+
+extern displaced_step_finish_status linux_displaced_step_finish
+  (gdbarch *arch, thread_info *thread, gdb_signal sig);
+
+/* Implementation of gdbarch_displaced_step_copy_insn_closure_by_addr.  */
+
+extern const displaced_step_copy_insn_closure *
+  linux_displaced_step_copy_insn_closure_by_addr
+    (inferior *inf, CORE_ADDR addr);
+
+/* Implementation of gdbarch_displaced_step_restore_all_in_ptid.  */
+
+extern void linux_displaced_step_restore_all_in_ptid (inferior *parent_inf,
+						      ptid_t ptid);
+
+extern void linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch,
+			    bool supports_displaced_step);
 
 extern int linux_is_uclinux (void);
 
diff --git a/gdb/m32r-linux-tdep.c b/gdb/m32r-linux-tdep.c
index 60a52e60b6e..961d54a2ca2 100644
--- a/gdb/m32r-linux-tdep.c
+++ b/gdb/m32r-linux-tdep.c
@@ -449,7 +449,7 @@ static void
 m32r_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Since EVB register is not available for native debug, we reduce
      the number of registers.  */
diff --git a/gdb/m68k-linux-tdep.c b/gdb/m68k-linux-tdep.c
index 63dd53e71d6..509333558ec 100644
--- a/gdb/m68k-linux-tdep.c
+++ b/gdb/m68k-linux-tdep.c
@@ -385,7 +385,7 @@ m68k_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   tdep->jb_pc = M68K_LINUX_JB_PC;
   tdep->jb_elt_size = M68K_LINUX_JB_ELEMENT_SIZE;
diff --git a/gdb/microblaze-linux-tdep.c b/gdb/microblaze-linux-tdep.c
index be710bedb64..2a91e1bb39a 100644
--- a/gdb/microblaze-linux-tdep.c
+++ b/gdb/microblaze-linux-tdep.c
@@ -117,7 +117,7 @@ static void
 microblaze_linux_init_abi (struct gdbarch_info info,
 			   struct gdbarch *gdbarch)
 {
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   set_gdbarch_memory_remove_breakpoint (gdbarch,
 					microblaze_linux_memory_remove_breakpoint);
diff --git a/gdb/mips-linux-tdep.c b/gdb/mips-linux-tdep.c
index aa2bd53348d..9ca59e5b296 100644
--- a/gdb/mips-linux-tdep.c
+++ b/gdb/mips-linux-tdep.c
@@ -1531,7 +1531,7 @@ mips_linux_init_abi (struct gdbarch_info info,
   enum mips_abi abi = mips_abi (gdbarch);
   struct tdesc_arch_data *tdesc_data = info.tdesc_data;
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Get the syscall number from the arch's register.  */
   set_gdbarch_get_syscall_number (gdbarch, mips_linux_get_syscall_number);
diff --git a/gdb/mn10300-linux-tdep.c b/gdb/mn10300-linux-tdep.c
index b5c391c0ccc..27645b1260c 100644
--- a/gdb/mn10300-linux-tdep.c
+++ b/gdb/mn10300-linux-tdep.c
@@ -704,7 +704,7 @@ am33_linux_sigframe_cache_init (const struct tramp_frame *self,
 static void
 am33_linux_init_osabi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   set_gdbarch_iterate_over_regset_sections
     (gdbarch, am33_iterate_over_regset_sections);
diff --git a/gdb/nios2-linux-tdep.c b/gdb/nios2-linux-tdep.c
index e6703ae9bcc..b5c12852c70 100644
--- a/gdb/nios2-linux-tdep.c
+++ b/gdb/nios2-linux-tdep.c
@@ -219,7 +219,7 @@ nios2_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Shared library handling.  */
   set_gdbarch_skip_trampoline_code (gdbarch, find_solib_trampoline_target);
diff --git a/gdb/or1k-linux-tdep.c b/gdb/or1k-linux-tdep.c
index 15677f9cfaf..33ddd10e85f 100644
--- a/gdb/or1k-linux-tdep.c
+++ b/gdb/or1k-linux-tdep.c
@@ -140,7 +140,7 @@ or1k_linux_sigframe_init (const struct tramp_frame *self,
 static void
 or1k_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   set_solib_svr4_fetch_link_map_offsets (gdbarch,
 					 svr4_ilp32_fetch_link_map_offsets);
diff --git a/gdb/ppc-linux-tdep.c b/gdb/ppc-linux-tdep.c
index 8cd8ef893ad..e6c10dd83db 100644
--- a/gdb/ppc-linux-tdep.c
+++ b/gdb/ppc-linux-tdep.c
@@ -1993,7 +1993,7 @@ ppc_linux_init_abi (struct gdbarch_info info,
   static const char *const stap_register_indirection_suffixes[] = { ")",
 								    NULL };
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* PPC GNU/Linux uses either 64-bit or 128-bit long doubles; where
      128-bit, they can be either IBM long double or IEEE quad long double.
@@ -2143,9 +2143,6 @@ ppc_linux_init_abi (struct gdbarch_info info,
 	}
     }
 
-  set_gdbarch_displaced_step_location (gdbarch,
-				       linux_displaced_step_location);
-
   /* Support reverse debugging.  */
   set_gdbarch_process_record (gdbarch, ppc_process_record);
   set_gdbarch_process_record_signal (gdbarch, ppc_linux_record_signal);
diff --git a/gdb/riscv-linux-tdep.c b/gdb/riscv-linux-tdep.c
index fce838097e2..a2238ad786e 100644
--- a/gdb/riscv-linux-tdep.c
+++ b/gdb/riscv-linux-tdep.c
@@ -159,7 +159,7 @@ riscv_linux_sigframe_init (const struct tramp_frame *self,
 static void
 riscv_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   set_gdbarch_software_single_step (gdbarch, riscv_software_single_step);
 
diff --git a/gdb/rs6000-aix-tdep.c b/gdb/rs6000-aix-tdep.c
index 03e573b436a..b5b398ed4d7 100644
--- a/gdb/rs6000-aix-tdep.c
+++ b/gdb/rs6000-aix-tdep.c
@@ -1132,10 +1132,12 @@ rs6000_aix_init_osabi (struct gdbarch_info info, struct gdbarch *gdbarch)
   set_gdbarch_software_single_step (gdbarch, rs6000_software_single_step);
 
   /* Displaced stepping is currently not supported in combination with
-     software single-stepping.  */
+     software single-stepping.  These override the values set by
+     rs6000_gdbarch_init.  */
   set_gdbarch_displaced_step_copy_insn (gdbarch, NULL);
   set_gdbarch_displaced_step_fixup (gdbarch, NULL);
-  set_gdbarch_displaced_step_location (gdbarch, NULL);
+  set_gdbarch_displaced_step_prepare (gdbarch, NULL);
+  set_gdbarch_displaced_step_finish (gdbarch, NULL);
 
   set_gdbarch_push_dummy_call (gdbarch, rs6000_push_dummy_call);
   set_gdbarch_return_value (gdbarch, rs6000_return_value);
diff --git a/gdb/rs6000-tdep.c b/gdb/rs6000-tdep.c
index 626b47e244f..1a7c7fcb9f1 100644
--- a/gdb/rs6000-tdep.c
+++ b/gdb/rs6000-tdep.c
@@ -153,6 +153,31 @@ static const char *const powerpc_vector_strings[] =
 static enum powerpc_vector_abi powerpc_vector_abi_global = POWERPC_VEC_AUTO;
 static const char *powerpc_vector_abi_string = "auto";
 
+/* PowerPC-related per-inferior data.  */
+
+struct ppc_inferior_data
+{
+  /* This is an optional in case we add more fields to ppc_inferior_data, we
+     don't want it instantiated as soon as we get the ppc_inferior_data for an
+     inferior.  */
+  gdb::optional<displaced_step_buffer> disp_step_buf;
+};
+
+static inferior_key<ppc_inferior_data> ppc_inferior_data_key;
+
+/* Get the per-inferior PowerPC data for INF.  */
+
+static ppc_inferior_data *
+get_ppc_per_inferior (inferior *inf)
+{
+  ppc_inferior_data *per_inf = ppc_inferior_data_key.get (inf);
+
+  if (per_inf == nullptr)
+    per_inf = ppc_inferior_data_key.emplace (inf);
+
+  return per_inf;
+}
+
 /* To be used by skip_prologue.  */
 
 struct rs6000_framedata
@@ -979,6 +1004,53 @@ ppc_displaced_step_fixup (struct gdbarch *gdbarch,
 				    from + offset);
 }
 
+/* Implementation of gdbarch_displaced_step_prepare.  */
+
+static displaced_step_prepare_status
+ppc_displaced_step_prepare  (gdbarch *arch, thread_info *thread,
+			     CORE_ADDR &displaced_pc)
+{
+  ppc_inferior_data *per_inferior = get_ppc_per_inferior (thread->inf);
+
+  if (!per_inferior->disp_step_buf.has_value ())
+    {
+      /* Figure out where the displaced step buffer is.  */
+      CORE_ADDR disp_step_buf_addr
+	= displaced_step_at_entry_point (thread->inf->gdbarch);
+
+      per_inferior->disp_step_buf.emplace (disp_step_buf_addr);
+    }
+
+  return per_inferior->disp_step_buf->prepare (thread, displaced_pc);
+}
+
+/* Implementation of gdbarch_displaced_step_finish.  */
+
+static displaced_step_finish_status
+ppc_displaced_step_finish (gdbarch *arch, thread_info *thread,
+			   gdb_signal sig)
+{
+  ppc_inferior_data *per_inferior = get_ppc_per_inferior (thread->inf);
+
+  gdb_assert (per_inferior->disp_step_buf.has_value ());
+
+  return per_inferior->disp_step_buf->finish (arch, thread, sig);
+}
+
+/* Implementation of gdbarch_displaced_step_restore_all_in_ptid.  */
+
+static void
+ppc_displaced_step_restore_all_in_ptid (inferior *parent_inf, ptid_t ptid)
+{
+  ppc_inferior_data *per_inferior = ppc_inferior_data_key.get (parent_inf);
+
+  if (per_inferior == nullptr
+      || !per_inferior->disp_step_buf.has_value ())
+    return;
+
+  per_inferior->disp_step_buf->restore_in_ptid (ptid);
+}
+
 /* Always use hardware single-stepping to execute the
    displaced instruction.  */
 static bool
@@ -6990,8 +7062,10 @@ rs6000_gdbarch_init (struct gdbarch_info info, struct gdbarch_list *arches)
   set_gdbarch_displaced_step_hw_singlestep (gdbarch,
 					    ppc_displaced_step_hw_singlestep);
   set_gdbarch_displaced_step_fixup (gdbarch, ppc_displaced_step_fixup);
-  set_gdbarch_displaced_step_location (gdbarch,
-				       displaced_step_at_entry_point);
+  set_gdbarch_displaced_step_prepare (gdbarch, ppc_displaced_step_prepare);
+  set_gdbarch_displaced_step_finish (gdbarch, ppc_displaced_step_finish);
+  set_gdbarch_displaced_step_restore_all_in_ptid
+    (gdbarch, ppc_displaced_step_restore_all_in_ptid);
 
   set_gdbarch_max_insn_length (gdbarch, PPC_INSN_SIZE);
 
diff --git a/gdb/s390-linux-tdep.c b/gdb/s390-linux-tdep.c
index 8020832d978..14e92d2c6f3 100644
--- a/gdb/s390-linux-tdep.c
+++ b/gdb/s390-linux-tdep.c
@@ -1119,7 +1119,7 @@ s390_linux_init_abi_any (struct gdbarch_info info, struct gdbarch *gdbarch)
 
   tdep->s390_syscall_record = s390_linux_syscall_record;
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Register handling.  */
   set_gdbarch_core_read_description (gdbarch, s390_core_read_description);
diff --git a/gdb/s390-tdep.c b/gdb/s390-tdep.c
index 5429d718c3b..76eeda64c7c 100644
--- a/gdb/s390-tdep.c
+++ b/gdb/s390-tdep.c
@@ -7047,7 +7047,10 @@ s390_gdbarch_init (struct gdbarch_info info, struct gdbarch_list *arches)
   set_gdbarch_displaced_step_copy_insn (gdbarch,
 					s390_displaced_step_copy_insn);
   set_gdbarch_displaced_step_fixup (gdbarch, s390_displaced_step_fixup);
-  set_gdbarch_displaced_step_location (gdbarch, linux_displaced_step_location);
+  set_gdbarch_displaced_step_prepare (gdbarch, linux_displaced_step_prepare);
+  set_gdbarch_displaced_step_finish (gdbarch, linux_displaced_step_finish);
+  set_gdbarch_displaced_step_restore_all_in_ptid
+    (gdbarch, linux_displaced_step_restore_all_in_ptid);
   set_gdbarch_displaced_step_hw_singlestep (gdbarch, s390_displaced_step_hw_singlestep);
   set_gdbarch_software_single_step (gdbarch, s390_software_single_step);
   set_gdbarch_max_insn_length (gdbarch, S390_MAX_INSTR_SIZE);
diff --git a/gdb/sh-linux-tdep.c b/gdb/sh-linux-tdep.c
index 91272001561..b7c66b70294 100644
--- a/gdb/sh-linux-tdep.c
+++ b/gdb/sh-linux-tdep.c
@@ -184,7 +184,7 @@ static struct tramp_frame sh_linux_rt_sigreturn_tramp_frame = {
 static void
 sh_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* GNU/Linux uses SVR4-style shared libraries.  */
   set_gdbarch_skip_trampoline_code (gdbarch, find_solib_trampoline_target);
diff --git a/gdb/sparc-linux-tdep.c b/gdb/sparc-linux-tdep.c
index 693ca6d8e34..3dbc65ca24d 100644
--- a/gdb/sparc-linux-tdep.c
+++ b/gdb/sparc-linux-tdep.c
@@ -422,7 +422,7 @@ sparc32_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   tdep->gregset = &sparc32_linux_gregset;
   tdep->sizeof_gregset = 152;
diff --git a/gdb/sparc64-linux-tdep.c b/gdb/sparc64-linux-tdep.c
index a7bb64a1673..10a6eb52778 100644
--- a/gdb/sparc64-linux-tdep.c
+++ b/gdb/sparc64-linux-tdep.c
@@ -365,7 +365,7 @@ sparc64_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   tdep->gregset = &sparc64_linux_gregset;
   tdep->sizeof_gregset = 288;
diff --git a/gdb/testsuite/ChangeLog b/gdb/testsuite/ChangeLog
index d3350bcc506..2c4553d6448 100644
--- a/gdb/testsuite/ChangeLog
+++ b/gdb/testsuite/ChangeLog
@@ -1,5 +1,11 @@
 2020-12-04  Simon Marchi  <simon.marchi@efficios.com>
 
+	* gdb.arch/amd64-disp-step-avx.exp: Adjust pattern.
+	* gdb.threads/forking-threads-plus-breakpoint.exp: Likewise.
+	* gdb.threads/non-stop-fair-events.exp: Likewise.
+
+2020-12-04  Simon Marchi  <simon.marchi@efficios.com>
+
 	* gdb.threads/step-over-exec.exp: New.
 	* gdb.threads/step-over-exec.c: New.
 	* gdb.threads/step-over-exec-execd.c: New.
diff --git a/gdb/testsuite/gdb.arch/amd64-disp-step-avx.exp b/gdb/testsuite/gdb.arch/amd64-disp-step-avx.exp
index 5dd827a40ae..a2cd682d543 100644
--- a/gdb/testsuite/gdb.arch/amd64-disp-step-avx.exp
+++ b/gdb/testsuite/gdb.arch/amd64-disp-step-avx.exp
@@ -97,7 +97,7 @@ proc disp_step_func { func } {
     gdb_test_no_output "set debug displaced on"
 
     gdb_test "continue" \
-	"Continuing.*displaced pc to.*Breakpoint.*, ${test_end_label} ().*" \
+	"Continuing.*prepared successfully .*Breakpoint.*, ${test_end_label} ().*" \
 	"continue to ${test_end_label}"
 
     gdb_test_no_output "set debug displaced off"
diff --git a/gdb/testsuite/gdb.threads/forking-threads-plus-breakpoint.exp b/gdb/testsuite/gdb.threads/forking-threads-plus-breakpoint.exp
index c4c9596be31..4ce30154bdf 100644
--- a/gdb/testsuite/gdb.threads/forking-threads-plus-breakpoint.exp
+++ b/gdb/testsuite/gdb.threads/forking-threads-plus-breakpoint.exp
@@ -48,7 +48,7 @@ proc probe_displaced_stepping_support {} {
 	# that breakpoint.
 	gdb_test_no_output "set debug displaced 1"
 	gdb_test_multiple "next" "probe" {
-	    -re "displaced pc to.*$gdb_prompt $" {
+	    -re "prepared successfully .*$gdb_prompt $" {
 		pass "supported"
 	    }
 	    -re ".*$gdb_prompt $" {
diff --git a/gdb/testsuite/gdb.threads/non-stop-fair-events.exp b/gdb/testsuite/gdb.threads/non-stop-fair-events.exp
index bc1351f688e..11776616150 100644
--- a/gdb/testsuite/gdb.threads/non-stop-fair-events.exp
+++ b/gdb/testsuite/gdb.threads/non-stop-fair-events.exp
@@ -69,7 +69,7 @@ set displaced_stepping_enabled 0
 set msg "check displaced-stepping"
 gdb_test_no_output "set debug displaced 1"
 gdb_test_multiple "next" $msg {
-    -re "displaced pc to.*$gdb_prompt $" {
+    -re "prepared successfully .*$gdb_prompt $" {
 	set displaced_stepping_enabled 1
     }
     -re ".*$gdb_prompt $" {
diff --git a/gdb/thread.c b/gdb/thread.c
index 2bbb016d4d9..856bdee97b3 100644
--- a/gdb/thread.c
+++ b/gdb/thread.c
@@ -362,10 +362,10 @@ step_over_chain_enqueue (struct thread_info **list_p, struct thread_info *tp)
     }
 }
 
-/* Remove TP from step-over chain LIST_P.  */
+/* See gdbthread.h.  */
 
-static void
-thread_step_over_chain_remove (struct thread_info **list_p, struct thread_info *tp)
+void
+thread_step_over_chain_remove (thread_info **list_p, thread_info *tp)
 {
   gdb_assert (tp->step_over_next != NULL);
   gdb_assert (tp->step_over_prev != NULL);
@@ -385,12 +385,20 @@ thread_step_over_chain_remove (struct thread_info **list_p, struct thread_info *
 
 /* See gdbthread.h.  */
 
+thread_info *
+thread_step_over_chain_next (thread_info *chain_head, thread_info *tp)
+{
+  thread_info *next = tp->step_over_next;
+
+  return next == chain_head ? NULL : next;
+}
+
+/* See gdbthread.h.  */
+
 struct thread_info *
 global_thread_step_over_chain_next (struct thread_info *tp)
 {
-  struct thread_info *next = tp->step_over_next;
-
-  return (next == global_thread_step_over_chain_head ? NULL : next);
+  return thread_step_over_chain_next (global_thread_step_over_chain_head, tp);
 }
 
 /* See gdbthread.h.  */
@@ -403,17 +411,65 @@ thread_is_in_step_over_chain (struct thread_info *tp)
 
 /* See gdbthread.h.  */
 
+int
+thread_step_over_chain_length (thread_info *tp)
+{
+  if (tp == nullptr)
+    return 0;
+
+  gdb_assert (thread_is_in_step_over_chain (tp));
+
+  int num = 1;
+
+  for (thread_info *iter = tp->step_over_next;
+       iter != tp;
+       iter = iter->step_over_next)
+    ++num;
+
+  return num;
+}
+
+/* See gdbthread.h.  */
+
 void
 global_thread_step_over_chain_enqueue (struct thread_info *tp)
 {
+  infrun_debug_printf ("enqueueing thread %s in global step over chain",
+		       target_pid_to_str (tp->ptid).c_str ());
+
   step_over_chain_enqueue (&global_thread_step_over_chain_head, tp);
 }
 
 /* See gdbthread.h.  */
 
 void
+global_thread_step_over_chain_enqueue_chain (thread_info *chain_head)
+{
+  gdb_assert (chain_head->step_over_next != nullptr);
+  gdb_assert (chain_head->step_over_prev != nullptr);
+
+  if (global_thread_step_over_chain_head == nullptr)
+    global_thread_step_over_chain_head = chain_head;
+  else
+    {
+      thread_info *global_last = global_thread_step_over_chain_head->step_over_prev;
+      thread_info *chain_last = chain_head->step_over_prev;
+
+      chain_last->step_over_next = global_thread_step_over_chain_head;
+      global_last->step_over_next = chain_head;
+      global_thread_step_over_chain_head->step_over_prev = chain_last;
+      chain_head->step_over_prev = global_last;
+    }
+}
+
+/* See gdbthread.h.  */
+
+void
 global_thread_step_over_chain_remove (struct thread_info *tp)
 {
+  infrun_debug_printf ("removing thread %s from global step over chain",
+		       target_pid_to_str (tp->ptid).c_str ());
+
   thread_step_over_chain_remove (&global_thread_step_over_chain_head, tp);
 }
 
diff --git a/gdb/tic6x-linux-tdep.c b/gdb/tic6x-linux-tdep.c
index 1b626b5a174..5a8d7c7f825 100644
--- a/gdb/tic6x-linux-tdep.c
+++ b/gdb/tic6x-linux-tdep.c
@@ -167,7 +167,7 @@ tic6x_uclinux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   struct gdbarch_tdep *tdep = gdbarch_tdep (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   /* Shared library handling.  */
   set_solib_ops (gdbarch, &dsbt_so_ops);
diff --git a/gdb/tilegx-linux-tdep.c b/gdb/tilegx-linux-tdep.c
index 18dfe8892d8..14cfafe33de 100644
--- a/gdb/tilegx-linux-tdep.c
+++ b/gdb/tilegx-linux-tdep.c
@@ -111,7 +111,7 @@ tilegx_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
 {
   int arch_size = gdbarch_addr_bit (gdbarch);
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   tramp_frame_prepend_unwinder (gdbarch, &tilegx_linux_rt_sigframe);
 
diff --git a/gdb/xtensa-linux-tdep.c b/gdb/xtensa-linux-tdep.c
index b419970ecea..c2aeb8e9397 100644
--- a/gdb/xtensa-linux-tdep.c
+++ b/gdb/xtensa-linux-tdep.c
@@ -110,7 +110,7 @@ xtensa_linux_init_abi (struct gdbarch_info info, struct gdbarch *gdbarch)
       set_gdbarch_num_pseudo_regs (gdbarch, tdep->num_pseudo_regs);
     }
 
-  linux_init_abi (info, gdbarch);
+  linux_init_abi (info, gdbarch, false);
 
   set_solib_svr4_fetch_link_map_offsets
     (gdbarch, svr4_ilp32_fetch_link_map_offsets);