Changing the Output Mode

When debugging across multiple ranks, mdb can produce a large amount of output. By default, each rank’s output is displayed independently, separated by divider lines. This is called separate mode and is the default behaviour.

When the output from all ranks is identical, this can be redundant. The combined output mode reduces this verbosity by merging common lines across ranks into a single block, prefixing them with a combined rank range.

This tutorial demonstrates the difference between the two modes using the set output command.

The Example Program

We will use the simple-mpi.exe binary from the examples/ directory (see Quick Start for details on compiling it).

Launching the Debug Target

First, launch the program with two processes:

$ mdb launch -n 2 -t ./simple-mpi.exe

This starts the debug server and waits for an mdb attach connection.

Separate Output Mode (Default)

In separate mode, each rank’s output is printed independently. The output from each rank is prefixed with its rank ID (e.g., 0: or 1:) and blocks for different ranks are separated by a line of asterisks (****).

Create a script file script-separate.mdb:

set output separate
broadcast start
info proc
break 15
continue
list
continue
quit
quit

Run it with:

$ mdb attach -x script-separate.mdb > separate.out

Here is a shortened excerpt of the output from the info proc and list commands:

info proc in separate mode:

0: process 44900
0: cmdline = '/home/melt/sync/cambridge/projects/side/mdb/examples/simple-mpi.exe'
0: cwd = '/home/melt/sync/cambridge/projects/side/mdb'
0: exe = '/home/melt/sync/cambridge/projects/side/mdb/examples/simple-mpi.exe'
************************************************************************
1: process 44903
1: cmdline = '/home/melt/sync/cambridge/projects/side/mdb/examples/simple-mpi.exe'
1: cwd = '/home/melt/sync/cambridge/projects/side/mdb'
1: exe = '/home/melt/sync/cambridge/projects/side/mdb/examples/simple-mpi.exe'

Each rank’s output appears in its own block, prefixed with just the rank number, separated by **** dividers.

list in separate mode:

0: 10
0: 11   call mpi_init(ierror)
0: 12   call mpi_comm_size(mpi_comm_world, size_of_cluster, ierror)
0: 13   call mpi_comm_rank(mpi_comm_world, process_rank, ierror)
0: 14
0: 15   var = 10.*process_rank
0: 16
0: 17   if (process_rank == 0) then
0: 18     print *, 'process 0 sleeping for 3s...'
0: 19     do i = 1, 3
************************************************************************
1: 10
1: 11   call mpi_init(ierror)
1: 12   call mpi_comm_size(mpi_comm_world, size_of_cluster, ierror)
1: 13   call mpi_comm_rank(mpi_comm_world, process_rank, ierror)
1: 14
1: 15   var = 10.*process_rank
1: 16
1: 17   if (process_rank == 0) then
1: 18     print *, 'process 0 sleeping for 3s...'
1: 19     do i = 1, 3

Since the source code is identical across ranks, the list output is duplicated for each rank.

Combined Output Mode

In combined mode, lines that are identical across all ranks are merged and prefixed with a combined rank range (e.g., 0-1:). Lines that differ between ranks are still shown per-rank with a single rank prefix (e.g., `` 0:). No ``**** dividers are used.

Create a script file script-combined.mdb:

set output combined
broadcast start
info proc
break 15
continue
list
continue
quit
quit

Run it with:

$ mdb attach -x script-combined.mdb > combined.out

info proc in combined mode:

  0: process 45004
0-1: cmdline = '/home/melt/sync/cambridge/projects/side/mdb/examples/simple-mpi.exe'
0-1: cwd = '/home/melt/sync/cambridge/projects/side/mdb'
0-1: exe = '/home/melt/sync/cambridge/projects/side/mdb/examples/simple-mpi.exe'
  1: process 45005

Notice how the three identical lines (cmdline, cwd, exe) are merged under a single 0-1: prefix. The process line differs between ranks (different PIDs) so it is shown individually for each rank.

list in combined mode:

0-1: 10
0-1: 11   call mpi_init(ierror)
0-1: 12   call mpi_comm_size(mpi_comm_world, size_of_cluster, ierror)
0-1: 13   call mpi_comm_rank(mpi_comm_world, process_rank, ierror)
0-1: 14
0-1: 15   var = 10.*process_rank
0-1: 16
0-1: 17   if (process_rank == 0) then
0-1: 18     print *, 'process 0 sleeping for 3s...'
0-1: 19     do i = 1, 3

Since all ranks share the same source code, every line is identical and merged into a single compact block. This avoids the duplication seen in separate mode.