From 2D landmarks to 3D orientantions: the araponga workflow

library(araponga)

Overview

araponga estimates possible 3D orientations of objects from 2D landmarks in images or videos.

The central challenge is that a 2D projection does not uniquely determine a 3D angle: the same apparent orientation in an image can be produced by many combinations of true orientation, camera position, and viewing angle. Instead of returning a single potentially misleading estimate, araponga returns the set of 3D angles consistent with the available information.

This vignette introduces the main araponga workflow using a real example: estimating how widely bare-throated bellbirds — “araponga”, in Brazil — open their beaks while producing some of the loudest sounds recorded among birds.

The general workflow is:

2D landmarks
  -> projected 2D pitch
    -> candidate 3D configurations
      -> compatible 3D angles
        -> optional filtering using system-specific information

Before getting to the case study, let’s define a few angle conventions used throughout the package.

Angle conventions

We call pitch the up–down orientation of an object. Pitch varies from -90° (pointed straight down), through 0° (horizontal), to 90° (pointed straight up). For example, this is a pitch of 45°:

plot.angles(45, type = "pitch")

We call yaw the left–right orientation of an object. Yaw varies over the interval (-180°, 180°]: -90° means pointed toward the camera, 0° means pointed right, 90° means pointed away from the camera, and 180° means pointed left. For example, this is a yaw of -80°:

plot.angles(-80, type = "yaw")

Finally, view elevation is the vertical angle of the camera relative to the object. It varies from -90° (viewed from directly below), through 0° (eye level), to 90° (viewed from directly above). For example:

plot.angles(-15, type = "view_elevation")

You can remind yourself of the package conventions at any time with conventions().

With that clarified, let’s move on to the case study, and the reason why the package was originally developed.

Case study: estimating beak gape angle in bellbirds

Note The main functions in this package rely on precomputed simulation data. Run download.simdata() once before using the package for the first time.

During my PhD, I wanted to find out how widely bare-throated bellbirds open their beaks, and whether extreme gape helps them produce some of the loudest vocalizations among terrestrial animals.

To answer this, I recorded videos of singing male bellbirds in the Atlantic Forest of Brazil. Here is one video frame of a male singing at his display perch near Macuquinho Lodge:

$Male bare-throated bellbird singing in Brazil.$

Male bare-throated bellbird singing in Brazil.

How widely is he opening his beak? Let’s find out.

The first step is to identify landmarks. They are highlighted in the video frame: tip (dark) and base (light) of the upper (blue) and lower (red) mandibles. Here are their coordinates, obtained with labeling software:

Note In araponga, x coordinates increase to the right and y coordinates increase upward. Some labeling software uses different coordinate systems.

Important The camera is assumed to be horizontally aligned. If the image is tilted because the camera was rolled clockwise or counterclockwise, rotate/correct the frame or coordinates before analysis.

# upper tip
x_ut <- -718.5216072
y_ut <- -78.5485277

# upper base
x_ub <- -782.5250905
y_ub <- -105.2195211

# lower tip
x_lt <- -689.0089234
y_lt <- -210.4188121

# lower base
x_lb <- -751.2784311
y_lb <- -178.5397719

Before estimating gape angle, we need to first estimate the 3D pitch of each mandible, then calculate the difference between them. We’ll start with the upper mandible pitch.

Step 1: projected 2D pitch

The starting point is the apparent pitch visible in the image, which can be extracted with pitch2d.from.xy():

upper_p2d <- pitch2d.from.xy(x_ut, y_ut, x_ub, y_ub, plot = TRUE)

If the bird had been filmed perfectly side-on and at eye level, this 2D projection would already equal the true 3D pitch:

cat(upper_p2d)
#> 22.62206

We can verify this with find.pitch() by specifying candidate_view_elevations = 0 (eye-level) and candidate_yaws = 0 (pointed straight right). This function necessarily takes some amount of labeling error, which we’ll assume to be ± 1 pixel for this exercise.

Note Angle values — input and output — are treated at 1° resolution, so small numerical differences should not be overinterpreted.

Important The returned angles are not “the true angle”; they are the angles compatible with the observed 2D projection, labeling error, and candidate constraints.

upper_pitch <- find.pitch(upper_p2d,
                   candidate_view_elevations = 0,
                   candidate_yaws = 0,
                   label_error = 1)

upper_pitch
#> [1] 21 22 23 24
plot.angles(upper_pitch, "pitch")

Assuming ±1 pixel of labeling uncertainty, the upper mandible pitch would be between 21° and 24°.

Step 2: dealing with real camera angles

But of course that’s not real life — at least not for field biologists. Just by looking at the frame, we can tell that the bird is neither exactly eye-level nor exactly side-on.

If we provide no information about viewing geometry, many 3D pitch orientations remain possible:

upper_pitch <- find.pitch(upper_p2d,
                   label_error = 1)

upper_pitch
#>   [1] -89 -88 -87 -86 -85 -84 -83 -82 -81 -80 -79 -78 -77 -76 -75 -74 -73 -72
#>  [19] -71 -70 -69 -68 -67 -66 -65 -64 -63 -62 -61 -60 -59 -58 -57 -56 -55 -54
#>  [37] -53 -52 -51 -50 -49 -48 -47 -46 -45 -44 -43 -42 -41 -40 -39 -38 -37 -36
#>  [55] -35 -34 -33 -32 -31 -30 -29 -28 -27 -26 -25 -24 -23 -22 -21 -20 -19 -18
#>  [73] -17 -16 -15 -14 -13 -12 -11 -10  -9  -8  -7  -6  -5  -4  -3  -2  -1   0
#>  [91]   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18
#> [109]  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36
#> [127]  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54
#> [145]  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72
#> [163]  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89
plot.angles(upper_pitch, "pitch")

The result is almost completely unconstrained: the mandible could plausibly point anywhere from nearly straight down to nearly straight up. This is expected. A single 2D projection alone cannot resolve 3D orientation.

The power of araponga comes from adding whatever information you have.

Step 3: adding known view elevation

In this case, I measured my filming angle with a laser range finder:

measured_elevation <- -34.2

plot.angles(measured_elevation,
            type = "view_elevation",
            main = "measured view elevation")

Including this information narrows the possible pitches:

upper_pitch <- find.pitch(upper_p2d,
                   candidate_view_elevations = measured_elevation,
                   label_error = 1)

upper_pitch
#>  [1] -33 -32 -31 -30 -29 -28 -27 -26 -25 -24 -23 -22 -21 -20 -19 -18 -17 -16 -15
#> [20] -14 -13 -12 -11 -10  -9  -8  -7  -6  -5  -4  -3  -2  -1   0   1   2   3   4
#> [39]   5   6   7   8   9  10  11  12  13  14  15  16  17  18  19  20  21  22  23
#> [58]  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41
plot.angles(upper_pitch, "pitch")

Step 4: adding approximate yaw

Although I did not know the exact yaw of the bird, the image provides useful information: the beak appears to point somewhere between side-on and slightly facing the camera.

We can encode this uncertainty as a range:

assumed_yaw <- -45:0

plot.angles(assumed_yaw,
            type = "yaw",
            labels = TRUE,
            main = "assumed yaw range")

and include it for pitch estimation:

upper_pitch <- find.pitch(upper_p2d,
                   candidate_view_elevations = measured_elevation,
                   candidate_yaws = assumed_yaw,
                   label_error = 1)

upper_pitch
#>  [1] -8 -7 -6 -5 -4 -3 -2 -1  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16
#> [26] 17 18 19 20 21 22 23 24 25 26 27 28 29
plot.angles(upper_pitch, "pitch")

Now the possible pitch range is much narrower. Importantly, this is not because araponga became more certain — it is because we provided additional assumptions.

Let’s repeat the process for the lower mandible:

lower_p2d <- pitch2d.from.xy(x_lt, y_lt, x_lb, y_lb)

lower_pitch <- find.pitch(lower_p2d,
                   candidate_view_elevations = measured_elevation,
                   candidate_yaws = assumed_yaw,
                   label_error = 1)

lower_pitch
#>  [1] -43 -42 -41 -40 -39 -38 -37 -36 -35 -34 -33 -32 -31 -30

and visualize both estimates:

plot.angles(upper_pitch, "pitch")
plot.angles(lower_pitch, "pitch", add = TRUE)

Step 5: calculating gape angle

Finally, gape is the difference between upper- and lower-mandible pitch:

gapes <- sort(unique(as.vector(outer(upper_pitch, lower_pitch, "-"))))
gapes
#>  [1] 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
#> [26] 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
#> [51] 72

Given our assumptions, the possible gape angles range from 22° to 72°.

This calculation allows any possible upper mandible pitch to combine with any possible lower mandible pitch. If we assume the beak opens in a vertical plane — i.e., if the head is not tilted sideways, a reasonable assumption for singing bellbirds —, we can extract possible gapes on a per-yaw basis. The paired argument in find.pitch() facilitates that:

upper_pitch_paired <- find.pitch(upper_p2d,
           candidate_view_elevations = measured_elevation,
           candidate_yaws = assumed_yaw,
           label_error = 1,
           paired = TRUE)

lower_pitch_paired <- find.pitch(lower_p2d,
           candidate_view_elevations = measured_elevation,
           candidate_yaws = assumed_yaw,
           label_error = 1,
           paired = TRUE)

paired = TRUE gives us a data frame with the possible pitches for each possible yaw. Let’s extract per-yaw possible gapes and pool them:

all_yaws <- unique(c(upper_pitch_paired$yaw, lower_pitch_paired$yaw))

per_yaw_gapes <- NULL
for(y in all_yaws){
  
  rel_upper_pitch <- upper_pitch_paired$pitch[upper_pitch_paired$yaw == y]
  rel_lower_pitch <- lower_pitch_paired$pitch[lower_pitch_paired$yaw == y]
  per_yaw_gapes <- c(per_yaw_gapes,
                     unique(as.vector(outer(rel_upper_pitch, rel_lower_pitch,"-"))))
}

per_yaw_gapes <- sort(unique(per_yaw_gapes))

per_yaw_gapes
#>  [1] 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58
#> [26] 59 60 61 62 63

Gape is now estimated between 34° and 63°.

What if I’m interested in yaw?

We can apply a similar workflow to estimate the yaw at which the bellbird is singing in the frame. The input is still the projected 2D pitch, but now we ask which yaws — rather than which pitches — are compatible with it.

To do so, we use find.yaw() — which, like find.pitch(), is a convenient wrapper to the generic find.3d(). Let’s start by assuming the upper mandible always opens approximately upward:

upper_yaw <- find.yaw(upper_p2d,
           candidate_view_elevations = measured_elevation,
           candidate_pitches = -20:90,
           label_error = 1)

upper_yaw
#>   [1] -64 -63 -62 -61 -60 -59 -58 -57 -56 -55 -54 -53 -52 -51 -50 -49 -48 -47
#>  [19] -46 -45 -44 -43 -42 -41 -40 -39 -38 -37 -36 -35 -34 -33 -32 -31 -30 -29
#>  [37] -28 -27 -26 -25 -24 -23 -22 -21 -20 -19 -18 -17 -16 -15 -14 -13 -12 -11
#>  [55] -10  -9  -8  -7  -6  -5  -4  -3  -2  -1   0   1   2   3   4   5   6   7
#>  [73]   8   9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25
#>  [91]  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43
#> [109]  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61
#> [127]  62  63  64  65  66  67  68  69  70  71  72  73  74  75  76  77  78  79
#> [145]  80  83  84  87
plot.angles(upper_yaw, "yaw")

and the lower mandible approximately downward:

lower_yaw <- find.yaw(lower_p2d,
           candidate_view_elevations = measured_elevation,
           candidate_pitches = -90:20,
           label_error = 1)

lower_yaw
#>   [1] -85 -82 -80 -79 -78 -76 -75 -74 -73 -72 -71 -70 -69 -68 -67 -66 -65 -64
#>  [19] -63 -62 -61 -60 -59 -58 -57 -56 -55 -54 -53 -52 -51 -50 -49 -48 -47 -46
#>  [37] -45 -44 -43 -42 -41 -40 -39 -38 -37 -36 -35 -34 -33 -32 -31 -30 -29 -28
#>  [55] -27 -26 -25 -24 -23 -22 -21 -20 -19 -18 -17 -16 -15 -14 -13 -12 -11 -10
#>  [73]  -9  -8  -7  -6  -5  -4  -3  -2  -1   0   1   2   3   4   5   6   7   8
#>  [91]   9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26
#> [109]  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44
#> [127]  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62
#> [145]  63  64  65  66  67
plot.angles(lower_yaw, "yaw")

Under the no-head-tilt assumption, they should also share the same yaw. We can therefore restrict possible yaws to values compatible with both mandibles:

both_yaw <- intersect(lower_yaw, upper_yaw)
plot.angles(both_yaw, "yaw")

We can go one step further. Some yaw values may be possible for both mandibles independently, but only under pitch combinations that are biologically unrealistic. For example, a yaw that only allows negative gape angles would imply the lower mandible is above the upper mandible.

To test this, we can again use paired = TRUE to preserve which pitches are possible under each yaw:

upper_yaw_paired <- find.yaw(upper_p2d,
           candidate_view_elevations = measured_elevation,
           candidate_pitches = -20:90,
           label_error = 1,
           paired = TRUE)

lower_yaw_paired <- find.yaw(lower_p2d,
           candidate_view_elevations = measured_elevation,
           candidate_pitches = -90:20,
           label_error = 1,
           paired = TRUE)

max_gape_per_yaw <- data.frame(yaw = both_yaw,
                               gape = NA)

for(i in seq_along(max_gape_per_yaw$yaw)){
  rel_yaw <- max_gape_per_yaw$yaw[i]
  
  rel_upper_pitch <- upper_yaw_paired$pitch[upper_yaw_paired$yaw == rel_yaw]
  rel_lower_pitch <- lower_yaw_paired$pitch[lower_yaw_paired$yaw == rel_yaw]
  
  max_gape_per_yaw$gape[i] <- max(outer(rel_upper_pitch, rel_lower_pitch, "-"))
}

yaw <- max_gape_per_yaw$yaw[max_gape_per_yaw$gape >= 0]

any(max_gape_per_yaw$gape < 0)
#> [1] FALSE

No yaw was excluded in this example, but this illustrates how information about your system can be used to remove unrealistic orientations.

Another way to narrow down yaw is by comparing multiple observations. Suppose that, in the next frame, you can visually tell that the bird rotates its head right (clockwise in araponga convention) by no more than 90°. Suppose also that possible yaws for that frame are estimated as:

next_frame_yaw <- -50:90

We can use trim.yaws() to keep only yaw values consistent with this movement:

# red = excluded; green = retained
trimmed <- trim.yaws(ccw_yaws = yaw,
                     cw_yaws = next_frame_yaw,
                     min_sep = 0,
                     max_sep = 90,
                     plot = TRUE)


yaw <- trimmed$trimmed_ccw_yaws
next_frame_yaw <- trimmed$trimmed_cw_yaws

The logic is the same throughout araponga: start with the projected 2D pitch, then use available information to determine which 3D configurations are compatible with it.

Choosing candidate values

The candidate_... arguments are the main way of telling araponga what orientations are plausible for your system.

There is no universally correct range: the best choice depends on what information you have before estimating the angle of interest.

For example, candidate values can account for measurement uncertainty:

# view elevation measured as -30 degrees, allowed ±5 degrees measurement error
candidate_view_elevations = -35:-25

They can represent approximate field observations:

# camera approximately level with the object
candidate_view_elevations = -20:20

# object approximately side-on facing left
candidate_yaws = c(-160:-179, 160:180)

# object generally facing the camera
candidate_yaws = -120:-60

or incorporate anatomical or behavioral constraints:

# structure cannot realistically point downward
candidate_pitches = 0:90

# lower mandible of a bird opening mostly downward
candidate_pitches = -90:20

Broader candidate ranges make fewer assumptions but return more uncertainty. Narrower ranges return more precise estimates, but only if the assumptions behind them are justified.