Re-starting from Square One, CTPM Applied Skills

It has been months since the initial lecture introducing Camera Tracking.  In order to re-acquaint myself with the basics of this process and the algorithm of developing even the simplest pipeline that could generate specifically a 3D track (and also insert geometry), I opted to mentally bracket the two pipelines as provided by our coursework and see if I could generate a rudimentary “bare bones” 3D camera track pipeline independently.

As below, using the term2 alleyway footage provided in class, and then employing the Analyze, Solve, and ‘Create Scene Camera’ (and create cube) options from the 8 steps as summarised above, I was able to generate a functional bare bones 3D tracking pipeline which follows an (imported) cube thru the scene.   
(of note, the following screen capture was recorded w the internal HP screen record software, and for whatever reason it doesn't record some of the Nuke operations, e.g. the 'tab' procedure for invoking nodes is apparently invisible to the recording software)

Term Two Lecture: A More Elaborate Pipeline

As presented below, the current (Term2) review of 3D tracking (CTPM) involves a more elaborate pipeline and as presented further below, additional points that extend the content from our first lecture.  With the problematic covid structure permeating the educational efficiency of the term at hand, I did not practice with these issues, but do review them below so that I am generally familiar w these points as potential tracking issues. 

​

• Feature Point Editing


• As summarized by Aldo, 3D tracking works by triangulation.  Thus, an important issue to address prior to conducting a track is to evaluate whether any of the feature points have been automatically placed at the juncture of 2 noncontinuous surfaces that may create notable contrast at a couple frames in the clip, but will eventually prove to serve as poor tracking points because the image and perspective at the juncture point will change as the footage moves forward.  This is apparent at the cursor in the IMG below.   Because this tracking point lies precisely at the interface of the ground and a wall that happens to actually be a few feet in front and away from that grounding point, this tracker point needs to be removed prior to conducting the track.

​Contrast at Distant Overlapping Surfaces that do Contact: Another Landmine 

Beyond merely removing a specific feature point of concern, entire edge transitions can be removed before the tracking solve is implemented.  The point is to assure that what is being used as a feature point is truly one continuous independent solid object, not merely the contrast as might be introduced within a 2D scene  by transition from one surface to another distant surface.  This would involve a change in the image and perspective as the camera continues through the scene and thus would obviously not be a valid tracking feature point.  As highlighted in the following IMG, in the frame on the L, the roto effort relevant to eliminating the contrast edge between the wooden wall and the background wall is used to delete problematic tracking points.  The same basic issue underlies the contrast as created by the puddle on the street.  Specifically, as the scene progresses, the reflection IMG within the puddle will change and thus lose its capacity to serve as a grounded stable contrast or stationary feature point (R frame in IMG below).

​White Walls

In addition to the aforementioned issues, it is notable that white walls can be problematic for the tracker, thus the WW was also removed by roto (note the red retcangle at edges of the top of the frame). 

The More Elaborate Pipeline, Outcome of 'Solve'

Despite the attention to blocking out all the aforementioned sources of problematic tracking points, the error from the solve of the track proved to be quite substantial (3.38).  This is apparent from the below IMG which shows many red and some orange streaks.  The orange streaks are more apparent in the 2nd IMG below, they are removed by means of the ‘delete unsolved’ option.  The delete rejected option removes the red streaks.

Refinement

As a supplement to the tactics for addressing a poor solve (i.e. high error) that were outlined in term 1, Aldo commented on the benefit of the Refinement option, which includes selecting focal length, position, and rotation from the Refinement section of Autotracks, and then selecting Refine solve (note bottom of following IMG).

          Tracker Cones: Applied Skills

To produce a scene w tracking cones targeting specific axis points in the scene (note IMG below), an axis point is first introduced to the Scene (IMG on right).  As further detailed in the screen recording further below, the 'tracking gyzmo' is then used to create a tracker cone.  Next, the TAB key is used to create a Scene (type 'scene' into the TAB entry field), and finally, the cone is made upright by modifying the Z axis rotation.  
(upon review of the screen recording, as mentioned previously, certain functions of the Nuke software program are somehow invisible to the HP screen recording software, e.g. the 'create > axis' image as on the right will not be viewable in the screen recording...at the moment this occurs, what you will see is that functional outcomes will just appear on the screen for no apparent reason, but in reality, the recording software is simply not visualizing this particular operation)

---

​


Ground Plane

Before Exporting Scene, the group plane needs to be identified.  This requires a marquee of points on the ground, then R click, then choosing ground plane > set to selected.

​Card Texturing

To insert a card into the 2D scene, press the tab key and type card (for the 2D card node), and then use the properties bin to texture it (note the display and render drop downs).

Supplemental Research

Despite the unlikelihood of my becoming a compositor, the CTPM topic is certainly interesting.  This is further apparent upon searching the myriad of possible applications (e.g. physical pixels and augmented realty projection mapping).

Bibliography

Dobbert T, (2013). Chapter 1: The Basics of Matchmoving, in Matchmove: The Invisible Art of Camera Tracking. Sybex, John Wiley and Sons Publishers, Indianapolis, Indiana.

Physical Pixels:
http://www.interactivearchitecture.org/real-time-object-tracking-systems.html

Augmented Reality Projection Mapping:
https://www.arts.ac.uk/subjects/animation-interactive-film-and-sound/short-courses/interactive-design/spatial-augmented-reality-projections-with-madmapper-lcc