DataJoint U24 - Workflow DeepLabCut¶

Workflow Automation¶

In the previous notebook 03-Process, we ran through the workflow in detailed steps, manually adding each. The current notebook provides a more automated approach.

The commands here run a workflow using example data from the 00-DownloadData notebook, but note where placeholders could be changed for a different dataset.

In [1]:

            
                Copied!
                
                    
                    
                
                

        
import os; from pathlib import Path
# change to the upper level folder to detect dj_local_conf.json
if os.path.basename(os.getcwd())=='notebooks': os.chdir('..')
assert os.path.basename(os.getcwd())=='workflow-deeplabcut', ("Please move to the "
                                                              + "workflow directory")
from workflow_deeplabcut.pipeline import lab, subject, session, train, model
from workflow_deeplabcut import process
import os; from pathlib import Path
# change to the upper level folder to detect dj_local_conf.json
if os.path.basename(os.getcwd())=='notebooks': os.chdir('..')
assert os.path.basename(os.getcwd())=='workflow-deeplabcut', ("Please move to the "
                                                              + "workflow directory")
from workflow_deeplabcut.pipeline import lab, subject, session, train, model
from workflow_deeplabcut import process

Connecting cbroz@dss-db.datajoint.io:3306

We'll be using the process.py script to automatically loop through all make functions, as a shortcut for calling each individually.

If you previously completed the 03-Process notebook, you may want to delete the contents ingested there, to avoid duplication errors.

In [3]:

            
                Copied!
                
safemode=True # Set to false to turn off confirmation prompts
(session.Session & 'subject="subject6"').delete(safemode=safemode)
train.TrainingParamSet.delete(safemode=safemode)
train.VideoSet.delete(safemode=safemode)
safemode=True # Set to false to turn off confirmation prompts
(session.Session & 'subject="subject6"').delete(safemode=safemode)
train.TrainingParamSet.delete(safemode=safemode)
train.VideoSet.delete(safemode=safemode)

Deleting 4 rows from `u24_dlc_session`.`session_directory`
Deleting 4 rows from `u24_dlc_session`.`session_note`
Deleting 4 rows from `u24_dlc_session`.`session`
Deleting 3 rows from `u24_dlc_train`.`#training_param_set`
Deleting 0 rows from `u24_dlc_train`.`video_set`

Out[3]:

Ingestion of subjects, sessions, videos and training parameters¶

Refer to the user_data folder in the workflow.

Fill subject and session information in files subjects.csv and sessions.csv
Fill in recording and parameter information in recordings.csv and config_params.csv
- Add both training and estimation videos to the recording list
- Additional columns in config_params.csv will be treated as model training parameters
Run automatic scripts prepared in workflow_deeplabcut.ingest for ingestion:
- ingest_subjects for subject.Subject
- ingest_sessions - for session tables Session, SessionDirectory, and SessionNote
- ingest_dlc_items - for ...
  - train.ModelTrainingParamSet
  - train.VideoSet and the corresponding File part table
  - model.VideoRecording and the corresponding File part table

In [2]:

            
                Copied!
                
from workflow_deeplabcut.ingest import ingest_subjects, ingest_sessions, ingest_dlc_items
ingest_subjects()
ingest_sessions()
ingest_dlc_items()
from workflow_deeplabcut.ingest import ingest_subjects, ingest_sessions, ingest_dlc_items
ingest_subjects()
ingest_sessions()
ingest_dlc_items()

---- Inserting 0 entry(s) into subject ----

---- Inserting 4 entry(s) into session ----

---- Inserting 4 entry(s) into session_directory ----

---- Inserting 4 entry(s) into session_note ----

---- Inserting 3 entry(s) into #model_training_param_set ----

---- Inserting 3 entry(s) into video_set ----

---- Inserting 10 entry(s) into video_set__file ----

---- Inserting 1 entry(s) into video_recording ----

---- Inserting 1 entry(s) into video_recording__file ----

Setting project variables¶

Set your root directory in your DataJoint config file, under custom as dlc_root_data_dir.

In [3]:

            
                Copied!
                
import datajoint as dj; dj.config.load('dj_local_conf.json')
from element_interface.utils import find_full_path
data_dir = find_full_path(dj.config['custom']['dlc_root_data_dir'], # root from config
                          'from_top_tracking')                      # DLC project dir
config_path = (data_dir / 'config.yaml')
import datajoint as dj; dj.config.load('dj_local_conf.json')
from element_interface.utils import find_full_path
data_dir = find_full_path(dj.config['custom']['dlc_root_data_dir'], # root from config
                          'from_top_tracking')                      # DLC project dir
config_path = (data_dir / 'config.yaml')

Next, we pair training files with training parameters, and launch training via process.
- Some tables may try to populate without upstream keys.
- Others, like RecordingInfo already have keys loaded.
- Note: DLC's model processes (e.g., Training, Evaluation) log a lot of information to the console, to quiet this, pass verbose=False to process

In [ ]:

            
                Copied!
                
key={'paramset_idx':0,'training_id':0,'video_set_id':0, 
     'project_path':'from_top_tracking/'}
train.TrainingTask.insert1(key, skip_duplicates=True)
process.run(verbose=True, display_progress=True)
model.RecordingInfo()
key={'paramset_idx':0,'training_id':0,'video_set_id':0, 
     'project_path':'from_top_tracking/'}
train.TrainingTask.insert1(key, skip_duplicates=True)
process.run(verbose=True, display_progress=True)
model.RecordingInfo()

For the purposes of this demo, we'll want to use an older model, so the folling function will reload the original checkpoint file.

In [5]:

            
                Copied!
                
from workflow_deeplabcut.load_demo_data import revert_checkpoint_file
revert_checkpoint_file()
from workflow_deeplabcut.load_demo_data import revert_checkpoint_file
revert_checkpoint_file()

Now to add such a model upstream key
- Include a user-friendly model_name
- Include the relative path for the project's config.yaml
- Add shuffle and trainingsetindex
- insert_new_model will prompt before inserting, but this can be skipped with prompt=False

In [7]:

            
                Copied!
                
                    
                    
                
                

        
model.Model.insert_new_model(model_name='FromTop-latest', 
                             dlc_config=config_path,
                             shuffle=1,
                             trainingsetindex=0,
                             paramset_idx=1, 
                             prompt=True, # True is the default behavior
                             model_description='FromTop - latest snapshot',
                             params={"snapshotindex":-1})
process.run()
model.Model.insert_new_model(model_name='FromTop-latest', 
                             dlc_config=config_path,
                             shuffle=1,
                             trainingsetindex=0,
                             paramset_idx=1, 
                             prompt=True, # True is the default behavior
                             model_description='FromTop - latest snapshot',
                             params={"snapshotindex":-1})
process.run()

--- DLC Model specification to be inserted ---
	model_name: FromTop-latest
	model_description: FromTop - latest snapshot
	scorer: DLCmobnet100fromtoptrackingFeb23shuffle1
	task: from_top_tracking
	date: Feb23
	iteration: 0
	snapshotindex: -1
	shuffle: 1
	trainingsetindex: 0
	project_path: from_top_tracking
	paramset_idx: 1
	-- Template/Contents of config.yaml --
		Task: from_top_tracking
		scorer: DJ
		date: Feb23
		video_sets: {'/tmp/test_data/from_top_tracking/videos/train1.mp4': {'crop': '0, 500, 0, 500'}}
		bodyparts: ['head', 'bodycenter', 'tailbase']
		start: 0
		stop: 1
		numframes2pick: 20
		pcutoff: 0.6
		dotsize: 3
		alphavalue: 0.7
		colormap: viridis
		TrainingFraction: [0.95]
		iteration: 0
		default_net_type: resnet_50
		snapshotindex: -1
		batch_size: 8
		cropping: False
		x1: 0
		x2: 640
		y1: 277
		y2: 624
		corner2move2: [50, 50]
		move2corner: True
		croppedtraining: None
		default_augmenter: default
		identity: None
		maxiters: 5
		modelprefix: 
		multianimalproject: False
		scorer_legacy: False
		shuffle: 1
		skeleton: [['bodypart1', 'bodypart2'], ['objectA', 'bodypart3']]
		skeleton_color: black
		train_fraction: 0.95
		trainingsetindex: 0
		project_path: /tmp/test_data/from_top_tracking

---- Populating __model_training ----

ModelTraining: 0it [00:00, ?it/s]

---- Populating _recording_info ----

RecordingInfo: 0it [00:00, ?it/s]

---- Populating __model_evaluation ----

ModelEvaluation:   0%|          | 0/1 [00:00<?, ?it/s]Config:
{'all_joints': [[0], [1], [2]],
 'all_joints_names': ['head', 'bodycenter', 'tailbase'],
 'batch_size': 1,
 'crop_pad': 0,
 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_from_top_trackingFeb23/from_top_tracking_DJ95shuffle1.mat',
 'dataset_type': 'imgaug',
 'deterministic': False,
 'fg_fraction': 0.25,
 'global_scale': 0.8,
 'init_weights': '/Volumes/GoogleDrive/My '
                 'Drive/ref/DeepLabCut/deeplabcut/pose_estimation_tensorflow/models/pretrained/mobilenet_v2_1.0_224.ckpt',
 'intermediate_supervision': False,
 'intermediate_supervision_layer': 12,
 'location_refinement': True,
 'locref_huber_loss': True,
 'locref_loss_weight': 1.0,
 'locref_stdev': 7.2801,
 'log_dir': 'log',
 'mean_pixel': [123.68, 116.779, 103.939],
 'mirror': False,
 'net_type': 'mobilenet_v2_1.0',
 'num_joints': 3,
 'optimizer': 'sgd',
 'pairwise_huber_loss': True,
 'pairwise_predict': False,
 'partaffinityfield_predict': False,
 'regularize': False,
 'scoremap_dir': 'test',
 'shuffle': True,
 'snapshot_prefix': '/tmp/test_data/from_top_tracking/dlc-models/iteration-0/from_top_trackingFeb23-trainset95shuffle1/test/snapshot',
 'stride': 8.0,
 'weigh_negatives': False,
 'weigh_only_present_joints': False,
 'weigh_part_predictions': False,
 'weight_decay': 0.0001}
ModelEvaluation: 100%|██████████| 1/1 [00:00<00:00,  1.95it/s]

Running  DLC_mobnet_100_from_top_trackingFeb23shuffle1_103000  with # of training iterations: 103000
This net has already been evaluated!

---- Populating __pose_estimation ----

PoseEstimation: 0it [00:00, ?it/s]

Add a pose estimation task, and launch via process.
- Get all primary key information for a given recording
- Add the model and task_mode (i.e., load vs. trigger) to be applied
- Add any additional analysis parameters for deeplabcut.analyze_videos

In [8]:

            
                Copied!
                
key=(model.VideoRecording & 'recording_id=1').fetch1('KEY')
key.update({'model_name': 'FromTop-latest', 'task_mode': 'trigger'})
analyze_params={'save_as_csv':True} # add any others from deeplabcut.analyze_videos
model.PoseEstimationTask.insert_estimation_task(key,params=analyze_params)
process.run()
key=(model.VideoRecording & 'recording_id=1').fetch1('KEY')
key.update({'model_name': 'FromTop-latest', 'task_mode': 'trigger'})
analyze_params={'save_as_csv':True} # add any others from deeplabcut.analyze_videos
model.PoseEstimationTask.insert_estimation_task(key,params=analyze_params)
process.run()

---- Populating __model_training ----

ModelTraining: 0it [00:00, ?it/s]

---- Populating _recording_info ----

RecordingInfo: 0it [00:00, ?it/s]

---- Populating __model_evaluation ----

ModelEvaluation: 0it [00:00, ?it/s]

---- Populating __pose_estimation ----

PoseEstimation:   0%|          | 0/1 [00:00<?, ?it/s]Config:
{'all_joints': [[0], [1], [2]],
 'all_joints_names': ['head', 'bodycenter', 'tailbase'],
 'batch_size': 1,
 'crop_pad': 0,
 'dataset': 'training-datasets/iteration-0/UnaugmentedDataSet_from_top_trackingFeb23/from_top_tracking_DJ95shuffle1.mat',
 'dataset_type': 'imgaug',
 'deterministic': False,
 'fg_fraction': 0.25,
 'global_scale': 0.8,
 'init_weights': '/Volumes/GoogleDrive/My '
                 'Drive/ref/DeepLabCut/deeplabcut/pose_estimation_tensorflow/models/pretrained/mobilenet_v2_1.0_224.ckpt',
 'intermediate_supervision': False,
 'intermediate_supervision_layer': 12,
 'location_refinement': True,
 'locref_huber_loss': True,
 'locref_loss_weight': 1.0,
 'locref_stdev': 7.2801,
 'log_dir': 'log',
 'mean_pixel': [123.68, 116.779, 103.939],
 'mirror': False,
 'net_type': 'mobilenet_v2_1.0',
 'num_joints': 3,
 'optimizer': 'sgd',
 'pairwise_huber_loss': True,
 'pairwise_predict': False,
 'partaffinityfield_predict': False,
 'regularize': False,
 'scoremap_dir': 'test',
 'shuffle': True,
 'snapshot_prefix': '/tmp/test_data/from_top_tracking/dlc-models/iteration-0/from_top_trackingFeb23-trainset95shuffle1/test/snapshot',
 'stride': 8.0,
 'weigh_negatives': False,
 'weigh_only_present_joints': False,
 'weigh_part_predictions': False,
 'weight_decay': 0.0001}
/Users/cb/miniconda3/envs/ele/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1694: UserWarning: `layer.apply` is deprecated and will be removed in a future version. Please use `layer.__call__` method instead.
  warnings.warn('`layer.apply` is deprecated and '

Using snapshot-103000 for model /tmp/test_data/from_top_tracking/dlc-models/iteration-0/from_top_trackingFeb23-trainset95shuffle1
Starting to analyze %  /tmp/test_data/from_top_tracking/videos/test-2s.mp4
The videos are analyzed. Now your research can truly start! 
 You can create labeled videos with 'create_labeled_video'
If the tracking is not satisfactory for some videos, consider expanding the training set. You can use the function 'extract_outlier_frames' to extract a few representative outlier frames.

PoseEstimation: 100%|██████████| 1/1 [00:05<00:00,  5.98s/it]

Retrieve estimated position data.

In [9]:

            
                Copied!
                
model.PoseEstimation.get_trajectory(key)
model.PoseEstimation.get_trajectory(key)

Out[9]:

scorer	FromTop-latest
bodyparts	bodycenter				head				tailbase
coords	x	y	z	likelihood	x	y	z	likelihood	x	y	z	likelihood
0	246.782684	298.728088	0.0	0.999998	241.036957	316.332489	0.0	0.999850	256.203064	278.553314	0.0	0.999998
1	246.217529	299.358063	0.0	0.999997	239.048737	319.177002	0.0	0.999905	255.819626	280.200745	0.0	0.999996
2	244.459579	301.309235	0.0	0.999999	240.238800	320.525696	0.0	0.999899	255.705093	280.939056	0.0	0.999995
3	242.014755	302.865204	0.0	0.999999	238.536774	322.324463	0.0	0.999941	254.424484	282.015778	0.0	0.999990
4	240.900177	303.459167	0.0	0.999998	237.967987	324.072327	0.0	0.999941	252.180603	280.899200	0.0	0.999977
...	...	...	...	...	...	...	...	...	...	...	...	...
118	248.682251	364.709869	0.0	0.999965	270.854980	371.893127	0.0	0.999961	234.899185	356.035583	0.0	0.999996
119	250.326385	366.870361	0.0	0.999972	271.488495	373.099884	0.0	0.999991	235.644073	356.815125	0.0	0.999989
120	251.634140	367.709198	0.0	0.999972	272.043884	373.402893	0.0	0.999995	236.953812	358.651459	0.0	0.999977
121	255.393692	364.111145	0.0	0.999979	273.417572	373.906799	0.0	0.999997	238.825363	361.561798	0.0	0.999885
122	257.736847	365.264008	0.0	0.999996	276.008667	373.901245	0.0	0.999992	239.148163	364.029297	0.0	0.999962

123 rows × 12 columns

Summary and next step¶

This notebook runs through the workflow in an automatic manner.
The next notebook 05-Visualization demonstrates how to plot this data and label videos on disk.