DLC Models

Position- DeepLabCut from Scratch¶

Overview¶

Developer Note: if you may make a PR in the future, be sure to copy this notebook, and use the gitignore prefix temp to avoid future conflicts.

This is one notebook in a multi-part series on Spyglass.

• To set up your Spyglass environment and database, see the Setup notebook
• For additional info on DataJoint syntax, including table definitions and inserts, see the Insert Data notebook

This tutorial will extract position via DeepLabCut (DLC). It will walk through...

• creating a DLC project
• extracting and labeling frames
• training your model
• executing pose estimation on a novel behavioral video
• processing the pose estimation output to extract a centroid and orientation
• inserting the resulting information into the PositionOutput table

Note 2: Make sure you are running this within the spyglass-position Conda environment (instructions for install are in the environment_position.yml)

Here is a schematic showing the tables used in this pipeline.

Table of Contents¶

DLCProject
DLCModelTraining
DLCModel
DLCPoseEstimation
DLCSmoothInterp
DLCCentroid
DLCOrientation
DLCPosV1
DLCPosVideo
PositionOutput

You can click on any header to return to the Table of Contents

Imports¶

In [12]:

%load_ext autoreload
%autoreload 2

%load_ext autoreload %autoreload 2

In [13]:

import os
import datajoint as dj

import spyglass.common as sgc
import spyglass.position.v1 as sgp

import numpy as np
import pandas as pd
import pynwb
from spyglass.position import PositionOutput

# change to the upper level folder to detect dj_local_conf.json
if os.path.basename(os.getcwd()) == "notebooks":
    os.chdir("..")
dj.config.load("dj_local_conf.json")  # load config for database connection info

# ignore datajoint+jupyter async warnings
import warnings

warnings.simplefilter("ignore", category=DeprecationWarning)
warnings.simplefilter("ignore", category=ResourceWarning)

import os import datajoint as dj import spyglass.common as sgc import spyglass.position.v1 as sgp import numpy as np import pandas as pd import pynwb from spyglass.position import PositionOutput # change to the upper level folder to detect dj_local_conf.json if os.path.basename(os.getcwd()) == "notebooks": os.chdir("..") dj.config.load("dj_local_conf.json") # load config for database connection info # ignore datajoint+jupyter async warnings import warnings warnings.simplefilter("ignore", category=DeprecationWarning) warnings.simplefilter("ignore", category=ResourceWarning)

DLCProject ¶

Notes:

• The cells within this DLCProject step need to be performed in a local Jupyter notebook to allow for use of the frame labeling GUI.
• Please do not add to the BodyPart table in the production database unless necessary.

Body Parts¶

We'll begin by looking at the BodyPart table, which stores standard names of body parts used in DLC models throughout the lab with a concise description.

In [14]:

sgp.BodyPart()

sgp.BodyPart()

Out[14]:

bodypart	bodypart_description
back	middle of the rat's back
driveBack	back of drive
driveFront	front of drive
earL	left ear of the rat
earR	right ear of the rat
forelimbL	left forelimb of the rat
forelimbR	right forelimb of the rat
greenLED	greenLED
hindlimbL	left hindlimb of the rat
hindlimbR	right hindlimb of the rat
nose	tip of the nose of the rat
redLED_C	redLED_C

...

Total: 23

If the bodyparts you plan to use in your model are not yet in the table, here is code to add bodyparts:

sgp.BodyPart.insert(
    [
        {"bodypart": "bp_1", "bodypart_description": "concise descrip"},
        {"bodypart": "bp_2", "bodypart_description": "concise descrip"},
    ],
    skip_duplicates=True,
)

Define videos and camera name (optional) for training set¶

To train a model, we'll need to extract frames, which we can label as training data. We can construct a list of videos from which we'll extract frames.

The list can either contain dictionaries identifying behavioral videos for NWB files that have already been added to Spyglass, or absolute file paths to the videos you want to use.

For this tutorial, we'll use two videos for which we already have frames labeled.

Defining camera name is optional: it should be done in cases where there are multiple cameras streaming per epoch, but not necessary otherwise.
example: camera_name = "HomeBox_camera"

NOTE: The official release of Spyglass does not yet support multicamera projects. You can monitor progress on the effort to add this feature by checking this PR or use this experimental branch, which takes the keys nwb_file_name and epoch, and camera_name in the video_list variable.

In [15]:

video_list = [
    {"nwb_file_name": "J1620210529_.nwb", "epoch": 2},
    {"nwb_file_name": "peanut20201103_.nwb", "epoch": 4},
]

video_list = [ {"nwb_file_name": "J1620210529_.nwb", "epoch": 2}, {"nwb_file_name": "peanut20201103_.nwb", "epoch": 4}, ]

Path variables¶

The position pipeline also keeps track of paths for project, video, and output. Just like we saw in Setup, you can manage these either with environmental variables...

export DLC_PROJECT_DIR="/nimbus/deeplabcut/projects"
export DLC_VIDEO_DIR="/nimbus/deeplabcut/video"
export DLC_OUTPUT_DIR="/nimbus/deeplabcut/output"

Or these can be set in your datajoint config:

{
  "custom": {
    "dlc_dirs": {
      "base": "/nimbus/deeplabcut/",
      "project": "/nimbus/deeplabcut/projects",
      "video": "/nimbus/deeplabcut/video",
      "output": "/nimbus/deeplabcut/output"
    }
  }
}

NOTE: If only base is specified as shown above, spyglass will assume the relative directories shown.

You can check the result of this setup process with...

In [16]:

from spyglass.settings import config

config

from spyglass.settings import config config

Out[16]:

{'debug_mode': False,
 'prepopulate': True,
 'SPYGLASS_BASE_DIR': '/stelmo/nwb',
 'SPYGLASS_RAW_DIR': '/stelmo/nwb/raw',
 'SPYGLASS_ANALYSIS_DIR': '/stelmo/nwb/analysis',
 'SPYGLASS_RECORDING_DIR': '/stelmo/nwb/recording',
 'SPYGLASS_SORTING_DIR': '/stelmo/nwb/sorting',
 'SPYGLASS_WAVEFORMS_DIR': '/stelmo/nwb/waveforms',
 'SPYGLASS_TEMP_DIR': '/stelmo/nwb/tmp/spyglass',
 'SPYGLASS_VIDEO_DIR': '/stelmo/nwb/video',
 'KACHERY_CLOUD_DIR': '/stelmo/nwb/.kachery-cloud',
 'KACHERY_STORAGE_DIR': '/stelmo/nwb/kachery_storage',
 'KACHERY_TEMP_DIR': '/stelmo/nwb/tmp',
 'DLC_PROJECT_DIR': '/nimbus/deeplabcut/projects',
 'DLC_VIDEO_DIR': '/nimbus/deeplabcut/video',
 'DLC_OUTPUT_DIR': '/nimbus/deeplabcut/output',
 'KACHERY_ZONE': 'franklab.default',
 'FIGURL_CHANNEL': 'franklab2',
 'DJ_SUPPORT_FILEPATH_MANAGEMENT': 'TRUE',
 'KACHERY_CLOUD_EPHEMERAL': 'TRUE',
 'HD5_USE_FILE_LOCKING': 'FALSE'}

Before creating our project, we need to define a few variables.

• A team name, as shown in LabTeam for setting permissions. Here, we'll use "LorenLab".
• A project_name, as a unique identifier for this DLC project. Here, we'll use "tutorial_scratch_yourinitials"
• bodyparts is a list of body parts for which we want to extract position. The pre-labeled frames we're using include the bodyparts listed below.
• Number of frames to extract/label as frames_per_video. Note that the DLC creators recommend having 200 frames as the minimum total number for each project.

In [17]:

team_name = sgc.LabTeam.fetch("team_name")[0]  # If on lab DB, "LorenLab"
project_name = "tutorial_scratch_DG"
frames_per_video = 100
bodyparts = ["redLED_C", "greenLED", "redLED_L", "redLED_R", "tailBase"]
project_key = sgp.DLCProject.insert_new_project(
    project_name=project_name,
    bodyparts=bodyparts,
    lab_team=team_name,
    frames_per_video=frames_per_video,
    video_list=video_list,
    skip_duplicates=True,
)

team_name = sgc.LabTeam.fetch("team_name")[0] # If on lab DB, "LorenLab" project_name = "tutorial_scratch_DG" frames_per_video = 100 bodyparts = ["redLED_C", "greenLED", "redLED_L", "redLED_R", "tailBase"] project_key = sgp.DLCProject.insert_new_project( project_name=project_name, bodyparts=bodyparts, lab_team=team_name, frames_per_video=frames_per_video, video_list=video_list, skip_duplicates=True, )

project name: tutorial_scratch_DG is already in use.

Now that we've initialized our project we'll need to extract frames which we will then label.

In [ ]:

# comment this line out after you finish frame extraction for each project
sgp.DLCProject().run_extract_frames(project_key)

# comment this line out after you finish frame extraction for each project sgp.DLCProject().run_extract_frames(project_key)

This is the line used to label the frames you extracted, if you wish to use the DLC GUI on the computer you are currently using.

#comment
sgp.DLCProject().run_label_frames(project_key)

Otherwise, it is best/easiest practice to label the frames on your local computer (like a MacBook) that can run DeepLabCut's GUI well. Instructions:

1. Install DLC on your local (preferably into a 'Src' folder): https://deeplabcut.github.io/DeepLabCut/docs/installation.html
2. Upload frames extracted and saved in nimbus (should be /nimbus/deeplabcut/<YOUR_PROJECT_NAME>/labeled-data) AND the project's associated config file (should be /nimbus/deeplabcut/<YOUR_PROJECT_NAME>/config.yaml) to Box (we get free with UCSF)
3. Download labeled-data and config files on your local from Box
4. Create a 'projects' folder where you installed DeepLabCut; create a new folder with your complete project name there; save the downloaded files there.
5. Edit the config.yaml file: line 9 defining project_path needs to be the file path where it is saved on your local (ex: /Users/lorenlab/Src/DeepLabCut/projects/tutorial_sratch_DG-LorenLab-2023-08-16)
6. Open the DLC GUI through terminal
   (ex: conda activate miniconda/envs/DEEPLABCUT_M1
pythonw -m deeplabcut)
7. Load an existing project; choose the config.yaml file
8. Label frames; labeling tutorial: https://www.youtube.com/watch?v=hsA9IB5r73E.
9. Once all frames are labeled, you should re-upload labeled-data folder back to Box and overwrite it in the original nimbus location so that your completed frames are ready to be used in the model.

Now we can check the DLCProject.File part table and see all of our training files and videos there!

In [18]:

sgp.DLCProject.File & project_key

sgp.DLCProject.File & project_key

Out[18]:

Paths of training files (e.g., labeled pngs, CSV or video)

project_name name of DLC project	file_name Concise name to describe file	file_ext extension of file	file_path
tutorial_scratch_DG	20201103_peanut_04_r2	mp4	/nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/videos/20201103_peanut_04_r2.mp4
tutorial_scratch_DG	20201103_peanut_04_r2_labeled_data	h5	/nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/labeled-data/20201103_peanut_04_r2/CollectedData_LorenLab.h5
tutorial_scratch_DG	20210529_J16_02_r1	mp4	/nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/videos/20210529_J16_02_r1.mp4
tutorial_scratch_DG	20210529_J16_02_r1_labeled_data	h5	/nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/labeled-data/20210529_J16_02_r1/CollectedData_LorenLab.h5

Total: 4

This step and beyond should be run on a GPU-enabled machine.

DLCModelTraining¶

Please make sure you're running this notebook on a GPU-enabled machine.

Now that we've imported existing frames, we can get ready to train our model.

First, we'll need to define a set of parameters for DLCModelTrainingParams, which will get used by DeepLabCut during training. Let's start with gputouse, which determines which GPU core to use.

The cell below determines which core has space and set the gputouse variable accordingly.

In [19]:

sgp.dlc_utils.get_gpu_memory()

sgp.dlc_utils.get_gpu_memory()

Out[19]:

{0: 305}

Set GPU core:

In [20]:

gputouse = 1  # 1-9

gputouse = 1 # 1-9

Now we'll define the rest of our parameters and insert the entry.

To see all possible parameters, try:

sgp.DLCModelTrainingParams.get_accepted_params()

In [21]:

training_params_name = "tutorial"
sgp.DLCModelTrainingParams.insert_new_params(
    paramset_name=training_params_name,
    params={
        "trainingsetindex": 0,
        "shuffle": 1,
        "gputouse": gputouse,
        "net_type": "resnet_50",
        "augmenter_type": "imgaug",
    },
    skip_duplicates=True,
)

training_params_name = "tutorial" sgp.DLCModelTrainingParams.insert_new_params( paramset_name=training_params_name, params={ "trainingsetindex": 0, "shuffle": 1, "gputouse": gputouse, "net_type": "resnet_50", "augmenter_type": "imgaug", }, skip_duplicates=True, )

New param set not added
A param set with name: tutorial already exists

Next we'll modify the project_key from above to include the necessary entries for DLCModelTraining

In [22]:

# project_key['project_path'] = os.path.dirname(project_key['config_path'])
if "config_path" in project_key:
    del project_key["config_path"]

# project_key['project_path'] = os.path.dirname(project_key['config_path']) if "config_path" in project_key: del project_key["config_path"]

We can insert an entry into DLCModelTrainingSelection and populate DLCModelTraining.

Note: You can stop training at any point using I + I or interrupt the Kernel.

The maximum total number of training iterations is 1030000; you can end training before this amount if the loss rate (lr) and total loss plateau and are very close to 0.

In [23]:

sgp.DLCModelTrainingSelection.heading

sgp.DLCModelTrainingSelection.heading

Out[23]:

project_name         : varchar(100)                 # name of DLC project
dlc_training_params_name : varchar(50)                  # descriptive name of parameter set
training_id          : int                          # unique integer,
---
model_prefix=""      : varchar(32)                  # 

In [24]:

sgp.DLCModelTrainingSelection().insert1(
    {
        **project_key,
        "dlc_training_params_name": training_params_name,
        "training_id": 0,
        "model_prefix": "",
    }
)
model_training_key = (
    sgp.DLCModelTrainingSelection
    & {
        **project_key,
        "dlc_training_params_name": training_params_name,
    }
).fetch1("KEY")
sgp.DLCModelTraining.populate(model_training_key)

sgp.DLCModelTrainingSelection().insert1( { **project_key, "dlc_training_params_name": training_params_name, "training_id": 0, "model_prefix": "", } ) model_training_key = ( sgp.DLCModelTrainingSelection & { **project_key, "dlc_training_params_name": training_params_name, } ).fetch1("KEY") sgp.DLCModelTraining.populate(model_training_key)

2024-01-18 10:23:30.406102: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

Loading DLC 2.2.3...
OpenCV is built with OpenMP support. This usually results in poor performance. For details, see https://github.com/tensorpack/benchmarks/blob/master/ImageNet/benchmark-opencv-resize.py

---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
Cell In[24], line 16
      1 sgp.DLCModelTrainingSelection().insert1(
      2     {
      3         **project_key,
   (...)
      7     }
      8 )
      9 model_training_key = (
     10     sgp.DLCModelTrainingSelection
     11     & {
   (...)
     14     }
     15 ).fetch1("KEY")
---> 16 sgp.DLCModelTraining.populate(model_training_key)

File ~/anaconda3/envs/spyglass-position/lib/python3.9/site-packages/datajoint/autopopulate.py:241, in AutoPopulate.populate(self, suppress_errors, return_exception_objects, reserve_jobs, order, limit, max_calls, display_progress, processes, make_kwargs, *restrictions)
    237 if processes == 1:
    238     for key in (
    239         tqdm(keys, desc=self.__class__.__name__) if display_progress else keys
    240     ):
--> 241         error = self._populate1(key, jobs, **populate_kwargs)
    242         if error is not None:
    243             error_list.append(error)

File ~/anaconda3/envs/spyglass-position/lib/python3.9/site-packages/datajoint/autopopulate.py:292, in AutoPopulate._populate1(self, key, jobs, suppress_errors, return_exception_objects, make_kwargs)
    290 self.__class__._allow_insert = True
    291 try:
--> 292     make(dict(key), **(make_kwargs or {}))
    293 except (KeyboardInterrupt, SystemExit, Exception) as error:
    294     try:

File ~/Src/spyglass/src/spyglass/position/v1/position_dlc_training.py:150, in DLCModelTraining.make(self, key)
    144     from deeplabcut.utils.auxiliaryfunctions import (
    145         GetModelFolder as get_model_folder,
    146     )
    147 config_path, project_name = (DLCProject() & key).fetch1(
    148     "config_path", "project_name"
    149 )
--> 150 with OutputLogger(
    151     name="DLC_project_{project_name}_training",
    152     path=f"{os.path.dirname(config_path)}/log.log",
    153     print_console=True,
    154 ) as logger:
    155     dlc_config = read_config(config_path)
    156     project_path = dlc_config["project_path"]

File ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:192, in OutputLogger.__init__(self, name, path, level, **kwargs)
    191 def __init__(self, name, path, level="INFO", **kwargs):
--> 192     self.logger = self.setup_logger(name, path, **kwargs)
    193     self.name = self.logger.name
    194     self.level = getattr(logging, level)

File ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:244, in OutputLogger.setup_logger(self, name_logfile, path_logfile, print_console)
    241         logger.addHandler(self._get_stream_handler())
    243 else:
--> 244     file_handler = self._get_file_handler(path_logfile)
    245     logger.addHandler(file_handler)
    246     if print_console:

File ~/Src/spyglass/src/spyglass/position/v1/dlc_utils.py:255, in OutputLogger._get_file_handler(self, path)
    253 if not os.path.exists(output_dir):
    254     output_dir.mkdir(parents=True, exist_ok=True)
--> 255 file_handler = logging.FileHandler(path, mode="a")
    256 file_handler.setFormatter(self._get_formatter())
    257 return file_handler

File ~/anaconda3/envs/spyglass-position/lib/python3.9/logging/__init__.py:1146, in FileHandler.__init__(self, filename, mode, encoding, delay, errors)
   1144     self.stream = None
   1145 else:
-> 1146     StreamHandler.__init__(self, self._open())

File ~/anaconda3/envs/spyglass-position/lib/python3.9/logging/__init__.py:1175, in FileHandler._open(self)
   1170 def _open(self):
   1171     """
   1172     Open the current base file with the (original) mode and encoding.
   1173     Return the resulting stream.
   1174     """
-> 1175     return open(self.baseFilename, self.mode, encoding=self.encoding,
   1176                 errors=self.errors)

PermissionError: [Errno 13] Permission denied: '/nimbus/deeplabcut/projects/tutorial_scratch_DG-LorenLab-2023-08-16/log.log'

Here we'll make sure that the entry made it into the table properly!

In [ ]:

sgp.DLCModelTraining() & model_training_key

sgp.DLCModelTraining() & model_training_key

Populating DLCModelTraining automatically inserts the entry into DLCModelSource, which is used to select between models trained using Spyglass vs. other tools.

In [ ]:

sgp.DLCModelSource() & model_training_key

sgp.DLCModelSource() & model_training_key

The source field will only accept "FromImport" or "FromUpstream" as entries. Let's checkout the FromUpstream part table attached to DLCModelSource below.

In [ ]:

sgp.DLCModelSource.FromUpstream() & model_training_key

sgp.DLCModelSource.FromUpstream() & model_training_key

DLCModel ¶

Next we'll populate the DLCModel table, which holds all the relevant information for all trained models.

First, we'll need to determine a set of parameters for our model to select the correct model file. Here is the default:

In [ ]:

sgp.DLCModelParams.get_default()

sgp.DLCModelParams.get_default()

Here is the syntax to add your own parameter set:

dlc_model_params_name = "make_this_yours"
params = {
    "params": {},
    "shuffle": 1,
    "trainingsetindex": 0,
    "model_prefix": "",
}
sgp.DLCModelParams.insert1(
    {"dlc_model_params_name": dlc_model_params_name, "params": params},
    skip_duplicates=True,
)

We can insert sets of parameters into DLCModelSelection and populate DLCModel.

In [ ]:

temp_model_key = (sgp.DLCModelSource & model_training_key).fetch1("KEY")

temp_model_key = (sgp.DLCModelSource & model_training_key).fetch1("KEY")

In [ ]:

# comment these lines out after successfully inserting, for each project
sgp.DLCModelSelection().insert1(
    {**temp_model_key, "dlc_model_params_name": "default"}, skip_duplicates=True
)

# comment these lines out after successfully inserting, for each project sgp.DLCModelSelection().insert1( {**temp_model_key, "dlc_model_params_name": "default"}, skip_duplicates=True )

In [ ]:

model_key = (sgp.DLCModelSelection & temp_model_key).fetch1("KEY")
sgp.DLCModel.populate(model_key)

model_key = (sgp.DLCModelSelection & temp_model_key).fetch1("KEY") sgp.DLCModel.populate(model_key)

Again, let's make sure that everything looks correct in DLCModel.

In [ ]:

sgp.DLCModel() & model_key

sgp.DLCModel() & model_key

DLCPoseEstimation ¶

Alright, now that we've trained model and populated the DLCModel table, we're ready to set-up Pose Estimation on a behavioral video of your choice.

For this tutorial, you can choose to use an epoch of your choice, we can also use the one specified below. If you'd like to use your own video, just specify the nwb_file_name and epoch number and make sure it's in the VideoFile table!

In [3]:

nwb_file_name = "J1620210604_.nwb"
sgc.VideoFile() & {"nwb_file_name": nwb_file_name}

nwb_file_name = "J1620210604_.nwb" sgc.VideoFile() & {"nwb_file_name": nwb_file_name}

Out[3]:

nwb_file_name name of the NWB file	epoch the session epoch for this task and apparatus(1 based)	video_file_num	camera_name	video_file_object_id the object id of the file object
J1620210604_.nwb	1	0		178f5746-30e3-4957-891e-8024e23522dc
J1620210604_.nwb	2	0		d64ec979-326b-429f-b3fe-1bbfbf806293
J1620210604_.nwb	3	0		cf14bcd2-c0a9-457b-8791-42f3f28dd912
J1620210604_.nwb	4	0		183c9910-36fd-46c1-a24c-8d1c306d7248
J1620210604_.nwb	5	0		4677c7cd-8cd8-4801-8f6e-5b7bb14a6d6b
J1620210604_.nwb	6	0		0e46532b-483f-43af-ba6e-ba75ccf340ea
J1620210604_.nwb	7	0		c6d1d037-44ec-4d91-99d1-172d371bf82a
J1620210604_.nwb	8	0		4d7e070c-6220-47de-8173-993f013fafa8
J1620210604_.nwb	9	0		b50108ec-f587-46df-b1c8-3ca23091bde0
J1620210604_.nwb	10	0		b9b5da20-da39-4274-9be2-55610cfd1b5b
J1620210604_.nwb	11	0		6c827b8d-513c-4dba-ae75-0b36dcf4811f
J1620210604_.nwb	12	0		41bd2344-1b41-4737-8dfb-7c860d089155

...

Total: 20

In [5]:

epoch = 14  # change based on VideoFile entry
video_file_num = 0  # change based on VideoFile entry

epoch = 14 # change based on VideoFile entry video_file_num = 0 # change based on VideoFile entry

Using insert_estimation_task will convert out video to be in .mp4 format (DLC struggles with .h264) and determine the directory in which we'll store the pose estimation results.

• task_mode (trigger or load) determines whether or not populating DLCPoseEstimation triggers a new pose estimation, or loads an existing.
• video_file_num will be 0 in almost all cases.
• gputouse was already set during training. It may be a good idea to make sure that core is still free before moving forward.

The DLCPoseEstimationSelection insertion step will convert your .h264 video to an .mp4 first and save it in /nimbus/deeplabcut/video. If this video already exists here, the insertion will never complete.

We first delete any .mp4 that exists for this video from the nimbus folder. Remove the # to run this line. The ! tells the notebook that this is a system command to be run with a shell script instead of python. Be sure to change the string based on date and rat with which you are training the model

In [ ]:

#! find /nimbus/deeplabcut/video -type f -name '*20210604_J16*' -delete

#! find /nimbus/deeplabcut/video -type f -name '*20210604_J16*' -delete

In [6]:

pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(
    {
        "nwb_file_name": nwb_file_name,
        "epoch": epoch,
        "video_file_num": video_file_num,
        **model_key,
    },
    task_mode="trigger",  # trigger or load
    params={"gputouse": gputouse, "videotype": "mp4"},
)

pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task( { "nwb_file_name": nwb_file_name, "epoch": epoch, "video_file_num": video_file_num, **model_key, }, task_mode="trigger", # trigger or load params={"gputouse": gputouse, "videotype": "mp4"}, )

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[6], line 6
      1 pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(
      2     {
      3         "nwb_file_name": nwb_file_name,
      4         "epoch": epoch,
      5         "video_file_num": video_file_num,
----> 6         **model_key,
      7     },
      8     task_mode="trigger", #trigger or load
      9     params={"gputouse": gputouse, "videotype": "mp4"},
     10 )

NameError: name 'model_key' is not defined

If the above insertion step fails in either trigger or load mode for an epoch, run the following lines:

(pose_estimation_key = sgp.DLCPoseEstimationSelection.insert_estimation_task(
    {
        "nwb_file_name": nwb_file_name,
        "epoch": epoch,
        "video_file_num": video_file_num,
        **model_key,
    }).delete()

And now we populate DLCPoseEstimation! This might take some time for full datasets.

In [ ]:

sgp.DLCPoseEstimation().populate(pose_estimation_key)

sgp.DLCPoseEstimation().populate(pose_estimation_key)

Let's visualize the output from Pose Estimation

In [ ]:

(sgp.DLCPoseEstimation() & pose_estimation_key).fetch_dataframe()

(sgp.DLCPoseEstimation() & pose_estimation_key).fetch_dataframe()

DLCSmoothInterp ¶

Now that we've completed pose estimation, it's time to identify NaNs and optionally interpolate over low likelihood periods and smooth the resulting positions.
First we need to define some parameters for smoothing and interpolation. We can see the default parameter set below.
Note: it is recommended to use the just_nan parameters here and save interpolation and smoothing for the centroid step as this provides for a better end result.

In [ ]:

# The default parameter set to interpolate and smooth over each LED individually
print(sgp.DLCSmoothInterpParams.get_default())

# The default parameter set to interpolate and smooth over each LED individually print(sgp.DLCSmoothInterpParams.get_default())

In [ ]:

# The just_nan parameter set that identifies NaN indices and leaves smoothing and interpolation to the centroid step
print(sgp.DLCSmoothInterpParams.get_nan_params())
si_params_name = "just_nan"  # could also use "default"

# The just_nan parameter set that identifies NaN indices and leaves smoothing and interpolation to the centroid step print(sgp.DLCSmoothInterpParams.get_nan_params()) si_params_name = "just_nan" # could also use "default"

To change any of these parameters, one would do the following:

si_params_name = "your_unique_param_name"
params = {
    "smoothing_params": {
        "smoothing_duration": 0.00,
        "smooth_method": "moving_avg",
    },
    "interp_params": {"likelihood_thresh": 0.00},
    "max_plausible_speed": 0,
    "speed_smoothing_std_dev": 0.000,
}
sgp.DLCSmoothInterpParams().insert1(
    {"dlc_si_params_name": si_params_name, "params": params},
    skip_duplicates=True,
)

We'll create a dictionary with the correct set of keys for the DLCSmoothInterpSelection table

In [ ]:

si_key = pose_estimation_key.copy()
fields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys())
si_key = {key: val for key, val in si_key.items() if key in fields}
si_key

si_key = pose_estimation_key.copy() fields = list(sgp.DLCSmoothInterpSelection.fetch().dtype.fields.keys()) si_key = {key: val for key, val in si_key.items() if key in fields} si_key

We can insert all of the bodyparts we want to process into DLCSmoothInterpSelection
First lets visualize the bodyparts we have available to us.

In [ ]:

print((sgp.DLCPoseEstimation.BodyPart & pose_estimation_key).fetch("bodypart"))

print((sgp.DLCPoseEstimation.BodyPart & pose_estimation_key).fetch("bodypart"))

We can use insert1 to insert a single bodypart, but would suggest using insert to insert a list of keys with different bodyparts.

To insert a single bodypart, one would do the following:

sgp.DLCSmoothInterpSelection.insert1(
    {
        **si_key,
        'bodypart': 'greenLED',
        'dlc_si_params_name': si_params_name,
    },
    skip_duplicates=True)

We'll see a list of bodyparts and then insert them into DLCSmoothInterpSelection.

In [ ]:

bodyparts = ["greenLED", "redLED_C"]
sgp.DLCSmoothInterpSelection.insert(
    [
        {
            **si_key,
            "bodypart": bodypart,
            "dlc_si_params_name": si_params_name,
        }
        for bodypart in bodyparts
    ],
    skip_duplicates=True,
)

bodyparts = ["greenLED", "redLED_C"] sgp.DLCSmoothInterpSelection.insert( [ { **si_key, "bodypart": bodypart, "dlc_si_params_name": si_params_name, } for bodypart in bodyparts ], skip_duplicates=True, )

And verify the entry:

In [ ]:

sgp.DLCSmoothInterpSelection() & si_key

sgp.DLCSmoothInterpSelection() & si_key

Now, we populate DLCSmoothInterp, which will perform smoothing and interpolation on all of the bodyparts specified.

In [ ]:

sgp.DLCSmoothInterp().populate(si_key)

sgp.DLCSmoothInterp().populate(si_key)

And let's visualize the resulting position data using a scatter plot

In [ ]:

(
    sgp.DLCSmoothInterp() & {**si_key, "bodypart": bodyparts[0]}
).fetch1_dataframe().plot.scatter(x="x", y="y", s=1, figsize=(5, 5))

( sgp.DLCSmoothInterp() & {**si_key, "bodypart": bodyparts[0]} ).fetch1_dataframe().plot.scatter(x="x", y="y", s=1, figsize=(5, 5))

DLCSmoothInterpCohort ¶

After smoothing/interpolation, we need to select bodyparts from which we want to derive a centroid and orientation, which is performed by the DLCSmoothInterpCohort table.

First, let's make a key that represents the 'cohort', using dlc_si_cohort_selection_name. We'll need a bodypart dictionary using bodypart keys and smoothing/interpolation parameters used as value.

In [ ]:

cohort_key = si_key.copy()
if "bodypart" in cohort_key:
    del cohort_key["bodypart"]
if "dlc_si_params_name" in cohort_key:
    del cohort_key["dlc_si_params_name"]
cohort_key["dlc_si_cohort_selection_name"] = "green_red_led"
cohort_key["bodyparts_params_dict"] = {
    "greenLED": si_params_name,
    "redLED_C": si_params_name,
}
print(cohort_key)

cohort_key = si_key.copy() if "bodypart" in cohort_key: del cohort_key["bodypart"] if "dlc_si_params_name" in cohort_key: del cohort_key["dlc_si_params_name"] cohort_key["dlc_si_cohort_selection_name"] = "green_red_led" cohort_key["bodyparts_params_dict"] = { "greenLED": si_params_name, "redLED_C": si_params_name, } print(cohort_key)

We'll insert the cohort into DLCSmoothInterpCohortSelection and populate DLCSmoothInterpCohort, which collates the separately smoothed and interpolated bodyparts into a single entry.

In [ ]:

sgp.DLCSmoothInterpCohortSelection().insert1(cohort_key, skip_duplicates=True)
sgp.DLCSmoothInterpCohort.populate(cohort_key)

sgp.DLCSmoothInterpCohortSelection().insert1(cohort_key, skip_duplicates=True) sgp.DLCSmoothInterpCohort.populate(cohort_key)

And verify the entry:

In [ ]:

sgp.DLCSmoothInterpCohort.BodyPart() & cohort_key

sgp.DLCSmoothInterpCohort.BodyPart() & cohort_key

DLCCentroid ¶

With this cohort, we can determine a centroid using another set of parameters.

In [ ]:

# Here is the default set
print(sgp.DLCCentroidParams.get_default())
centroid_params_name = "default"

# Here is the default set print(sgp.DLCCentroidParams.get_default()) centroid_params_name = "default"

Here is the syntax to add your own parameters:

centroid_params = {
    "centroid_method": "two_pt_centroid",
    "points": {
        "greenLED": "greenLED",
        "redLED_C": "redLED_C",
    },
    "speed_smoothing_std_dev": 0.100,
}
centroid_params_name = "your_unique_param_name"
sgp.DLCCentroidParams.insert1(
    {
        "dlc_centroid_params_name": centroid_params_name,
        "params": centroid_params,
    },
    skip_duplicates=True,
)

We'll make a key to insert into DLCCentroidSelection.

In [ ]:

centroid_key = cohort_key.copy()
fields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys())
centroid_key = {key: val for key, val in centroid_key.items() if key in fields}
centroid_key["dlc_centroid_params_name"] = centroid_params_name
print(centroid_key)

centroid_key = cohort_key.copy() fields = list(sgp.DLCCentroidSelection.fetch().dtype.fields.keys()) centroid_key = {key: val for key, val in centroid_key.items() if key in fields} centroid_key["dlc_centroid_params_name"] = centroid_params_name print(centroid_key)

After inserting into the selection table, we can populate DLCCentroid

In [ ]:

sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True)
sgp.DLCCentroid.populate(centroid_key)

sgp.DLCCentroidSelection.insert1(centroid_key, skip_duplicates=True) sgp.DLCCentroid.populate(centroid_key)

Here we can visualize the resulting centroid position

In [ ]:

(sgp.DLCCentroid() & centroid_key).fetch1_dataframe().plot.scatter(
    x="position_x",
    y="position_y",
    c="speed",
    colormap="viridis",
    alpha=0.5,
    s=0.5,
    figsize=(10, 10),
)

(sgp.DLCCentroid() & centroid_key).fetch1_dataframe().plot.scatter( x="position_x", y="position_y", c="speed", colormap="viridis", alpha=0.5, s=0.5, figsize=(10, 10), )

DLCOrientation ¶

We'll now go through a similar process to identify the orientation.

In [ ]:

print(sgp.DLCOrientationParams.get_default())
dlc_orientation_params_name = "default"

print(sgp.DLCOrientationParams.get_default()) dlc_orientation_params_name = "default"

We'll prune the cohort_key we used above and add our dlc_orientation_params_name to make it suitable for DLCOrientationSelection.

In [ ]:

fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys())
orient_key = {key: val for key, val in cohort_key.items() if key in fields}
orient_key["dlc_orientation_params_name"] = dlc_orientation_params_name
print(orient_key)

fields = list(sgp.DLCOrientationSelection.fetch().dtype.fields.keys()) orient_key = {key: val for key, val in cohort_key.items() if key in fields} orient_key["dlc_orientation_params_name"] = dlc_orientation_params_name print(orient_key)

We'll insert into DLCOrientationSelection and populate DLCOrientation

In [ ]:

sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True)
sgp.DLCOrientation().populate(orient_key)

sgp.DLCOrientationSelection().insert1(orient_key, skip_duplicates=True) sgp.DLCOrientation().populate(orient_key)

We can fetch the orientation as a dataframe as quality assurance.

In [ ]:

(sgp.DLCOrientation() & orient_key).fetch1_dataframe()

(sgp.DLCOrientation() & orient_key).fetch1_dataframe()

DLCPosV1 ¶

After processing the position data, we have to do a few table manipulations to standardize various outputs.

To summarize, we brought in a pretrained DLC project, used that model to run pose estimation on a new behavioral video, smoothed and interpolated the result, formed a cohort of bodyparts, and determined the centroid and orientation of this cohort.

Now we'll populate DLCPos with our centroid/orientation entries above.

In [ ]:

fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys())
dlc_key = {key: val for key, val in centroid_key.items() if key in fields}
dlc_key["dlc_si_cohort_centroid"] = centroid_key["dlc_si_cohort_selection_name"]
dlc_key["dlc_si_cohort_orientation"] = orient_key[
    "dlc_si_cohort_selection_name"
]
dlc_key["dlc_orientation_params_name"] = orient_key[
    "dlc_orientation_params_name"
]
print(dlc_key)

fields = list(sgp.DLCPosV1.fetch().dtype.fields.keys()) dlc_key = {key: val for key, val in centroid_key.items() if key in fields} dlc_key["dlc_si_cohort_centroid"] = centroid_key["dlc_si_cohort_selection_name"] dlc_key["dlc_si_cohort_orientation"] = orient_key[ "dlc_si_cohort_selection_name" ] dlc_key["dlc_orientation_params_name"] = orient_key[ "dlc_orientation_params_name" ] print(dlc_key)

Now we can insert into DLCPosSelection and populate DLCPos with our dlc_key

In [ ]:

sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True)
sgp.DLCPosV1().populate(dlc_key)

sgp.DLCPosSelection().insert1(dlc_key, skip_duplicates=True) sgp.DLCPosV1().populate(dlc_key)

We can also make sure that all of our data made it through by fetching the dataframe attached to this entry.
We should expect 8 columns:

time
video_frame_ind
position_x
position_y
orientation
velocity_x
velocity_y
speed

In [ ]:

(sgp.DLCPosV1() & dlc_key).fetch1_dataframe()

(sgp.DLCPosV1() & dlc_key).fetch1_dataframe()

And even more, we can fetch the pose_eval_result that is calculated during this step. This field contains the percentage of frames that each bodypart was below the likelihood threshold of 0.95 as a means of assessing the quality of the pose estimation.

In [ ]:

(sgp.DLCPosV1() & dlc_key).fetch1("pose_eval_result")

(sgp.DLCPosV1() & dlc_key).fetch1("pose_eval_result")

DLCPosVideo ¶

We can create a video with the centroid and orientation overlaid on the original video. This will also plot the likelihood of each bodypart used in the cohort. This is optional, but a good quality assurance step.

In [ ]:

sgp.DLCPosVideoParams.insert_default()

sgp.DLCPosVideoParams.insert_default()

In [ ]:

params = {
    "percent_frames": 0.05,
    "incl_likelihood": True,
}
sgp.DLCPosVideoParams.insert1(
    {"dlc_pos_video_params_name": "five_percent", "params": params},
    skip_duplicates=True,
)

params = { "percent_frames": 0.05, "incl_likelihood": True, } sgp.DLCPosVideoParams.insert1( {"dlc_pos_video_params_name": "five_percent", "params": params}, skip_duplicates=True, )

In [ ]:

sgp.DLCPosVideoSelection.insert1(
    {**dlc_key, "dlc_pos_video_params_name": "five_percent"},
    skip_duplicates=True,
)

sgp.DLCPosVideoSelection.insert1( {**dlc_key, "dlc_pos_video_params_name": "five_percent"}, skip_duplicates=True, )

In [ ]:

sgp.DLCPosVideo().populate(dlc_key)

sgp.DLCPosVideo().populate(dlc_key)

PositionOutput ¶

PositionOutput is the final table of the pipeline and is automatically populated when we populate DLCPosV1

In [ ]:

sgp.PositionOutput.merge_get_part(dlc_key)

sgp.PositionOutput.merge_get_part(dlc_key)

PositionOutput also has a part table, similar to the DLCModelSource table above. Let's check that out as well.

In [ ]:

PositionOutput.DLCPosV1() & dlc_key

PositionOutput.DLCPosV1() & dlc_key

In [ ]:

(PositionOutput.DLCPosV1() & dlc_key).fetch1_dataframe()

(PositionOutput.DLCPosV1() & dlc_key).fetch1_dataframe()

CONGRATULATIONS!!¶

Please treat yourself to a nice tea break :-)

Return To Table of Contents