OTN Tool Box

The OTN Suite of Tools

White-Mihoff False Filtering

Distance Matrix / Merge 

Mihoff Interval Data Product 

Compressed Detection Data Product

Cohort Data Product 

Add Unique Id

Links

Installing the OTN Sandbox Virtual Machine

Other Documentation on Using the OTN Toolbox

Installing and Using the OTN Sandbox - Youtube Video Guide

Contribute to the OTN Toolbox with GitLab (requires OTN login)

 

White-Mihoff False Filtering

(White, E., Mihoff, M., Jones, B., Bajona, L., Halfyard, E. 2014. White-Mihoff False Filtering Tool)

OTN has developed a tool which will assist with filtering false detections. The first level of filtering involves identifying isolated detections. The original concept came from work done by Easton White. He was kind enough to share his research database with OTN. We did some preliminary research and developed a proposal for a filtering tool based on what Easton had done. This proof of concept was presented to Steve Kessel and Eddie Halfyard in December 2013 and a decision was made to develop a tool for general use.

This is a very simple tool, the first in a suite that OTN is developing. It will take an input file of detections and based on an input parameter will identify suspect detections. The suspect detections will be put into a file which the user can examine. There will be enough information for each suspect detection for the user to understand why it was flagged. There is also enough information to be able to reference the detection in the original file if the user wants to see what was happening at the same time.

The input parameter is a time in minutes. We used 60 as the default as this is what was used in Easton’s code. This value can be changed by the user. The output file contains a record for each detection for which there has been more than xx minutes since the previous detection (of that tag/animal) and more than the same amount of time until the next detection. It ignores which receiver the detection occurred at. That is all it does, nothing more and nothing less.

Once you are happy with the file of false detections you can then request the tool delete them. The original file will never be overwritten. Details are in White-Mihoff False Filtering Instructions.docx

There is an option to output a distance matrix. This will be used in the next tool development (Interval Data).

Distance Matrix / Merge

This takes a file created by the White-Mihoff False Filtering tool. The file contains rows of station pairs with the straight line distance between them calculated in metres. A station pair will only be in the file if an animal traveled between the stations. If an animal goes from stn1 to stn2 and then to stn3, pairs stn1-stn2 and stn2-stn3 will be in the file. If no animal goes between stn1 and stn3, that pair will not be in the file. The tool also takes a file that the researcher provides of ‘real distances’.  The output will be a file which looks like the first file with the ‘real distance’ column updated. Details are in Distance Matrix Merge Instructions.docx

Mihoff Interval Data Product

(Mihoff, M., Jones, B., Bajona, L., Halfyard, E. 2014. Mihoff Interval Data Tool)

This tool will take a file of detections and a distance matrix file and output two files. One will be a compressed detection file. Records will contain the animal id, the arrival time at a station, the departure time at a station and the detection count at a station. The other file is the Interval data file. The Interval data file will contain records of the animal id, the arrival time at stn1, the departure time at stn1, the detection count at stn1, the arrival time at stn2, time between detections at the two stations, the interval in seconds, the distance between stations, and the velocity of the animal in m/s. Details are in Mihoff Interval Data Instructions.docx

Compressed Detection Data Product

This is the same as the first step of the Mihoff Interval Data Tool. It will produce only the compressed data file. The file is used for several tools so there is a need for a separate tool. Details are in Compression Tool Instructions.docx

Cohort Data Product

The tool takes a file of compressed detections and a time parameter in minutes. It identifies groups of animals traveling together. Each station a animal visits is checked for other animals detected there within the specified time period. Details are in Cohort Data Tool.docx

Add Unique Id

This tool will add a column to any file. The unique id will be sequential integers. No validation is done on the input file. Details in Add Column unqdetecid Tool.docx


Tips for a better processing experience:

  • Limit detection files to no more than two million records.  If your files are larger than that break them down by year, or a set of animals.
  • Bookmark the URL. This is not an internet link but a link to the OTN Sandbox running on your desktop. The link may not work if you are running a VPN.
  • Link to most up to date version of the software and tools https://members.oceantrack.org/toolbox/.
  • Installation Instructions in: Install OTN Sandbox.docx.
  • URL to open in your favourite browser http://localhost:18787. This only works if you have the OTN sandbox installed and have it running.
  • User/PW  vagrant/otn123
  • Each tool has its own detailed instruction document.
  • When loading a new version of the OTN Sandbox follow instructions in documents:

 


Things Running Slowly?

When you get to using the tool and your submission is taking longer than half an hour you may need to allocate more CPU to the Sandbox VM. Instructions are in the Appendices (see Installation Instructions documentation). If this does not work please contact us and we will see what we can do. There are other options but they are platform-specific.

Have fun! And please contact marta.mihoff@dal.ca with any errors you find. Or if something is not clear in the documentation. Or if there is something you wish worked differently. Or simply something you wish for.