8. Cartech Plugin

The Cartech Plugin is similar to the BMS plugin. It is responsible for publishing battery data for the FCA Cartech team. The data is typically provided as .csv. The output on d3View is 5 published responses: JobModule#, Current, Voltage, Temp, Ah. It offers functionality for publishing ‘continued’ data sets, provided as ‘test’_Resume.csv data. Support has been added for an additional type of test containing data files ending with PART1.005, which are tsv-like files (similar to csv files but with tabs as the delimiter rather than commas).

8.1. Command Line Arguments

Command Line Args
Name Mapping (dest) Default Action Metavar Help Group
-c config_file   NA NA config_file NA
-d recursive_directory   NA NA Directory NA
-f cartech_filename   NA NA CarTech file name NA
-recursive recursive   store_true NA True if the directory should be searched recursively NA
-rm-repeating rm_repeating NA store_true NA Remove repeating values NA
-timeout timeout   NA NA Number of seconds before killing the plugin NA
-prescan prescan   store_true NA Scan for tests in a directory which have not been published. Logs results, optional: output CSV with -scan-output NA
-prescan-output prescan_output missing_physical_tests NA NA Output Missing Files NA
-prescan-output-dir prescan_output_dir   NA NA Dir for missing file output NA
-publish publish   store_true NA Use this flag to publish a physical test to d3VIEW Publishing
-d3view-url api_url NA NA api_url Url used to navigate to d3VIEW. Publishing
-d3view-port api_port NA NA api_port Port used to navigate to d3VIEW. Publishing
-u, -user user_id NA NA username Name of the user as shown in d3VIEW Publishing
-a, -api-key api_key NA NA api_key Api key corresponding to the d3VIEW user specified Publishing
-application-key application_key NA NA API_KEY Application key Publishing
-program-name program_name   NA NA Name of the program associated with the physical test Publishing
-project-name project_name   NA NA Name of the project associated with the physical test Publishing
-template response_template   NA NA Template to apply Publishing
-replace remove_previous NA store_true NA Replace existing physical test Publishing
-restart restart NA store_true NA Restart using an extracted folder Publishing
-skip-files skip_files NA store_true NA Skip posting files Publishing
-ppt ppt NA store_true NA Generate PowerPoint Publishing
-scratch-dir scratch_dir . NA NA Scratch directory Publishing
-apply-template apply_template NA store_true NA Use this flag to apply a template to physical test in d3VIEW Update
-meta-data meta_data NA store_true NA Update only metadata for physical test Update
-project-update project_update NA store_true NA Update the Project for a given Physical Test NA
-update-names update_names NA store_true NA Updates Car-Tech Physical Tests’ names with a date prefix NA
-update-existing update_existing NA store_true NA When provided, updates Existing Physical Tests with new data found NA
-prefix prefix NA NA NA Prefix for published test name Publishing
-suffix suffix NA NA NA Suffix for published test name Publishing
-thread-count thread_count 4 NA NA Thread count Publishing
-verbose unencrypt NA store_true NA NA NA
-run-parser run_parser NA store_true NA Run the parser for all file names in the file test_data/cartech/all_ct_files_names.csv NA
-chunk-lines chunk_lines 10000 NA NA Run the parser for all file names in the file test_data/cartech/all_ct_files_names.csv NA

8.2. Cartech Features

Features
Feature Function
-publish (and none of the below) Extract responses, Publish Physical Test, Apply Template (if provided with -r ‘template_id’)
-apply-template Apply a template to an already existing test, or repair existing template responses
-prescan Generate a .csv output of containing the names of all tests that scanned that are not present on d3VIEW
-project-update Change which “Project” a PT is saved under
-meta-data Update Meta Data for a published physical test
-update-names Update and repair any missing files associated with PT (example, .wr files)

8.3. Key Arguments

-publish : Enables publishing to d3VIEW

-d : The directory to search for cartech csv files

-recursive : the flag to indicate that the cartech csv files may be nested in sub-directories

-replace : If a previously existing test is a match for the current submission, delete the test and republish

8.4. Example Scripts / Cronjobs

The CarTech plugin shares many of the same options as the BMS plugin.

8.5. Publish New Script / Cronjob

Given a target directory, search for recently published tests (within $days) and attempt to publish

#!/usr/bin/env bash

pattern="*.csv"
thread_count=1
days=1
lucy_install=/cae/hpcjobs/service/test_lucy_june7
import_dir=/cae/hpcjobs/BMS_Data/FCA_Cell_Test_Data_R/
scratch_dir=/TMP1/d3view/scratch/current
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
#d3VIEW_url="http://d3view.intra.chrysler.com:3080"

num_of_files=$(find "$import_dir" -type f -name "$pattern" -mtime -"$days" -printf '.' | wc -c)
echo "Number of Test Files to process: ${num_of_files}"
time_to_process=$((($num_of_files * 272)/60))
echo "Estimated time till completion: ${time_to_process} minutes "

files=`find "$import_dir" -type f -name "$pattern" -mtime -"$days" | sort`

for file in $files; do
echo "Found file: $file"

"$lucy_install"/bin/lucy plugins cartech -publish \
                                     -user t5913sb \
                                     -f "$file" \
                                     -application-key lucy \
                                     -d3view-url ${d3VIEW_url} \
                                     -d "$import_dir" \
                                     -rm-repeating  \
                                     -timeout 1200 \
                                     -thread-count ${thread_count} \
                                     -recursive \
                     -scratch-dir ${scratch_dir} \
                     -project-name 176 \
                     -verbose  \
                     -update-existing

done

8.6. Prescan / Cronjob

#!/usr/bin/env bash

pattern=".*\.csv"
thread_count=8
days=60
lucy_install=/cae/hpcjobs/service/lucy_june2
import_dir=/cae/hpcjobs/BMS_Data/FCA_Cell_Test_Data_R/
# scratch_dir=/TMP1/d3view/scratch/current
scratch_dir=/cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish/local_scratch
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
#d3VIEW_url="http://d3view.intra.chrysler.com:3080"

num_of_files=$(find "$import_dir" -type f -name "$pattern" -mtime -"$days" -printf '.' | wc -c)
echo "Number of Test Files to process: ${num_of_files}"
time_to_process=$((($num_of_files * 272)/60))
echo "Estimated time till completion: ${time_to_process} minutes "

files=`find "$import_dir" -type f -name "$pattern" -mtime -"$days" | sort`

cd /TMP1/d3view/scratch/current/cron_scratch
mkdir $$
cd $$

"$lucy_install"/bin/lucy plugins cartech -publish \
                                     -prescan \
                                     -user t5913sb \
                                     -f "$pattern" \
                                     -application-key lucy \
                                     -d3view-url ${d3VIEW_url} \
                                     -d "$import_dir" \
                                     -timeout 1200 \
                                     -thread-count ${thread_count} \
                                     -recursive \
                                     -rm-repeating \
                                     -scratch-dir ${scratch_dir} \
                                     -project-name 176 \
                                     -verbose \
                                     -prescan-output-dir /cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish


mv lucy.log ../$$_lucy.log
cd ..
rm -r $$

8.7. Post Scan Publish Missing / Cronjob

Read from the prescan output file and post those tests

#!/usr/bin/env bash

pattern=".*\.csv"
thread_count=1
days=60
lucy_install=/cae/hpcjobs/service/lucy_june2
import_dir=/cae/hpcjobs/BMS_Data/FCA_Cell_Test_Data_R/
# scratch_dir=/TMP1/d3view/scratch/current
scratch_dir=/cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish/local_scratch
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
#d3VIEW_url="http://d3view.intra.chrysler.com:3080"

num_of_files=$(find "$import_dir" -type f -name "$pattern" -mtime -"$days" -printf '.' | wc -c)
echo "Number of Test Files to process: ${num_of_files}"
time_to_process=$((($num_of_files * 272)/60))
echo "Estimated time till completion: ${time_to_process} minutes "

files=`find "$import_dir" -type f -name "$pattern" -mtime -"$days" | sort`

for log_file in $(ls /cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish | grep missing_physical_tests); do
    echo $log_file
    while IFS= read fname; do

"$lucy_install"/bin/lucy plugins cartech -publish \
                                     -user t5913sb \
                                     -f "${fname//,}.csv" \
                                     -application-key lucy \
                                     -d3view-url ${d3VIEW_url} \
                                     -d "$import_dir" \
                                     -timeout 1200 \
                                     -thread-count ${thread_count} \
                                     -recursive \
                                    -rm-repeating \
                     -scratch-dir ${scratch_dir} \
                     -project-name 176 \
                     -verbose  \
                     -update-existing

    done < /cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish/$log_file;
done;