8. Cartech Plugin¶
The Cartech Plugin is similar to the BMS plugin. It is responsible for publishing battery data for the FCA Cartech team. The data is typically provided as .csv. The output on d3View is 5 published responses: JobModule#, Current, Voltage, Temp, Ah. It offers functionality for publishing ‘continued’ data sets, provided as ‘test’_Resume.csv data. Support has been added for an additional type of test containing data files ending with PART1.005, which are tsv-like files (similar to csv files but with tabs as the delimiter rather than commas).
8.1. Command Line Arguments¶
| Name | Mapping (dest) | Default | Action | Metavar | Help | Group |
|---|---|---|---|---|---|---|
| -c | config_file | NA | NA | config_file | NA | |
| -d | recursive_directory | NA | NA | Directory | NA | |
| -f | cartech_filename | NA | NA | CarTech file name | NA | |
| -recursive | recursive | store_true | NA | True if the directory should be searched recursively | NA | |
| -rm-repeating | rm_repeating | NA | store_true | NA | Remove repeating values | NA |
| -timeout | timeout | NA | NA | Number of seconds before killing the plugin | NA | |
| -prescan | prescan | store_true | NA | Scan for tests in a directory which have not been published. Logs results, optional: output CSV with -scan-output | NA | |
| -prescan-output | prescan_output | missing_physical_tests | NA | NA | Output Missing Files | NA |
| -prescan-output-dir | prescan_output_dir | NA | NA | Dir for missing file output | NA | |
| -publish | publish | store_true | NA | Use this flag to publish a physical test to d3VIEW | Publishing | |
| -d3view-url | api_url | NA | NA | api_url | Url used to navigate to d3VIEW. | Publishing |
| -d3view-port | api_port | NA | NA | api_port | Port used to navigate to d3VIEW. | Publishing |
| -u, -user | user_id | NA | NA | username | Name of the user as shown in d3VIEW | Publishing |
| -a, -api-key | api_key | NA | NA | api_key | Api key corresponding to the d3VIEW user specified | Publishing |
| -application-key | application_key | NA | NA | API_KEY | Application key | Publishing |
| -program-name | program_name | NA | NA | Name of the program associated with the physical test | Publishing | |
| -project-name | project_name | NA | NA | Name of the project associated with the physical test | Publishing | |
| -template | response_template | NA | NA | Template to apply | Publishing | |
| -replace | remove_previous | NA | store_true | NA | Replace existing physical test | Publishing |
| -restart | restart | NA | store_true | NA | Restart using an extracted folder | Publishing |
| -skip-files | skip_files | NA | store_true | NA | Skip posting files | Publishing |
| -ppt | ppt | NA | store_true | NA | Generate PowerPoint | Publishing |
| -scratch-dir | scratch_dir | . | NA | NA | Scratch directory | Publishing |
| -apply-template | apply_template | NA | store_true | NA | Use this flag to apply a template to physical test in d3VIEW | Update |
| -meta-data | meta_data | NA | store_true | NA | Update only metadata for physical test | Update |
| -project-update | project_update | NA | store_true | NA | Update the Project for a given Physical Test | NA |
| -update-names | update_names | NA | store_true | NA | Updates Car-Tech Physical Tests’ names with a date prefix | NA |
| -update-existing | update_existing | NA | store_true | NA | When provided, updates Existing Physical Tests with new data found | NA |
| -prefix | prefix | NA | NA | NA | Prefix for published test name | Publishing |
| -suffix | suffix | NA | NA | NA | Suffix for published test name | Publishing |
| -thread-count | thread_count | 4 | NA | NA | Thread count | Publishing |
| -verbose | unencrypt | NA | store_true | NA | NA | NA |
| -run-parser | run_parser | NA | store_true | NA | Run the parser for all file names in the file test_data/cartech/all_ct_files_names.csv | NA |
| -chunk-lines | chunk_lines | 10000 | NA | NA | Run the parser for all file names in the file test_data/cartech/all_ct_files_names.csv | NA |
8.2. Cartech Features¶
| Feature | Function |
|---|---|
| -publish (and none of the below) | Extract responses, Publish Physical Test, Apply Template (if provided with -r ‘template_id’) |
| -apply-template | Apply a template to an already existing test, or repair existing template responses |
| -prescan | Generate a .csv output of containing the names of all tests that scanned that are not present on d3VIEW |
| -project-update | Change which “Project” a PT is saved under |
| -meta-data | Update Meta Data for a published physical test |
| -update-names | Update and repair any missing files associated with PT (example, .wr files) |
8.3. Key Arguments¶
-publish : Enables publishing to d3VIEW
-d : The directory to search for cartech csv files
-recursive : the flag to indicate that the cartech csv files may be nested in sub-directories
-replace : If a previously existing test is a match for the current submission, delete the test and republish
8.4. Example Scripts / Cronjobs¶
The CarTech plugin shares many of the same options as the BMS plugin.
8.5. Publish New Script / Cronjob¶
Given a target directory, search for recently published tests (within $days) and attempt to publish
#!/usr/bin/env bash
pattern="*.csv"
thread_count=1
days=1
lucy_install=/cae/hpcjobs/service/test_lucy_june7
import_dir=/cae/hpcjobs/BMS_Data/FCA_Cell_Test_Data_R/
scratch_dir=/TMP1/d3view/scratch/current
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
#d3VIEW_url="http://d3view.intra.chrysler.com:3080"
num_of_files=$(find "$import_dir" -type f -name "$pattern" -mtime -"$days" -printf '.' | wc -c)
echo "Number of Test Files to process: ${num_of_files}"
time_to_process=$((($num_of_files * 272)/60))
echo "Estimated time till completion: ${time_to_process} minutes "
files=`find "$import_dir" -type f -name "$pattern" -mtime -"$days" | sort`
for file in $files; do
echo "Found file: $file"
"$lucy_install"/bin/lucy plugins cartech -publish \
-user t5913sb \
-f "$file" \
-application-key lucy \
-d3view-url ${d3VIEW_url} \
-d "$import_dir" \
-rm-repeating \
-timeout 1200 \
-thread-count ${thread_count} \
-recursive \
-scratch-dir ${scratch_dir} \
-project-name 176 \
-verbose \
-update-existing
done
8.6. Prescan / Cronjob¶
#!/usr/bin/env bash
pattern=".*\.csv"
thread_count=8
days=60
lucy_install=/cae/hpcjobs/service/lucy_june2
import_dir=/cae/hpcjobs/BMS_Data/FCA_Cell_Test_Data_R/
# scratch_dir=/TMP1/d3view/scratch/current
scratch_dir=/cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish/local_scratch
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
#d3VIEW_url="http://d3view.intra.chrysler.com:3080"
num_of_files=$(find "$import_dir" -type f -name "$pattern" -mtime -"$days" -printf '.' | wc -c)
echo "Number of Test Files to process: ${num_of_files}"
time_to_process=$((($num_of_files * 272)/60))
echo "Estimated time till completion: ${time_to_process} minutes "
files=`find "$import_dir" -type f -name "$pattern" -mtime -"$days" | sort`
cd /TMP1/d3view/scratch/current/cron_scratch
mkdir $$
cd $$
"$lucy_install"/bin/lucy plugins cartech -publish \
-prescan \
-user t5913sb \
-f "$pattern" \
-application-key lucy \
-d3view-url ${d3VIEW_url} \
-d "$import_dir" \
-timeout 1200 \
-thread-count ${thread_count} \
-recursive \
-rm-repeating \
-scratch-dir ${scratch_dir} \
-project-name 176 \
-verbose \
-prescan-output-dir /cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish
mv lucy.log ../$$_lucy.log
cd ..
rm -r $$
8.7. Post Scan Publish Missing / Cronjob¶
Read from the prescan output file and post those tests
#!/usr/bin/env bash
pattern=".*\.csv"
thread_count=1
days=60
lucy_install=/cae/hpcjobs/service/lucy_june2
import_dir=/cae/hpcjobs/BMS_Data/FCA_Cell_Test_Data_R/
# scratch_dir=/TMP1/d3view/scratch/current
scratch_dir=/cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish/local_scratch
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
#d3VIEW_url="http://d3view.intra.chrysler.com:3080"
num_of_files=$(find "$import_dir" -type f -name "$pattern" -mtime -"$days" -printf '.' | wc -c)
echo "Number of Test Files to process: ${num_of_files}"
time_to_process=$((($num_of_files * 272)/60))
echo "Estimated time till completion: ${time_to_process} minutes "
files=`find "$import_dir" -type f -name "$pattern" -mtime -"$days" | sort`
for log_file in $(ls /cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish | grep missing_physical_tests); do
echo $log_file
while IFS= read fname; do
"$lucy_install"/bin/lucy plugins cartech -publish \
-user t5913sb \
-f "${fname//,}.csv" \
-application-key lucy \
-d3view-url ${d3VIEW_url} \
-d "$import_dir" \
-timeout 1200 \
-thread-count ${thread_count} \
-recursive \
-rm-repeating \
-scratch-dir ${scratch_dir} \
-project-name 176 \
-verbose \
-update-existing
done < /cae/hpcjobs/apurva/BMS/cronjobs/ct_parallel_publish/$log_file;
done;