7. Battery Data Publishing¶
This guide focuses on the Lucy CLI implementation of the BMS plugin, of which there is also an equivalent user friendlier GUI implementation within the desktop application.
The BMS plugin is responsible for publishing Battery Lab Test data, typically provided via a BMS.ssv file but the data file(s) may have any name. If BMS.ssv is the BMS data file, included with it should be a .wr file which includes meta-data on the test. BMS Can also publish “elements” data in csv files, which have names like: 2021-03-31_HEAD_CATL-1017_167Ah_0C_BOT_Drives_data.csv The only difference in publishing the BMS.ssv data and Elements.csv data is in the filename pattern provided. Response names are ‘mapped’ and converted into more human-readable response names. The templates rely on successful mappings.
7.1. Command Line Arguments¶
| Name | Mapping (dest) | Default | Action | Metavar | Help | Group |
|---|---|---|---|---|---|---|
| -d | recursive_directory | ‘empty string’ | NA | NA | Directory | NA |
| -f | bms_filename | ‘empty string’ | NA | NA | BMS file name | NA |
| -recursive | recursive | ‘empty string’ | store_true | NA | True if the directory should be searched recursively | NA |
| -time-channel | time_channel | Test_Time | NA | NA | Name of the time channel | NA |
| -drop-channel | drop_channel | False | NA | NA | Channels to drop | NA |
| -rsp-per-file | responses_per_file | ‘empty string’ | NA | NA | Number of responses per file | NA |
| -config-file | config_file | ‘empty string’ | NA | NA | config_file | NA |
| -rm-repeating | rm_repeating | NA | store_true | NA | Remove repeating values (only the points between the start and end, non-inclusive, of any consecutively repeating points whose size is greater than 2) | NA |
| -error-factor | error_factor | 0 | NA | NA | Error factor | NA |
| -point-increment | point_increment | ‘empty string’ | NA | NA | Parse every point_increment values from bms data | NA |
| -downsize | downsize | NA | store_true | NA | Use this flag to downsize the data points | NA |
| -clean-initial-point | clean_initial_point | NA | NA | NA | Clean out the initial points from specified curves | NA |
| -clean-final-zero | clean_trailing_zero | NA | store_true | NA | Clean out the trailing zero points | NA |
| -timeout | timeout | ‘empty string’ | NA | NA | Number of seconds before killing the plugin | NA |
| -program-filter | program_filter | ‘empty string’ | NA | NA | List of programs to filter based on | NA |
| -database-file | database_file | ‘empty string’ | NA | NA | Database file | NA |
| -xform-responses-per-call | response_chunk_size | ‘empty string’ | NA | NA | Database file | NA |
| -pre-scan | prescan | NA | store_true | NA | Scan before running | Prescan |
| -output-file | output_file | NA | NA | NA | Prescan output file | Prescan |
| -scratch-dir | scratch_dir | . | NA | NA | Scratch directory | Prescan |
| -thread-count | thread_count | 4 | NA | NA | Thread count | Prescan |
| -publish | publish | NA | store_true | NA | Use this flag to publish a physical test to d3VIEW | Publishing |
| -prescan-file | scan_database | NA | NA | NA | Pre scan data location | Publishing |
| -post-files | post_files | NA | NA | NA | Post files | Publishing |
| -d3view-url | api_url | NA | NA | api_url | Url used to navigate to d3VIEW. | Publishing |
| -d3view-port | api_port | NA | NA | api_port | Port used to navigate to d3VIEW. | Publishing |
| -u, -user | user_id | NA | NA | username | Name of the user as shown in d3VIEW | Publishing |
| -a, -api-key | api_key | NA | NA | api_key | Api key corresponding to the d3VIEW user specified | Publishing |
| -application-key | application_key | NA | NA | API_KEY | Application key | Publishing |
| -program-name | program_name | ‘empty string’ | NA | NA | Name of the program associated with the physical test | Publishing |
| -project-name | project_name | ‘empty string’ | NA | NA | Name of the project associated with the physical test | Publishing |
| -template | response_template | ‘empty string’ | NA | NA | Template to apply | Publishing |
| -template-mapping | template_mapping | NA | store_true | NA | Use the d3VIEW template to map data | Publishing |
| -replace | remove_previous | NA | store_true | NA | Replace differing responses in existing physical test | Publishing |
| -force-replace | force_remove_previous | NA | store_true | NA | Replace existing physical test | Publishing |
| -verify | verify_responses | NA | store_true | NA | Verify local responses against existing physical test | Publishing |
| -restart | restart | NA | store_true | NA | Restart using an extracted folder | Publishing |
| -single-upload | single_upload | NA | store_true | NA | Upload responses one at a time | Publishing |
| -skip-files | skip_files | NA | store_true | NA | Skip posting files | Publishing |
| -ppt | ppt | NA | store_true | NA | Generate PowerPoint | Publishing |
| -remove-duplicate-responses, -rdr | remove_duplicate_responses | NA | store_true | NA | Pre-Publish: Remove empty duplicate response curves before sending to d3VIEW. Post: Remove duplicated responses. | Publishing |
| -update-overlays, -uo | update_overlays | ‘empty string’ | NA | NA | Deletes and then re-extracts existing overlays. | Updating |
| -update-files | update_files | ‘empty string’ | NA | NA | Use this flag to apply a template to physical test in d3VIEW | Updating |
| -apply-template | apply_template | NA | store_true | NA | Use this flag to apply a template to physical test in d3VIEW | Updating |
| -meta-data | meta_data | NA | store_true | NA | Update only metadata for physical test | Updating |
| -project-update | project_update | NA | store_true | NA | Update the Project for a given Physical Test | Updating |
| -send-mail | send_mail | NA | store_true | NA | Sends a report of published/skipped tests via sendmail | |
| -from | from | ‘empty string’ | NA | NA | From email address for published/skipped test report | |
| -to | to | ‘empty string’ | NA | NA | To email address for published/skipped test report | |
| -prefix | prefix | NA | NA | NA | Prefix for published test name | Publishing |
| -suffix | suffix | NA | NA | NA | Suffix for published test name | Publishing |
| -mapping-file | mapping_file | ‘empty string’ | NA | NA | Mapping file path | Publishing |
| -mapping-output | mapping_output | NA | store_true | NA | Generates output file for physical test mapping | NA |
| -verbose | unencrypt | NA | store_true | NA | NA | NA |
| -cache | cache | NA | store_true | NA | NA | NA |
| -sanitize-responses | sanitize_responses | NA | store_true | NA | NA | NA |
| -api-cleaner | api_response_cleaner | NA | store_true | NA | NA | NA |
| -use-mapper | use_mapper | NA | store_true | NA | Specify whether to use the mapper file to map source and destination names. Typically this should always be provided | Publishing |
| -use-mse | use_mse | NA | store_true | NA | Specify whether to use MSE-based mapping (even if CrossCheck column values contain ‘yes’ in the mapping file) | Publishing |
| -rdp | rdp | 0.0 | NA | NA | Use RDP (Ramer-Douglas-Peucker) algorithm to reduce points for each response, argument is the epsilon value. The d3VIEW worker’s default epsilon value is 1.0, an argument of 0.0 (default value) indicates to not use the RDP algorithm | Publishing |
| -normalize | normalize | NA | store_true | NA | Use in conjunction with -rdp; applies min-max normalization (maps every y-value to a value within [0, 1]) | Publishing |
| -keep-columns | keep_columns | ‘’ | NA | NA | Comma-separated list of column names to keep from the data file. Only this list of columns will be used from he data file | Publishing |
| -skip-columns | skip_columns | ‘’ | NA | NA | Comma-separated list of column names to skip from the data file. This list of columns will not be used from he data file | Publishing |
| -compare-tests | compare_tests | ‘’ | NA | NA | Compare differences in responses between two fully published tests. Arguments shall be the test IDs of 2 physical tests separated by a comma | Publishing |
| -compare-tests-metric | compare_tests_metric | ‘raw’ | NA | NA | Metric to use when comparing differences between tests. Calls curves_match worker. Default value is raw if this option is not provided. Other possible values are: pdtw, abs, max, max-max-diff, critical_resultant, euclidean-distance, frechet, dynamic-time-warping, ratio_ydiff | Publishing |
| -compare-tests-output | compare_tests_output | ‘’ | NA | NA | Compare differences in responses between two fully published tests. Use in conjunction with -compare-tests. This argument should be the path of the output file that highlights the differences | Publishing |
7.2. BMS Features¶
| Feature | Function |
|---|---|
| -publish (and none of the below) | Extract responses, Publish Physical Test, Apply Template (if provided with -r ‘template_id’) |
| -sanitize-responses | Clean responses for an existing test (Remove duplicates) |
| -apply-template | Apply a template to an already existing test, or repair existing template responses |
| -prescan | Generate a table of data about all scanned Tests, including if it has been published to d3VIEW or not |
| -project-update | Change which “Project” a PT is saved under |
| -meta-data | Update Meta Data for a published physical test |
| -update-files | Update and repair any missing files associated with PT (example, .wr files) |
| ELEMENTS (Publishing ELEMENTS .csv files) | The BMS plugin can alternatively be provided with a .csv containing ‘Elements’ data and process it. |
7.3. Key Arguments¶
-d : Required. The directory to search for BMS.ssv (or ‘pattern’) files.
-f : Required. The search pattern for the BMS file name. If, for instance, you have multiple element tests in a folder, you can set that folder to be -d and then set -f to be “*.csv” to search for all csv files
-recursive : Recommended. The flag to indicate that the .ssv files may be nested in sub-directories
-time-channel : Required. The name of the time channel.
-mapping-file : Provides the relationship mapping data to take output from the BMS.ssv and make it more human readable. Mapped responses are also required for templates to apply.
-mapping-output <output directory/path> : Optional, but will provide an overview of the mapped responses as .csv file
-publish : without this, the results will be extracted, but we will not publish the PT
-replace : Optional, replaces differing responses when comparing the local responses vs remote for a previously published test
-force-replace : Deletes the original physical test, replaces with a new one
7.4. Example Scripts¶
Typically you will always want to provide the -r, -d, -u (or, equivalently, -user), -a (or, equivalently, -api-key), -d3view-url, -application-key, -time-channel. You will also very likely want to include -mapping-file.
7.5. Submit Script¶
Here is an example of a simple submit script to extract and publish a BMS Physical Test. The key flag is -publish, and if you are republishing a test that already exists, -force-replace will overwrite the existing test with your newly published test.
#!/bin/sh
# Short file
file=~/sandbox/data/21F0048/BMS.ssv
~/sandbox/lucymar11/bin/lucy plugins bms \
-d3view-url http://10.1.10.205 \
-d3view-port 3091 \
-a 'api-key' \
-application-key lucy \
-f ${file} \
-user eric.tang \
-publish \
-post-files wr,Protocol.txt \
-prefix '01_' \
-scratch-dir ~/sandbox/data/scratch/ \
-mapping-file ~/sandbox/data/21F0048/BMS_MAPPING_03-08-2021_17.23.csv \
-mapping-output True \
-replace \
-remove-duplicate-responses
7.6. Sanitize Responses¶
This example demonstrates how to sanitize responses from existing, published tests. Response sanitize includes the ability to remove responses with null values and remove duplicate responses from publication. The -uo option takes Overlay Names and updates the composite expression, then re-extracts.
#!/bin/sh
# Short file
file=~/sandbox/data/21F0048/BMS.ssv
~/sandbox/lucymar11/bin/lucy plugins bms \
-d3view-url http://10.1.10.205 \
-d3view-port 3091 \
-a 'api-key' \
-application-key lucy \
-f ${file} \
-r 'template_id' \
-user will.wharton \
-sanitize-responses \
-remove-duplicate-responses \
-uo 'Cell Voltage Overlays,Cell Temperature Overlays,Cell Temperatures Overlays,Cell Voltages Overlay'
7.7. Prescan¶
Scans a test file to identify if it has been published to d3VIEW or not (among other details like status, total channels, total points, file size, and datetime) and outputs to a tsv.
#!/bin/sh
# Short file
file=~/sandbox/data/21F0048/BMS.ssv
~/sandbox/lucymar11/bin/lucy plugins bms \
-d3view-url http://10.1.10.205 \
-d3view-port 3091 \
-a 'api-key' \
-application-key lucy \
-f ${file} \
-r 'template_id' \
-user will.wharton \
-prescan \
-prescan-output 'dir_path'
7.8. Apply Template¶
Takes an existing test and applies template extractions / responses.
#!/bin/sh
# Short file
file=~/sandbox/data/21F0048/BMS.ssv
~/sandbox/lucymar11/bin/lucy plugins bms \
-d3view-url http://10.1.10.205 \
-d3view-port 3091 \
-a 'api-key' \
-application-key lucy \
-f ${file} \
-user eric.tang \
-apply-template \
-r 'template_id'
7.9. Meta Data¶
Updates tests based on the content of the .wr folder
#!/bin/sh
# Short file
file=~/sandbox/data/21F0048/BMS.ssv
~/sandbox/lucymar11/bin/lucy plugins bms \
-d3view-url http://10.1.10.205 \
-d3view-port 3091 \
-a 'api-key' \
-application-key lucy \
-f ${file} \
-user eric.tang \
-meta-data
7.10. Update Files¶
Looks for missing files for a test and will upload the missing files if present
#!/bin/sh
# Short file
file=~/sandbox/data/21F0048/BMS.ssv
~/sandbox/lucymar11/bin/lucy plugins bms \
-d3view-url http://10.1.10.205 \
-d3view-port 3091 \
-a 'api-key' \
-application-key lucy \
-f ${file} \
-user eric.tang \
-update-files
7.11. Project Update¶
Updates the test to sync with a project (by id)
#!/bin/sh
# Short file
file=~/sandbox/data/21F0048/BMS.ssv
~/sandbox/lucymar11/bin/lucy plugins bms \
-d3view-url http://10.1.10.205 \
-d3view-port 3091 \
-a 'api-key' \
-application-key lucy \
-f ${file} \
-user eric.tang \
-project 174 \
-project-update
7.12. Comparing Tests and Filtering Responses¶
Compare the differences of the mutual responses shared between 2 physical tests, whether they be 2 existing published tests or one existing published test and the test that is to be published via execution of the current script (whose physical test ID is not yet known). The former can be executed by providing 2 physical test IDs separated by comma (remember to enclose in quotes if a space is present) and the latter can be done by just providing the single physical test ID to compare against.
plugins
bms
-publish
-user
eric.tang
-application-key
lucy
-a
a320f40eb02ccd5f3942a4f10e26b490cd44f782
-d3view-url
http://portal.d3view.com:3091
-mapping-file
/Users/erictang/Desktop/lucy-source/lucy/test_data/bms/element_test_2/BMS_MAPPING.csv
-d
/Users/erictang/Desktop/lucy-source/lucy/test_data/bms/element_test_2
-f
"*.\csv"
-scratch-dir
/Users/erictang/Desktop/lucy-source/lucy/test_data/bms/element_test_2/test_scratch
-time-channel
RecordingTime
-timeout
2400
-error-factor
0
-thread-count
1
-rsp-per-file
250
-template
30025
-recursive
-project-name
BMS_project
-clean-initial-point
outlier
-clean-final-zero
-match-type
-use-mapper
-auto-map
-post-files
wr,Protocol.txt
-verbose
-compare-tests-metric
pdtw
-compare-tests
"2325980, 2381534"
-compare-tests-output
/Users/erictang/Desktop/lucy-source/lucy/test_data/bms/element_test_2/compare_tests.json
-rm-repeating
-clean-initial-point
outlier
Cronjobs¶
Example Cronjob Importer Series:
- Morning Import
- Scan for Missing Tests
- Import Missing Tests
- Only imports tests flagged as missing during Scan
- Afternoon Import
- Night-time Import with -replace
7.13. Example Import¶
This script scans a directory for BMS test directories, copies the directories over based on date, and then publishes the data
years=(2021 2020)
from_dir_base=/mnt/uploads4/P/hevapps/uploads/WorkRequests/
to_dir_base="/TMP1/d3view/current/"
debug="no"
days=1
pattern=".*\.ssv"
lucy_install=/cae/hpcjobs/service/lucy_may21
bms_import_dir_base=$to_dir_base
scratch_dir=/TMP1/d3view/scratch/current
bms_data_dir=/cae/hpcjobs/apurva/BMS/data/
bms_mapping_file=${bms_data_dir}/BMS_MAPPING_03-17-2021_16.55.csv
bms_database_file=${bms_data_dir}/hevWrInfo.tsv
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
function transfer_file {
from_file="$1"
base_dir=`basename "$from_file"`
to_dir="$to_dir_base""$2"
echo " Copying ${from_file} to $to_dir ... "
if [[ $debug = 'yes' ]]; then
return;
fi
cp -r "$from_file" "$to_dir"/
return
}
function publish_to_d3view {
file="$1"
year="$2"
echo " Publishing $file to d3VIEW..."
if [[ $debug = 'yes' ]]; then
return;
fi
"$lucy_install"/bin/lucy plugins bms -publish \
-user t5913sb \
-f ${pattern} \
-application-key lucy \
-d3view-url ${d3VIEW_url} \
-d "$bms_import_dir_base"/"$year"/"$file" \
-mapping-file "$bms_mapping_file" \
-scratch-dir ${scratch_dir} \
-time-channel Test_Time \
-rm-repeating \
-timeout 2400 \
-error-factor 0 \
-thread-count 1 \
-rsp-per-file 250 \
-template 9370 \
-project-name 174 \
-recursive \
-clean-initial-point all \
-clean-final-zero \
-post-files wr,Protocol.txt \
-verbose
}
for year in "${years[@]}"; do
from_dir="$from_dir_base""$year"
if [[ ! -d $from_dir ]]; then
echo " No from dir: $from_dir"
echo " Exiting..."
break;
fi
parent_files=`find $from_dir ! -path $from_dir -type d -mtime -${days} -name "*[0-9]" | sort`
for file in $parent_files; do
SUB="_"
folder=`basename "$file"`
echo " Reviewing $file"
if [[ "$folder" == *"$SUB"* ]]; then
echo "iteration $folder";
IFS="_"
read -a TOKENS <<< "$folder"
echo $TOKENS
parent_name=${TOKENS[0]}
echo "parent_name is $parent_name"
if [[ -d $from_dir/$parent_name && -e $from_dir/$parent_name/Completed.txt && -e $from_dir/$parent_name/BMS.ssv && $from_dir/$parent_name/*.wr ]] ; then
transfer_file "$file" "$year"
publish_to_d3view "$folder" "$year"
else
echo " Skipping $file"
fi
else
if [[ -e $from_dir/$folder/Completed.txt && $from_dir/$folder/BMS.ssv && $from_dir/$folder/*.wr ]] ; then
transfer_file "$file" "$year"
publish_to_d3view "$folder" "$year"
else
echo " Skipping $file"
fi
fi
done
done
7.14. Prescan¶
Scans a directory of possible bms test sub-directories, outputs findings to -output-file <filepath/filename> - Note, Lucy appends .tsv to the output file after the fact
#!/bin/sh
years=(2021 2020)
from_dir_base=/mnt/uploads4/P/hevapps/uploads/WorkRequests/
to_dir_base="/TMP1/d3view/current/"
debug="no"
days=3
pattern=".*\.ssv"
lucy_install=/cae/hpcjobs/service/lucy_may21
bms_import_dir_base=$to_dir_base
scratch_dir=/TMP1/d3view/scratch/current
bms_data_dir=/cae/hpcjobs/apurva/BMS/data/
#bms_mapping_file=${bms_data_dir}/BMS_MAPPING_01-21-21_12-48.csv
bms_mapping_file=${bms_data_dir}/BMS_MAPPING_03-17-2021_16.55.csv
bms_database_file=${bms_data_dir}/hevWrInfo.tsv
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
cd /TMP1/d3view/scratch/current/cron_scratch
mkdir $$
cd $$
"$lucy_install"/bin/lucy plugins bms \
-user t5913sb \
-f ${pattern} \
-application-key lucy \
-d3view-url ${d3VIEW_url} \
-pre-scan \
-output-file "${scratch_dir}"/missing_bms_tests \
-d "$bms_import_dir_base"/2021/ \
-mapping-file "$bms_mapping_file" \
-scratch-dir ${scratch_dir} \
-time-channel Test_Time \
-timeout 2400 \
-error-factor 0 \
-thread-count 1 \
-template 9370 \
-recursive \
-verbose
mv lucy.log ../$$_lucy.log
cd ..
rm -r $$
7.15. Publish Missing¶
Publishes missing tests as identified by the prescan, requires the ‘-prescan-file’ arg to point to the prescan tsv output
#!/bin/sh
years=(2021 2020)
from_dir_base=/mnt/uploads4/P/hevapps/uploads/WorkRequests/
to_dir_base="/TMP1/d3view/current/"
debug="no"
days=1
pattern=".*\.ssv"
#lucy_install=/cae/hpcjobs/apurva/lucy_jan06_21
#lucy_install=/cae/hpcjobs/apurva/lucy_april_28_2021
lucy_install=/cae/hpcjobs/service/test_lucy_june7
bms_import_dir_base=$to_dir_base
scratch_dir=/TMP1/d3view/scratch/current
bms_data_dir=/cae/hpcjobs/apurva/BMS/data/
#bms_mapping_file=${bms_data_dir}/BMS_MAPPING_01-21-21_12-48.csv
bms_mapping_file=${bms_data_dir}/BMS_MAPPING_03-17-2021_16.55.csv
bms_database_file=${bms_data_dir}/hevWrInfo.tsv
d3VIEW_url="https://d3view-api.intra.chrysler.com:443"
cd /TMP1/d3view/scratch/current/cron_scratch
mkdir $$
cd $$
"$lucy_install"/bin/lucy plugins bms -publish \
-user t5913sb \
-f ${pattern} \
-application-key lucy \
-d3view-url ${d3VIEW_url} \
-mapping-file "$bms_mapping_file" \
-scratch-dir ${scratch_dir} \
-time-channel Test_Time \
-rm-repeating \
-prescan-file "${scratch_dir}"/missing_bms_tests.tsv \
-timeout 2400 \
-error-factor 0 \
-thread-count 1 \
-rsp-per-file 250 \
-template 9370 \
-project-name 174 \
-recursive \
-clean-initial-point all \
-clean-final-zero \
-post-files wr,Protocol.txt \
-verbose
mv lucy.log ../$$_lucy.log
cd ..
rm -r $$
7.16. Elements (CSV) Submit Script Example¶
This example includes a hardcoded file name (-f option argument), but a pattern can be used like with the .ssv BMS pattern
"$lucy_install"/bin/lucy plugins bms -publish \
-user t5913sb \
-f "R_EL_U_20160709094157_CustRec.csv" \
-application-key lucy \
-d3view-url ${d3VIEW_url} \
-mapping-file "$bms_mapping_file" \
-d /cae/hpcjobs/users/t5913sb/incoming \
-replace \
-scratch-dir ${scratch_dir} \
-time-channel RecordingTime \
-rm-repeating \
-timeout 2400 \
-error-factor 0 \
-thread-count 1 \
-rsp-per-file 250 \
-template 9451 \
-recursive \
-clean-initial-point all \
-replace \
-clean-final-zero \
-post-files wr,Protocol.txt \
-verbose