TouchPortalAPI.sdk_tools
Touch Portal Python SDK Tools
Features
This SDK tool provides features for generating and validating Touch Portal Plugin Description files, which are in JSON format and typically named "entry.tp" (as described in the TP API docs).
Generation is done based on dictionary structures defined in plugin scripts. These structures closely follow the format of the description file JSON, but are further expanded to make them also useful within the plugin scripts themselves. This avoids almost all the need for duplication of things like unique IDs, default values, settings names, and so on, since those details need to be available in both the definition JSON and in the script which will be using those definitions to communicate with Touch Portal.
Validation is performed on the generated descriptions, and can also be run against pre-existing definition files as a sort of "lint" utility. Currently the following things are checked:
- All required attributes are present.
- All attribute values are of the supported data type(s).
- No unknown attributes are present.
- Attributes are valid for the TP SDK version being used.
- Attribute values fall within allowed list of values (if relevant, eg.
Action.type
). - All ID strings are unique within the plugin (eg. for States, Actions, etc).
This tool can be used as a command-line utility, or imported as a module to use the available functions programmatically.
The script command is tppsdk
when the TouchPortalAPI is installed (via pip or setup), or sdk_tool.py
when run directly
from this source. To use within a script, simply import in the usual way as needed. For example:
from TouchPortalAPI.sdk_tools import generateDefinitionFromScript
All generation/validation routines are based on TP API specification data tables defined in TouchPortalAPI.sdk_spec
.
A comprehensive example plugin which utilizes the SDK features is included with the TouchPortalAPI project. The JSON definition "entry.tp" file for that example could be generated with a simple command from within a local folder containing the example script:
tppsdk plugin_example.py
Functions
- Generates an
entry.tp
definition file for a Touch Portal plugin based on variables specified in the plugin source code (generateDefinitionFromScript()
), from a loaded module (generateDefinitionFromModule()
) or specified by value (generateDefinitionFromDeclaration()
). - Validate an entire plugin definition (
entry.tp
) file (validateDefinitionFile()
), string (validateDefinitionString()
), or object (validateDefinitionObject()
). - Validate an
entry.tp
attribute value against the minimum SDK version, value type, value content, etc. (validateAttribValue()
).
Command-line Usage
The script command is tppsdk
when the TouchPortalAPI is installed (via pip or setup), or sdk_tool.py
when run directly from this source.
<script_command> [-h] [-g] [-v] [-o <file>] [-s] [-i <n>] [target]
positional arguments:
target Either a plugin script for `generate` or an entry.tp file for `validate`. Paths are relative to current working
directory. Defaults to './TPPEntry.py' and './entry.tp' respectively. Use 'stdin' (or '-') to read from input
stream instead. Another usage is if you pass in a normal entry.tp without `validate` argument, It can generate
Python version of entry.tp struct. It will be saved in `TPPEntry.py` if `output` is not given.
optional arguments:
-h, --help show this help message and exit
-g, --generate Generate a definition file from plugin script data. This is the default action.
-v, --validate Validate a definition JSON file (entry.tp). If given with `generate` then will validate the generated JSON output.
generator arguments:
-o <file> Output file for `generate` action. Default will be a file named 'entry.tp' in the same folder as the input script.
Paths are relative to current working directory. Use 'stdout' (or '-') to print the output to the console/stream instead.
-s, --skip-invalid Skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.
-i <n>, --indent <n> Indent level (spaces) for generated JSON. Use 0 for only newlines, or -1 for the most compact representation. Default is 2 spaces.
--noconfirm When generating python struct from entry.tp, you can pass this arg to bypass confirm if you want to contiune if any error is given for vaildating entry.tp before generate python struct.
This script exits with status code -1 (error) if generation or validation produces warning messages about malformed data.
All progress and warning messages are printed to stderr stream.
TODO/Ideas
- Document the default attribute values from sdk_specs table.
- Dynamic default values, eg. for action prefix or category id/name (see notes in sdk_spec tables).
- Dynamic ID generation and write-back to plugin at runtime.
- Allow plugin author to set their own defaults.
1#!/usr/bin/env python3 2""" 3# Touch Portal Python SDK Tools 4 5## Features 6 7This SDK tool provides features for generating and validating Touch Portal Plugin Description files, 8which are in JSON format and typically named "entry.tp" (as described in the [TP API docs](https://www.touch-portal.com/api/)). 9 10**Generation** is done based on dictionary structures defined in plugin scripts. These structures closely follow 11the format of the description file JSON, but are further expanded to make them also useful within the plugin 12scripts themselves. This avoids almost all the need for duplication of things like unique IDs, default values, 13settings names, and so on, since those details need to be available in both the definition JSON and in the 14script which will be using those definitions to communicate with Touch Portal. 15 16**Validation** is performed on the generated descriptions, and can also be run against pre-existing definition files 17as a sort of "lint" utility. Currently the following things are checked: 18- All required attributes are present. 19- All attribute values are of the supported data type(s). 20- No unknown attributes are present. 21- Attributes are valid for the TP SDK version being used. 22- Attribute values fall within allowed list of values (if relevant, eg. `Action.type`). 23- All ID strings are unique within the plugin (eg. for States, Actions, etc). 24 25This tool can be used as a command-line utility, or imported as a module to use the available functions programmatically. 26The script command is `tppsdk` when the TouchPortalAPI is installed (via pip or setup), or `sdk_tool.py` when run directly 27from this source. To use within a script, simply import in the usual way as needed. For example: 28 29```py 30from TouchPortalAPI.sdk_tools import generateDefinitionFromScript 31``` 32 33All generation/validation routines are based on TP API specification data tables defined in `TouchPortalAPI.sdk_spec`. 34 35A comprehensive example plugin which utilizes the SDK features is included with the TouchPortalAPI project. 36The JSON definition "entry.tp" file for that example could be generated with a simple command from within a local 37folder containing the example script: 38 39``` 40tppsdk plugin_example.py 41``` 42 43 44## Functions 45* Generates an `entry.tp` definition file for a Touch Portal plugin based 46on variables specified in the plugin source code (`generateDefinitionFromScript()`), 47from a loaded module (`generateDefinitionFromModule()`) or specified by value (`generateDefinitionFromDeclaration()`). 48* Validate an entire plugin definition (`entry.tp`) file (`validateDefinitionFile()`), 49string (`validateDefinitionString()`), or object (`validateDefinitionObject()`). 50* Validate an `entry.tp` attribute value against the minimum 51SDK version, value type, value content, etc. (`validateAttribValue()`). 52 53 54## Command-line Usage 55The script command is `tppsdk` when the TouchPortalAPI is installed (via pip or setup), or `sdk_tool.py` when run directly from this source. 56``` 57<script_command> [-h] [-g] [-v] [-o <file>] [-s] [-i <n>] [target] 58 59positional arguments: 60 target Either a plugin script for `generate` or an entry.tp file for `validate`. Paths are relative to current working 61 directory. Defaults to './TPPEntry.py' and './entry.tp' respectively. Use 'stdin' (or '-') to read from input 62 stream instead. Another usage is if you pass in a normal entry.tp without `validate` argument, It can generate 63 Python version of entry.tp struct. It will be saved in `TPPEntry.py` if `output` is not given. 64 65optional arguments: 66 -h, --help show this help message and exit 67 -g, --generate Generate a definition file from plugin script data. This is the default action. 68 -v, --validate Validate a definition JSON file (entry.tp). If given with `generate` then will validate the generated JSON output. 69 70generator arguments: 71 -o <file> Output file for `generate` action. Default will be a file named 'entry.tp' in the same folder as the input script. 72 Paths are relative to current working directory. Use 'stdout' (or '-') to print the output to the console/stream instead. 73 -s, --skip-invalid Skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them. 74 -i <n>, --indent <n> Indent level (spaces) for generated JSON. Use 0 for only newlines, or -1 for the most compact representation. Default is 2 spaces. 75 --noconfirm When generating python struct from entry.tp, you can pass this arg to bypass confirm if you want to contiune if any error is given for vaildating entry.tp before generate python struct. 76This script exits with status code -1 (error) if generation or validation produces warning messages about malformed data. 77All progress and warning messages are printed to stderr stream. 78``` 79 80## TODO/Ideas 81 82* Document the default attribute values from sdk_specs table. 83* Dynamic default values, eg. for action prefix or category id/name (see notes in sdk_spec tables). 84* Dynamic ID generation and write-back to plugin at runtime. 85* Allow plugin author to set their own defaults. 86""" 87 88__copyright__ = """ 89This file is part of the TouchPortal-API project. 90Copyright TouchPortal-API Developers 91Copyright (c) 2021 Maxim Paperno 92All rights reserved. 93 94This program is free software: you can redistribute it and/or modify 95it under the terms of the GNU General Public License as published by 96the Free Software Foundation, either version 3 of the License, or 97(at your option) any later version. 98 99This program is distributed in the hope that it will be useful, 100but WITHOUT ANY WARRANTY; without even the implied warranty of 101MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 102GNU General Public License for more details. 103 104You should have received a copy of the GNU General Public License 105along with this program. If not, see <https://www.gnu.org/licenses/>. 106""" 107 108import sys 109import os.path 110import importlib.util 111import json 112from types import ModuleType 113from typing import (Union, TextIO) 114from re import compile as re_compile 115 116sys.path.insert(0, os.path.dirname(os.path.realpath(__file__))) 117from sdk_spec import * 118from TpToPy import TpToPy 119 120## globals 121g_messages = [] # validation reporting 122g_seen_ids = {} # for validating unique IDs 123 124 125## Utils 126 127def getMessages(): 128 """ Gets a list of messages which may have been produced during generation/validation. 129 """ 130 return g_messages 131 132def clearMessages(): 133 """ Clears the list of validation warning messages. 134 Do this before invoking `validateAttribValue()` directly (outside of the other functions provided here). 135 """ 136 global g_messages 137 g_messages.clear() 138 139def _printMessages(messages:list): 140 for msg in messages: 141 _printToErr(msg) 142 143def _addMessage(msg): 144 global g_messages 145 g_messages.append(msg) 146 147 148def _seenIds(): 149 global g_seen_ids 150 return g_seen_ids.keys() 151 152def _addSeenId(id, path): 153 global g_seen_ids 154 g_seen_ids[id] = path 155 156def _clearSeenIds(): 157 global g_seen_ids 158 g_seen_ids.clear() 159 160 161def _printToErr(msg): 162 sys.stderr.write(msg + "\n") 163 164def _normPath(path): 165 if not isinstance(path, str): 166 return path 167 return os.path.normpath(os.path.join(os.getcwd(), path)) 168 169def _keyPath(path, key): 170 return ":".join(filter(None, [path, key])) 171 172## Generator functions 173 174def _dictFromItem(item:dict, table:dict, sdk_v:int, path:str="", skip_invalid:bool=False): 175 ret = {} 176 if not isinstance(item, dict): 177 return ret 178 for k, data in table.items(): 179 # try get explicit value from item 180 #if not data.get("doc"): continue 181 if (v := item.get(k)) is None: 182 # try get default value 183 v = data.get('d') 184 # check if there is nested data, eg. in an Action 185 if isinstance(v, dict) and data.get('t') is list: 186 v = _arrayFromDict(v, data.get('l', {}), sdk_v, path=_keyPath(path, k), skip_invalid=skip_invalid) 187 # check that the value is valid and add it to the dict if it is 188 if validateAttribValue(k, v, data, sdk_v, path) or (not skip_invalid and v != None): 189 ret[k] = v 190 # if this is the "sdk" value from TP_PLUGIN_INFO then reset the 191 # passed `sdk_v` param since it was originally set to TPSDK_DEFAULT_VERSION 192 if k == "sdk": 193 sdk_v = v 194 return ret 195 196 197def _arrayFromDict(d:dict, table:dict, sdk_v:int, category:str=None, path:str="", skip_invalid:bool=False): 198 ret = [] 199 if not isinstance(d, dict): 200 return ret 201 for key, item in d.items(): 202 if not category or not (cat := item.get('category')) or cat == category: 203 ret.append(_dictFromItem(item, table, sdk_v, f"{path}[{key}]", skip_invalid)) 204 if path in ["actions","connectors"]: 205 _replaceFormatTokens(ret) 206 return ret 207 208 209def _replaceFormatTokens(items:list): 210 for d in items: 211 if not isinstance(d, dict) or not 'format' in d.keys() or not 'data' in d.keys(): 212 continue 213 data_ids = {} 214 for data in d.get('data'): 215 if (did := data.get('id')): 216 data_ids[did.rsplit(".", 1)[-1]] = did 217 if not data_ids: 218 continue 219 fmt = d.get('format') 220 rx = re_compile(r'\$\[(\w+)\]') 221 begin = 0 222 while (m := rx.search(fmt, begin)): 223 idx = m.group(1) 224 if idx in data_ids.keys(): 225 val = data_ids.get(idx) 226 elif idx.isdigit() and (i := int(idx) - 1) >= 0 and i < len(data_ids): 227 val = list(data_ids.values())[i] 228 else: 229 begin = m.end() 230 _addMessage(f"WARNING: Could not find replacement for token '{idx}' in 'format' attribute for element `{d.get('id')}`. The data arry does not contain this name/index.") 231 continue 232 # print(m.span(), val) 233 fmt = fmt[:m.start()] + "{$" + val + "$}" + fmt[m.end():] 234 begin = m.start() + len(val) + 4 235 d['format'] = fmt 236 237 238def generateDefinitionFromScript(script:Union[str, TextIO], skip_invalid:bool=False): 239 """ 240 Returns an "entry.tp" Python `dict` which is suitable for direct conversion to JSON format. 241 242 `script` should be a valid python script, either a file path (ending in .py), string, or open file handle (like stdin). 243 The script should contain "SDK declaration variables" like `TP_PLUGIN_INFO`, `TP_PLUGIN_SETTINGS`, etc. 244 245 Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output). 246 Default behavior is to only warn about them. 247 248 Note that the script is interpreted (executed), so any actual "business" logic (like connecting to TP) should be in "__main__". 249 Also note that when using input from a file handle or string, the script's "__file__" attribute is set to the current working 250 directory and the file name "tp_plugin.py". 251 252 May raise an `ImportError` if the plugin script could not be loaded or is missing required variables. 253 Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation). 254 """ 255 script_str = "" 256 if hasattr(script, "read"): 257 script_str = script.read() 258 elif not script.endswith(".py"): 259 script_str = script 260 261 try: 262 if script_str: 263 # load from a string 264 spec = importlib.util.spec_from_loader("plugin", loader=None) 265 plugin = importlib.util.module_from_spec(spec) 266 setattr(plugin, "__file__", os.path.join(os.getcwd(), "tp_plugin.py")) 267 exec(script_str, plugin.__dict__) 268 else: 269 # load directly from a file path 270 spec = importlib.util.spec_from_file_location("plugin", script) 271 plugin = importlib.util.module_from_spec(spec) 272 spec.loader.exec_module(plugin) 273 # print(plugin.TP_PLUGIN_INFO) 274 except Exception as e: 275 input_name = "input stream" if script_str else script 276 raise ImportError(f"ERROR while trying to import plugin code from '{input_name}': {repr(e)}") 277 return generateDefinitionFromModule(plugin, skip_invalid) 278 279 280def generateDefinitionFromModule(plugin:ModuleType, skip_invalid:bool=False): 281 """ 282 Returns an "entry.tp" Python `dict`, which is suitable for direct conversion to JSON format. 283 `plugin` should be a loaded Python "module" which contains "SDK declaration variables" like 284 `TP_PLUGIN_INFO`, `TP_PLUGIN_SETTINGS`, etc. From within a plugin script this could be called like: 285 `generateDefinitionFromModule(sys.modules[__name__])`. 286 287 Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output). 288 Default behavior is to only warn about them. 289 290 May raise an `ImportError` if the plugin script is missing required variables `TP_PLUGIN_INFO` and `TP_PLUGIN_CATEGORIES`. 291 Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation). 292 """ 293 # Load the "standard SDK declaration variables" from plugin script into local scope 294 # INFO and CATEGORY are required, rest are optional. 295 if not (info := getattr(plugin, "TP_PLUGIN_INFO", {})): 296 raise ImportError(f"ERROR: Could not import required TP_PLUGIN_INFO variable from plugin source.") 297 if not (cats := getattr(plugin, "TP_PLUGIN_CATEGORIES", {})): 298 raise ImportError(f"ERROR: Could not import required TP_PLUGIN_CATEGORIES variable from plugin source.") 299 return generateDefinitionFromDeclaration( 300 info, cats, 301 settings = getattr(plugin, "TP_PLUGIN_SETTINGS", {}), 302 actions = getattr(plugin, "TP_PLUGIN_ACTIONS", {}), 303 states = getattr(plugin, "TP_PLUGIN_STATES", {}), 304 events = getattr(plugin, "TP_PLUGIN_EVENTS", {}), 305 connectors = getattr(plugin, "TP_PLUGIN_CONNECTORS", {}), 306 skip_invalid = skip_invalid 307 ) 308 309 310def generateDefinitionFromDeclaration(info:dict, categories:dict, skip_invalid:bool=False, **kwargs): 311 """ 312 Returns an "entry.tp" Python `dict` which is suitable for direct conversion to JSON format. 313 Arguments should contain SDK declaration dict values, for example as specified for `TP_PLUGIN_INFO`, etc. 314 315 The `info` and `category` values are required, the rest are optional. 316 317 Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output). 318 Default behavior is to warn about them but still include them in the output. 319 320 `**kwargs` can be one or more of: 321 - settings:dict = {}, 322 - actions:dict = {}, 323 - states:dict = {}, 324 - events:dict = {}, 325 - connectors:dict = {} 326 327 Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation). 328 """ 329 _clearSeenIds() 330 clearMessages() 331 settings = kwargs.get('settings', {}) 332 actions = kwargs.get('actions', {}) 333 states = kwargs.get('states', {}) 334 events = kwargs.get('events', {}) 335 connectors = kwargs.get('connectors', {}) 336 # print(info, categories, settings, actions, states, events, connectors) 337 338 # Start the root entry.tp object using basic plugin metadata 339 # This will also create an empty `categories` array in the root of the entry. 340 entry = _dictFromItem(info, TPSDK_ATTRIBS_ROOT, TPSDK_DEFAULT_VERSION, "info") 341 342 # Get the target SDK version (was either specified in plugin or is TPSDK_DEFAULT_VERSION) 343 tgt_sdk_v = entry['sdk'] 344 345 # Loop over each plugin category and set up actions, states, events, and connectors. 346 for cat, data in categories.items(): 347 path = f"category[{cat}]" 348 category = _dictFromItem(data, TPSDK_ATTRIBS_CATEGORY, tgt_sdk_v, path, skip_invalid) 349 category['actions'] = _arrayFromDict(actions, TPSDK_ATTRIBS_ACTION, tgt_sdk_v, cat, "actions", skip_invalid) 350 category['states'] = _arrayFromDict(states, TPSDK_ATTRIBS_STATE, tgt_sdk_v, cat, "states", skip_invalid) 351 category['events'] = _arrayFromDict(events, TPSDK_ATTRIBS_EVENT, tgt_sdk_v, cat, "events", skip_invalid) 352 if tgt_sdk_v >= 4: 353 category['connectors'] = _arrayFromDict(connectors, TPSDK_ATTRIBS_CONNECTOR, tgt_sdk_v, cat, "connectors", skip_invalid) 354 # add the category to entry's categories array 355 entry['categories'].append(category) 356 357 # Add Settings to root 358 if tgt_sdk_v >= 3: 359 entry['settings'].extend(_arrayFromDict(settings, TPSDK_ATTRIBS_SETTINGS, tgt_sdk_v, path = "settings", skip_invalid = skip_invalid)) 360 361 return entry 362 363 364## Validation functions 365 366def validateAttribValue(key:str, value, attrib_data:dict, sdk_v:int, path:str=""): 367 """ 368 Validates one attribute's value based on provided lookup table and target SDK version. 369 Returns `False` if any validation fails or `value` is `None`, `True` otherwise. 370 Error description message(s) can be retrieved with `getMessages()` and cleared with `clearMessages()`. 371 372 Args: 373 `key` is the attribute name; 374 `value` is what to validate; 375 `attrib_data` is the lookup table data for the given key (eg. `TPSDK_ATTRIBS_INFO[key]` ); 376 `sdk_v` is the TP SDK version being used (for validation). 377 `path` is just extra information to print before the key name in warning messages (to show where attribute is in the tree). 378 """ 379 global g_seen_ids 380 keypath = _keyPath(path, key) 381 if value is None: 382 if attrib_data.get('r'): 383 _addMessage(f"WARNING: Missing required attribute '{keypath}'.") 384 return False 385 if not isinstance(value, (exp_typ := attrib_data.get('t', str))): 386 _addMessage(f"WARNING: Wrong data type for attribute '{keypath}'. Expected {exp_typ} but got {type(value)}") 387 return False 388 if sdk_v < (min_sdk := attrib_data.get('v', sdk_v)): 389 _addMessage(f"WARNING: Wrong SDK version for attribute '{keypath}'. Minimum is v{min_sdk} but using v{sdk_v}") 390 return False 391 if (choices := attrib_data.get('c')) and value not in choices: 392 _addMessage(f"WARNING: Value error for attribute '{keypath}'. Got '{value}' but expected one of {choices}") 393 return False 394 if key == "id": 395 if not value in _seenIds(): 396 _addSeenId(value, keypath) 397 else: 398 _addMessage(f"WARNING: The ID '{value}' in '{keypath}' is not unique. It was previously seen in '{g_seen_ids.get(value)}'") 399 return False 400 return True 401 402def _validateDefinitionDict(d:dict, table:dict, sdk_v:int, path:str=""): 403 # iterate over existing attributes to validate them 404 for k, v in d.items(): 405 adata = table.get(k) 406 keypath = _keyPath(path, k) 407 if not adata: 408 _addMessage(f"WARNING: Attribute '{keypath}' is unknown.") 409 continue 410 if not validateAttribValue(k, v, adata, sdk_v, path): 411 continue 412 # print(k, v, type(v)) 413 if isinstance(v, list) and (ltable := adata.get('l')): 414 _validateDefinitionArray(v, ltable, sdk_v, keypath) 415 # iterate over table entries to check if all required attribs are present 416 for k, data in table.items(): 417 if data.get('r') and k not in d.keys(): 418 _addMessage(f"WARNING: Missing required attribute '{_keyPath(path, k)}'.") 419 420def _validateDefinitionArray(a:list, table:dict, sdk_v:int, path:str=""): 421 i = 0 422 for item in a: 423 if isinstance(item, dict): 424 _validateDefinitionDict(item, table, sdk_v, f"{path}[{i:d}]") 425 else: 426 _addMessage(f"WARNING: Unable to handle array member '{item}' in '{path}'.") 427 i += 1 428 429 430def validateDefinitionObject(data:dict): 431 """ 432 Validates a TP plugin definition structure from a Python `dict` object. 433 `data` is a de-serialized entry.tp JSON object (eg. `json.load('entry.tp')`) 434 Returns `True` if no problems were found, `False` otherwise. 435 Use `getMessages()` to check for any validation warnings which may be generated. 436 """ 437 _clearSeenIds() 438 clearMessages() 439 sdk_v = data.get('sdk', TPSDK_DEFAULT_VERSION) 440 _validateDefinitionDict(data, TPSDK_ATTRIBS_ROOT, sdk_v) 441 return len(g_messages) == 0 442 443def validateDefinitionString(data: dict): 444 """ 445 Validates a TP plugin definition structure from JSON string. 446 `data` is an entry.tp JSON string 447 Returns `True` if no problems were found, `False` otherwise. 448 Use `getMessages()` to check for any validation warnings which may be generated. 449 """ 450 return validateDefinitionObject(data) 451 452def validateDefinitionFile(file:Union[str, TextIO]): 453 """ 454 Validates a TP plugin definition structure from JSON file. 455 `file` is a valid system path to an entry.tp JSON file *or* an already-opened file handle (eg. sys.stdin). 456 Returns `True` if no problems were found, `False` otherwise. 457 Use `getMessages()` to check for any validation warnings which may be generated. 458 """ 459 fh = file 460 if isinstance(fh, str): 461 fh = open(file, 'r') 462 ret = validateDefinitionObject(json.load(fh)) 463 if fh != file: 464 fh.close() 465 return ret 466 467 468## CLI handlers 469 470def _generateDefinition(script, output_path, indent, skip_invalid:bool=False): 471 input_name = "input stream" 472 if isinstance(script, str): 473 if len(script.split(".")) < 2: 474 script = script + ".py" 475 input_name = script 476 indent = None if indent is None or int(indent) < 0 else indent 477 478 _printToErr(f"Generating plugin definition JSON from '{input_name}'...\n") 479 entry = generateDefinitionFromScript(script, skip_invalid) 480 entry_str = json.dumps(entry, indent=indent) + "\n" 481 valid = True 482 if (messages := getMessages()): 483 valid = False 484 _printMessages(messages) 485 _printToErr("") 486 # output 487 if output_path: 488 # write it to a file 489 with open(output_path, "w") as entry_file: 490 entry_file.write(entry_str) 491 _printToErr(f"Saved generated JSON to '{output_path}'\n") 492 else: 493 # send to stdout 494 print(entry_str) 495 _printToErr(f"Finished generating plugin definition JSON from '{input_name}'.\n") 496 return entry_str, valid 497 498 499def _validateDefinition(entry, as_str=False): 500 name = entry if isinstance(entry, str) and not as_str else "input stream" 501 _printToErr(f"Validating '{name}', any errors or warnings will be printed below...\n") 502 if as_str: 503 res = validateDefinitionString(entry) 504 else: 505 res = validateDefinitionFile(entry) 506 if res: 507 _printToErr("No problems found!") 508 else: 509 _printMessages(getMessages()) 510 _printToErr(f"\nFinished validating '{name}'.\n") 511 return res 512 513def generatePythonStruct(entry, name): 514 _printToErr("Generating Python struct from entry json...\n") 515 try: 516 tp_to_py = TpToPy(entry) 517 tp_to_py.writetoFile(name) 518 _printToErr(f"Saved generated Python struct to '{name}'\n") 519 return True 520 except Exception as e: 521 _printToErr(f"Error: {e}") 522 return False 523 524def main(sdk_args=None): 525 from argparse import ArgumentParser 526 527 parser = ArgumentParser(epilog="This script exits with status code -1 (error) if generation or validation produces warning messages about malformed data. " 528 "All progress and warning messages are printed to stderr stream.") 529 parser.add_argument("-g", "--generate", action='store_true', 530 help="Generate a definition file from plugin script data. This is the default action.") 531 parser.add_argument("-v", "--validate", action='store_true', 532 help="Validate a definition JSON file (entry.tp). If given with `generate` then will validate the generated JSON output.") 533 parser.add_argument("target", metavar="target", nargs="?", default="", 534 help="Either a plugin script for `generate` or an entry.tp file for `validate`. Paths are relative to current working directory. " 535 "Defaults to './TPPEntry.py' and './entry.tp' respectively. Use 'stdin' (or '-') to read from input stream instead. " 536 "Another usage is if you pass in a normal entry.tp without `validate` argument, It can generate " 537 "Python version of entry.tp struct. It will be saved in `TPPEntry.py` if `output` is not given.") 538 gen_grp = parser.add_argument_group("Generator arguments") 539 gen_grp.add_argument("-o", metavar="<file>", 540 help="Output file for `generate` action. Default will be a file named 'entry.tp' in the same folder as the input script. " 541 "Paths are relative to current working directory. Use 'stdout' (or '-') to print the output to the console/stream instead.") 542 gen_grp.add_argument("-s", "--skip-invalid", action='store_true', dest="skip_invalid", default=False, 543 help="Skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.") 544 gen_grp.add_argument("-i", "--indent", metavar="<n>", type=int, default=2, 545 help="Indent level (spaces) for generated JSON. Use 0 for only newlines, or -1 for the most compact representation. Default is %(default)s spaces.") 546 gen_grp.add_argument("--noconfirm", action='store_true', default=False, 547 help="When generating python struct from entry.tp, you can pass this arg to bypass confirm if you want to contiune if any error is given for vaildating entry.tp before generate python struct") 548 opts = parser.parse_args(sdk_args) 549 del parser 550 551 t = _normPath(opts.target or "TPPEntry.py") 552 if opts.target.endswith(".tp") and not opts.validate: 553 valid = _validateDefinition(t) 554 # Incase if file is invaild they will have choice to either contiune. but --noconfirm can override this. 555 # so that way if they use on github action it can still contiune if they wish. 556 if not valid and not opts.noconfirm: input("Found errors. Press Enter to build or Ctrl+C to exit...") 557 successful = generatePythonStruct(t, opts.o or "TPPEntry.py") 558 if not successful: _printToErr("Failed to generate exiting...") 559 560 return successful 561 562 # default action 563 opts.generate = opts.generate or not opts.validate 564 565 _printToErr("") 566 567 if opts.target in ("-","stdin"): 568 opts.target = sys.stdin 569 570 valid = True 571 entry_str = "" 572 if opts.generate: 573 opts.target = _normPath(opts.target or "TPPEntry.py") 574 output_path = None 575 if opts.o: 576 if opts.o not in ("-","stdout"): 577 output_path = opts.o 578 else: 579 out_dir = os.getcwd() if hasattr(opts.target, "read") else os.path.dirname(opts.target) 580 output_path = os.path.join(out_dir, "entry.tp") 581 sys.path.append(os.path.dirname(os.path.realpath(opts.target))) 582 entry_str, valid = _generateDefinition(opts.target, output_path, opts.indent, opts.skip_invalid) 583 if opts.validate and output_path: 584 opts.target = output_path 585 586 if opts.validate: 587 if entry_str: 588 valid = _validateDefinition(entry_str, True) 589 elif opts.target.endswith(".py"): # checks if is python file if It is then It will vaildate the python file by converting it to json first 590 valid = _validateDefinition(generateDefinitionFromScript(opts.target), as_str=True) # little hacky lol 591 else: 592 opts.target = _normPath(opts.target or "entry.tp") 593 valid = _validateDefinition(opts.target) 594 595 return 0 if valid else -1 596 597 598if __name__ == "__main__": 599 sys.exit(main())
128def getMessages(): 129 """ Gets a list of messages which may have been produced during generation/validation. 130 """ 131 return g_messages
Gets a list of messages which may have been produced during generation/validation.
133def clearMessages(): 134 """ Clears the list of validation warning messages. 135 Do this before invoking `validateAttribValue()` directly (outside of the other functions provided here). 136 """ 137 global g_messages 138 g_messages.clear()
Clears the list of validation warning messages.
Do this before invoking validateAttribValue()
directly (outside of the other functions provided here).
239def generateDefinitionFromScript(script:Union[str, TextIO], skip_invalid:bool=False): 240 """ 241 Returns an "entry.tp" Python `dict` which is suitable for direct conversion to JSON format. 242 243 `script` should be a valid python script, either a file path (ending in .py), string, or open file handle (like stdin). 244 The script should contain "SDK declaration variables" like `TP_PLUGIN_INFO`, `TP_PLUGIN_SETTINGS`, etc. 245 246 Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output). 247 Default behavior is to only warn about them. 248 249 Note that the script is interpreted (executed), so any actual "business" logic (like connecting to TP) should be in "__main__". 250 Also note that when using input from a file handle or string, the script's "__file__" attribute is set to the current working 251 directory and the file name "tp_plugin.py". 252 253 May raise an `ImportError` if the plugin script could not be loaded or is missing required variables. 254 Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation). 255 """ 256 script_str = "" 257 if hasattr(script, "read"): 258 script_str = script.read() 259 elif not script.endswith(".py"): 260 script_str = script 261 262 try: 263 if script_str: 264 # load from a string 265 spec = importlib.util.spec_from_loader("plugin", loader=None) 266 plugin = importlib.util.module_from_spec(spec) 267 setattr(plugin, "__file__", os.path.join(os.getcwd(), "tp_plugin.py")) 268 exec(script_str, plugin.__dict__) 269 else: 270 # load directly from a file path 271 spec = importlib.util.spec_from_file_location("plugin", script) 272 plugin = importlib.util.module_from_spec(spec) 273 spec.loader.exec_module(plugin) 274 # print(plugin.TP_PLUGIN_INFO) 275 except Exception as e: 276 input_name = "input stream" if script_str else script 277 raise ImportError(f"ERROR while trying to import plugin code from '{input_name}': {repr(e)}") 278 return generateDefinitionFromModule(plugin, skip_invalid)
Returns an "entry.tp" Python dict
which is suitable for direct conversion to JSON format.
script
should be a valid python script, either a file path (ending in .py), string, or open file handle (like stdin).
The script should contain "SDK declaration variables" like TP_PLUGIN_INFO
, TP_PLUGIN_SETTINGS
, etc.
Setting skip_invalid
to True
will skip attributes with invalid values (they will not be included in generated output).
Default behavior is to only warn about them.
Note that the script is interpreted (executed), so any actual "business" logic (like connecting to TP) should be in "__main__". Also note that when using input from a file handle or string, the script's "__file__" attribute is set to the current working directory and the file name "tp_plugin.py".
May raise an ImportError
if the plugin script could not be loaded or is missing required variables.
Use getMessages()
to check for any warnings/etc which may be generated (eg. from attribute validation).
281def generateDefinitionFromModule(plugin:ModuleType, skip_invalid:bool=False): 282 """ 283 Returns an "entry.tp" Python `dict`, which is suitable for direct conversion to JSON format. 284 `plugin` should be a loaded Python "module" which contains "SDK declaration variables" like 285 `TP_PLUGIN_INFO`, `TP_PLUGIN_SETTINGS`, etc. From within a plugin script this could be called like: 286 `generateDefinitionFromModule(sys.modules[__name__])`. 287 288 Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output). 289 Default behavior is to only warn about them. 290 291 May raise an `ImportError` if the plugin script is missing required variables `TP_PLUGIN_INFO` and `TP_PLUGIN_CATEGORIES`. 292 Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation). 293 """ 294 # Load the "standard SDK declaration variables" from plugin script into local scope 295 # INFO and CATEGORY are required, rest are optional. 296 if not (info := getattr(plugin, "TP_PLUGIN_INFO", {})): 297 raise ImportError(f"ERROR: Could not import required TP_PLUGIN_INFO variable from plugin source.") 298 if not (cats := getattr(plugin, "TP_PLUGIN_CATEGORIES", {})): 299 raise ImportError(f"ERROR: Could not import required TP_PLUGIN_CATEGORIES variable from plugin source.") 300 return generateDefinitionFromDeclaration( 301 info, cats, 302 settings = getattr(plugin, "TP_PLUGIN_SETTINGS", {}), 303 actions = getattr(plugin, "TP_PLUGIN_ACTIONS", {}), 304 states = getattr(plugin, "TP_PLUGIN_STATES", {}), 305 events = getattr(plugin, "TP_PLUGIN_EVENTS", {}), 306 connectors = getattr(plugin, "TP_PLUGIN_CONNECTORS", {}), 307 skip_invalid = skip_invalid 308 )
Returns an "entry.tp" Python dict
, which is suitable for direct conversion to JSON format.
plugin
should be a loaded Python "module" which contains "SDK declaration variables" like
TP_PLUGIN_INFO
, TP_PLUGIN_SETTINGS
, etc. From within a plugin script this could be called like:
generateDefinitionFromModule(sys.modules[__name__])
.
Setting skip_invalid
to True
will skip attributes with invalid values (they will not be included in generated output).
Default behavior is to only warn about them.
May raise an ImportError
if the plugin script is missing required variables TP_PLUGIN_INFO
and TP_PLUGIN_CATEGORIES
.
Use getMessages()
to check for any warnings/etc which may be generated (eg. from attribute validation).
311def generateDefinitionFromDeclaration(info:dict, categories:dict, skip_invalid:bool=False, **kwargs): 312 """ 313 Returns an "entry.tp" Python `dict` which is suitable for direct conversion to JSON format. 314 Arguments should contain SDK declaration dict values, for example as specified for `TP_PLUGIN_INFO`, etc. 315 316 The `info` and `category` values are required, the rest are optional. 317 318 Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output). 319 Default behavior is to warn about them but still include them in the output. 320 321 `**kwargs` can be one or more of: 322 - settings:dict = {}, 323 - actions:dict = {}, 324 - states:dict = {}, 325 - events:dict = {}, 326 - connectors:dict = {} 327 328 Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation). 329 """ 330 _clearSeenIds() 331 clearMessages() 332 settings = kwargs.get('settings', {}) 333 actions = kwargs.get('actions', {}) 334 states = kwargs.get('states', {}) 335 events = kwargs.get('events', {}) 336 connectors = kwargs.get('connectors', {}) 337 # print(info, categories, settings, actions, states, events, connectors) 338 339 # Start the root entry.tp object using basic plugin metadata 340 # This will also create an empty `categories` array in the root of the entry. 341 entry = _dictFromItem(info, TPSDK_ATTRIBS_ROOT, TPSDK_DEFAULT_VERSION, "info") 342 343 # Get the target SDK version (was either specified in plugin or is TPSDK_DEFAULT_VERSION) 344 tgt_sdk_v = entry['sdk'] 345 346 # Loop over each plugin category and set up actions, states, events, and connectors. 347 for cat, data in categories.items(): 348 path = f"category[{cat}]" 349 category = _dictFromItem(data, TPSDK_ATTRIBS_CATEGORY, tgt_sdk_v, path, skip_invalid) 350 category['actions'] = _arrayFromDict(actions, TPSDK_ATTRIBS_ACTION, tgt_sdk_v, cat, "actions", skip_invalid) 351 category['states'] = _arrayFromDict(states, TPSDK_ATTRIBS_STATE, tgt_sdk_v, cat, "states", skip_invalid) 352 category['events'] = _arrayFromDict(events, TPSDK_ATTRIBS_EVENT, tgt_sdk_v, cat, "events", skip_invalid) 353 if tgt_sdk_v >= 4: 354 category['connectors'] = _arrayFromDict(connectors, TPSDK_ATTRIBS_CONNECTOR, tgt_sdk_v, cat, "connectors", skip_invalid) 355 # add the category to entry's categories array 356 entry['categories'].append(category) 357 358 # Add Settings to root 359 if tgt_sdk_v >= 3: 360 entry['settings'].extend(_arrayFromDict(settings, TPSDK_ATTRIBS_SETTINGS, tgt_sdk_v, path = "settings", skip_invalid = skip_invalid)) 361 362 return entry
Returns an "entry.tp" Python dict
which is suitable for direct conversion to JSON format.
Arguments should contain SDK declaration dict values, for example as specified for TP_PLUGIN_INFO
, etc.
The info
and category
values are required, the rest are optional.
Setting skip_invalid
to True
will skip attributes with invalid values (they will not be included in generated output).
Default behavior is to warn about them but still include them in the output.
**kwargs
can be one or more of:
- settings:dict = {},
- actions:dict = {},
- states:dict = {},
- events:dict = {},
- connectors:dict = {}
Use getMessages()
to check for any warnings/etc which may be generated (eg. from attribute validation).
367def validateAttribValue(key:str, value, attrib_data:dict, sdk_v:int, path:str=""): 368 """ 369 Validates one attribute's value based on provided lookup table and target SDK version. 370 Returns `False` if any validation fails or `value` is `None`, `True` otherwise. 371 Error description message(s) can be retrieved with `getMessages()` and cleared with `clearMessages()`. 372 373 Args: 374 `key` is the attribute name; 375 `value` is what to validate; 376 `attrib_data` is the lookup table data for the given key (eg. `TPSDK_ATTRIBS_INFO[key]` ); 377 `sdk_v` is the TP SDK version being used (for validation). 378 `path` is just extra information to print before the key name in warning messages (to show where attribute is in the tree). 379 """ 380 global g_seen_ids 381 keypath = _keyPath(path, key) 382 if value is None: 383 if attrib_data.get('r'): 384 _addMessage(f"WARNING: Missing required attribute '{keypath}'.") 385 return False 386 if not isinstance(value, (exp_typ := attrib_data.get('t', str))): 387 _addMessage(f"WARNING: Wrong data type for attribute '{keypath}'. Expected {exp_typ} but got {type(value)}") 388 return False 389 if sdk_v < (min_sdk := attrib_data.get('v', sdk_v)): 390 _addMessage(f"WARNING: Wrong SDK version for attribute '{keypath}'. Minimum is v{min_sdk} but using v{sdk_v}") 391 return False 392 if (choices := attrib_data.get('c')) and value not in choices: 393 _addMessage(f"WARNING: Value error for attribute '{keypath}'. Got '{value}' but expected one of {choices}") 394 return False 395 if key == "id": 396 if not value in _seenIds(): 397 _addSeenId(value, keypath) 398 else: 399 _addMessage(f"WARNING: The ID '{value}' in '{keypath}' is not unique. It was previously seen in '{g_seen_ids.get(value)}'") 400 return False 401 return True
Validates one attribute's value based on provided lookup table and target SDK version.
Returns False
if any validation fails or value
is None
, True
otherwise.
Error description message(s) can be retrieved with getMessages()
and cleared with clearMessages()
.
Args
key
is the attribute name;value
is what to validate;attrib_data
is the lookup table data for the given key (eg.TPSDK_ATTRIBS_INFO[key]
);sdk_v
is the TP SDK version being used (for validation).path
is just extra information to print before the key name in warning messages (to show where attribute is in the tree).
431def validateDefinitionObject(data:dict): 432 """ 433 Validates a TP plugin definition structure from a Python `dict` object. 434 `data` is a de-serialized entry.tp JSON object (eg. `json.load('entry.tp')`) 435 Returns `True` if no problems were found, `False` otherwise. 436 Use `getMessages()` to check for any validation warnings which may be generated. 437 """ 438 _clearSeenIds() 439 clearMessages() 440 sdk_v = data.get('sdk', TPSDK_DEFAULT_VERSION) 441 _validateDefinitionDict(data, TPSDK_ATTRIBS_ROOT, sdk_v) 442 return len(g_messages) == 0
Validates a TP plugin definition structure from a Python dict
object.
data
is a de-serialized entry.tp JSON object (eg. json.load('entry.tp')
)
Returns True
if no problems were found, False
otherwise.
Use getMessages()
to check for any validation warnings which may be generated.
444def validateDefinitionString(data: dict): 445 """ 446 Validates a TP plugin definition structure from JSON string. 447 `data` is an entry.tp JSON string 448 Returns `True` if no problems were found, `False` otherwise. 449 Use `getMessages()` to check for any validation warnings which may be generated. 450 """ 451 return validateDefinitionObject(data)
Validates a TP plugin definition structure from JSON string.
data
is an entry.tp JSON string
Returns True
if no problems were found, False
otherwise.
Use getMessages()
to check for any validation warnings which may be generated.
453def validateDefinitionFile(file:Union[str, TextIO]): 454 """ 455 Validates a TP plugin definition structure from JSON file. 456 `file` is a valid system path to an entry.tp JSON file *or* an already-opened file handle (eg. sys.stdin). 457 Returns `True` if no problems were found, `False` otherwise. 458 Use `getMessages()` to check for any validation warnings which may be generated. 459 """ 460 fh = file 461 if isinstance(fh, str): 462 fh = open(file, 'r') 463 ret = validateDefinitionObject(json.load(fh)) 464 if fh != file: 465 fh.close() 466 return ret
Validates a TP plugin definition structure from JSON file.
file
is a valid system path to an entry.tp JSON file or an already-opened file handle (eg. sys.stdin).
Returns True
if no problems were found, False
otherwise.
Use getMessages()
to check for any validation warnings which may be generated.
514def generatePythonStruct(entry, name): 515 _printToErr("Generating Python struct from entry json...\n") 516 try: 517 tp_to_py = TpToPy(entry) 518 tp_to_py.writetoFile(name) 519 _printToErr(f"Saved generated Python struct to '{name}'\n") 520 return True 521 except Exception as e: 522 _printToErr(f"Error: {e}") 523 return False
525def main(sdk_args=None): 526 from argparse import ArgumentParser 527 528 parser = ArgumentParser(epilog="This script exits with status code -1 (error) if generation or validation produces warning messages about malformed data. " 529 "All progress and warning messages are printed to stderr stream.") 530 parser.add_argument("-g", "--generate", action='store_true', 531 help="Generate a definition file from plugin script data. This is the default action.") 532 parser.add_argument("-v", "--validate", action='store_true', 533 help="Validate a definition JSON file (entry.tp). If given with `generate` then will validate the generated JSON output.") 534 parser.add_argument("target", metavar="target", nargs="?", default="", 535 help="Either a plugin script for `generate` or an entry.tp file for `validate`. Paths are relative to current working directory. " 536 "Defaults to './TPPEntry.py' and './entry.tp' respectively. Use 'stdin' (or '-') to read from input stream instead. " 537 "Another usage is if you pass in a normal entry.tp without `validate` argument, It can generate " 538 "Python version of entry.tp struct. It will be saved in `TPPEntry.py` if `output` is not given.") 539 gen_grp = parser.add_argument_group("Generator arguments") 540 gen_grp.add_argument("-o", metavar="<file>", 541 help="Output file for `generate` action. Default will be a file named 'entry.tp' in the same folder as the input script. " 542 "Paths are relative to current working directory. Use 'stdout' (or '-') to print the output to the console/stream instead.") 543 gen_grp.add_argument("-s", "--skip-invalid", action='store_true', dest="skip_invalid", default=False, 544 help="Skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.") 545 gen_grp.add_argument("-i", "--indent", metavar="<n>", type=int, default=2, 546 help="Indent level (spaces) for generated JSON. Use 0 for only newlines, or -1 for the most compact representation. Default is %(default)s spaces.") 547 gen_grp.add_argument("--noconfirm", action='store_true', default=False, 548 help="When generating python struct from entry.tp, you can pass this arg to bypass confirm if you want to contiune if any error is given for vaildating entry.tp before generate python struct") 549 opts = parser.parse_args(sdk_args) 550 del parser 551 552 t = _normPath(opts.target or "TPPEntry.py") 553 if opts.target.endswith(".tp") and not opts.validate: 554 valid = _validateDefinition(t) 555 # Incase if file is invaild they will have choice to either contiune. but --noconfirm can override this. 556 # so that way if they use on github action it can still contiune if they wish. 557 if not valid and not opts.noconfirm: input("Found errors. Press Enter to build or Ctrl+C to exit...") 558 successful = generatePythonStruct(t, opts.o or "TPPEntry.py") 559 if not successful: _printToErr("Failed to generate exiting...") 560 561 return successful 562 563 # default action 564 opts.generate = opts.generate or not opts.validate 565 566 _printToErr("") 567 568 if opts.target in ("-","stdin"): 569 opts.target = sys.stdin 570 571 valid = True 572 entry_str = "" 573 if opts.generate: 574 opts.target = _normPath(opts.target or "TPPEntry.py") 575 output_path = None 576 if opts.o: 577 if opts.o not in ("-","stdout"): 578 output_path = opts.o 579 else: 580 out_dir = os.getcwd() if hasattr(opts.target, "read") else os.path.dirname(opts.target) 581 output_path = os.path.join(out_dir, "entry.tp") 582 sys.path.append(os.path.dirname(os.path.realpath(opts.target))) 583 entry_str, valid = _generateDefinition(opts.target, output_path, opts.indent, opts.skip_invalid) 584 if opts.validate and output_path: 585 opts.target = output_path 586 587 if opts.validate: 588 if entry_str: 589 valid = _validateDefinition(entry_str, True) 590 elif opts.target.endswith(".py"): # checks if is python file if It is then It will vaildate the python file by converting it to json first 591 valid = _validateDefinition(generateDefinitionFromScript(opts.target), as_str=True) # little hacky lol 592 else: 593 opts.target = _normPath(opts.target or "entry.tp") 594 valid = _validateDefinition(opts.target) 595 596 return 0 if valid else -1