TouchPortalAPI v1.7.8

TouchPortalAPI.sdk_tools

Touch Portal Python SDK Tools

Features

This SDK tool provides features for generating and validating Touch Portal Plugin Description files, which are in JSON format and typically named "entry.tp" (as described in the TP API docs).

Generation is done based on dictionary structures defined in plugin scripts. These structures closely follow the format of the description file JSON, but are further expanded to make them also useful within the plugin scripts themselves. This avoids almost all the need for duplication of things like unique IDs, default values, settings names, and so on, since those details need to be available in both the definition JSON and in the script which will be using those definitions to communicate with Touch Portal.

Validation is performed on the generated descriptions, and can also be run against pre-existing definition files as a sort of "lint" utility. Currently the following things are checked:

  • All required attributes are present.
  • All attribute values are of the supported data type(s).
  • No unknown attributes are present.
  • Attributes are valid for the TP SDK version being used.
  • Attribute values fall within allowed list of values (if relevant, eg. Action.type).
  • All ID strings are unique within the plugin (eg. for States, Actions, etc).

This tool can be used as a command-line utility, or imported as a module to use the available functions programmatically. The script command is tppsdk when the TouchPortalAPI is installed (via pip or setup), or sdk_tool.py when run directly from this source. To use within a script, simply import in the usual way as needed. For example:

from TouchPortalAPI.sdk_tools import generateDefinitionFromScript

All generation/validation routines are based on TP API specification data tables defined in TouchPortalAPI.sdk_spec.

A comprehensive example plugin which utilizes the SDK features is included with the TouchPortalAPI project. The JSON definition "entry.tp" file for that example could be generated with a simple command from within a local folder containing the example script:

tppsdk plugin_example.py

Functions

Command-line Usage

The script command is tppsdk when the TouchPortalAPI is installed (via pip or setup), or sdk_tool.py when run directly from this source.

<script_command> [-h] [-g] [-v] [-o <file>] [-s] [-i <n>] [target]

positional arguments:
  target                Either a plugin script for `generate` or an entry.tp file for `validate`. Paths are relative to current working
                        directory. Defaults to './TPPEntry.py' and './entry.tp' respectively. Use 'stdin' (or '-') to read from input
                        stream instead. Another usage is if you pass in a normal entry.tp without `validate` argument, It can generate
                        Python version of entry.tp struct. It will be saved in `TPPEntry.py` if `output` is not given.

optional arguments:
  -h, --help            show this help message and exit
  -g, --generate        Generate a definition file from plugin script data. This is the default action.
  -v, --validate        Validate a definition JSON file (entry.tp). If given with `generate` then will validate the generated JSON output.

generator arguments:
  -o <file>             Output file for `generate` action. Default will be a file named 'entry.tp' in the same folder as the input script.
                        Paths are relative to current working directory. Use 'stdout' (or '-') to print the output to the console/stream instead.
  -s, --skip-invalid    Skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.
  -i <n>, --indent <n>  Indent level (spaces) for generated JSON. Use 0 for only newlines, or -1 for the most compact representation. Default is 2 spaces.
  --noconfirm           When generating python struct from entry.tp, you can pass this arg to bypass confirm if you want to contiune if any error is given for vaildating entry.tp before generate python struct.
This script exits with status code -1 (error) if generation or validation produces warning messages about malformed data.
All progress and warning messages are printed to stderr stream.

TODO/Ideas

  • Document the default attribute values from sdk_specs table.
  • Dynamic default values, eg. for action prefix or category id/name (see notes in sdk_spec tables).
  • Dynamic ID generation and write-back to plugin at runtime.
  • Allow plugin author to set their own defaults.
  1#!/usr/bin/env python3
  2"""
  3# Touch Portal Python SDK Tools
  4
  5## Features
  6
  7This SDK tool provides features for generating and validating Touch Portal Plugin Description files,
  8which are in JSON format and typically named "entry.tp" (as described in the [TP API docs](https://www.touch-portal.com/api/)).
  9
 10**Generation** is done based on dictionary structures defined in plugin scripts. These structures closely follow
 11the format of the description file JSON, but are further expanded to make them also useful within the plugin
 12scripts themselves. This avoids almost all the need for duplication of things like unique IDs, default values,
 13settings names, and so on, since those details need to be available in both the definition JSON and in the
 14script which will be using those definitions to communicate with Touch Portal.
 15
 16**Validation** is performed on the generated descriptions, and can also be run against pre-existing definition files
 17as a sort of "lint" utility. Currently the following things are checked:
 18- All required attributes are present.
 19- All attribute values are of the supported data type(s).
 20- No unknown attributes are present.
 21- Attributes are valid for the TP SDK version being used.
 22- Attribute values fall within allowed list of values (if relevant, eg. `Action.type`).
 23- All ID strings are unique within the plugin (eg. for States, Actions, etc).
 24
 25This tool can be used as a command-line utility, or imported as a module to use the available functions programmatically.
 26The script command is `tppsdk` when the TouchPortalAPI is installed (via pip or setup), or `sdk_tool.py` when run directly
 27from this source. To use within a script, simply import in the usual way as needed. For example:
 28
 29```py
 30from TouchPortalAPI.sdk_tools import generateDefinitionFromScript
 31```
 32
 33All generation/validation routines are based on TP API specification data tables defined in `TouchPortalAPI.sdk_spec`.
 34
 35A comprehensive example plugin which utilizes the SDK features is included with the TouchPortalAPI project.
 36The JSON definition "entry.tp" file for that example could be generated with a simple command from within a local
 37folder containing the example script:
 38
 39```
 40tppsdk plugin_example.py
 41```
 42
 43
 44## Functions
 45* Generates an `entry.tp` definition file for a Touch Portal plugin based
 46on variables specified in the plugin source code (`generateDefinitionFromScript()`),
 47from a loaded module (`generateDefinitionFromModule()`) or specified by value (`generateDefinitionFromDeclaration()`).
 48* Validate an entire plugin definition (`entry.tp`) file (`validateDefinitionFile()`),
 49string (`validateDefinitionString()`), or object (`validateDefinitionObject()`).
 50* Validate an `entry.tp` attribute value against the minimum
 51SDK version, value type, value content, etc. (`validateAttribValue()`).
 52
 53
 54## Command-line Usage
 55The script command is `tppsdk` when the TouchPortalAPI is installed (via pip or setup), or `sdk_tool.py` when run directly from this source.
 56```
 57<script_command> [-h] [-g] [-v] [-o <file>] [-s] [-i <n>] [target]
 58
 59positional arguments:
 60  target                Either a plugin script for `generate` or an entry.tp file for `validate`. Paths are relative to current working
 61                        directory. Defaults to './TPPEntry.py' and './entry.tp' respectively. Use 'stdin' (or '-') to read from input
 62                        stream instead. Another usage is if you pass in a normal entry.tp without `validate` argument, It can generate
 63                        Python version of entry.tp struct. It will be saved in `TPPEntry.py` if `output` is not given.
 64
 65optional arguments:
 66  -h, --help            show this help message and exit
 67  -g, --generate        Generate a definition file from plugin script data. This is the default action.
 68  -v, --validate        Validate a definition JSON file (entry.tp). If given with `generate` then will validate the generated JSON output.
 69
 70generator arguments:
 71  -o <file>             Output file for `generate` action. Default will be a file named 'entry.tp' in the same folder as the input script.
 72                        Paths are relative to current working directory. Use 'stdout' (or '-') to print the output to the console/stream instead.
 73  -s, --skip-invalid    Skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.
 74  -i <n>, --indent <n>  Indent level (spaces) for generated JSON. Use 0 for only newlines, or -1 for the most compact representation. Default is 2 spaces.
 75  --noconfirm           When generating python struct from entry.tp, you can pass this arg to bypass confirm if you want to contiune if any error is given for vaildating entry.tp before generate python struct.
 76This script exits with status code -1 (error) if generation or validation produces warning messages about malformed data.
 77All progress and warning messages are printed to stderr stream.
 78```
 79
 80## TODO/Ideas
 81
 82* Document the default attribute values from sdk_specs table.
 83* Dynamic default values, eg. for action prefix or category id/name (see notes in sdk_spec tables).
 84* Dynamic ID generation and write-back to plugin at runtime.
 85* Allow plugin author to set their own defaults.
 86"""
 87
 88__copyright__ = """
 89This file is part of the TouchPortal-API project.
 90Copyright TouchPortal-API Developers
 91Copyright (c) 2021 Maxim Paperno
 92All rights reserved.
 93
 94This program is free software: you can redistribute it and/or modify
 95it under the terms of the GNU General Public License as published by
 96the Free Software Foundation, either version 3 of the License, or
 97(at your option) any later version.
 98
 99This program is distributed in the hope that it will be useful,
100but WITHOUT ANY WARRANTY; without even the implied warranty of
101MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
102GNU General Public License for more details.
103
104You should have received a copy of the GNU General Public License
105along with this program.  If not, see <https://www.gnu.org/licenses/>.
106"""
107
108import sys
109import os.path
110import importlib.util
111import json
112from types import ModuleType
113from typing import (Union, TextIO)
114from re import compile as re_compile
115
116sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
117from sdk_spec import *
118from TpToPy import TpToPy
119
120## globals
121g_messages = []  # validation reporting
122g_seen_ids = {}  # for validating unique IDs
123
124
125## Utils
126
127def getMessages():
128    """ Gets a list of messages which may have been produced during generation/validation.
129    """
130    return g_messages
131
132def clearMessages():
133    """ Clears the list of validation warning messages.
134    Do this before invoking `validateAttribValue()` directly (outside of the other functions provided here).
135    """
136    global g_messages
137    g_messages.clear()
138
139def _printMessages(messages:list):
140    for msg in messages:
141        _printToErr(msg)
142
143def _addMessage(msg):
144    global g_messages
145    g_messages.append(msg)
146
147
148def _seenIds():
149    global g_seen_ids
150    return g_seen_ids.keys()
151
152def _addSeenId(id, path):
153    global g_seen_ids
154    g_seen_ids[id] = path
155
156def _clearSeenIds():
157    global g_seen_ids
158    g_seen_ids.clear()
159
160
161def _printToErr(msg):
162    sys.stderr.write(msg + "\n")
163
164def _normPath(path):
165    if not isinstance(path, str):
166        return path
167    return os.path.normpath(os.path.join(os.getcwd(), path))
168
169def _keyPath(path, key):
170    return ":".join(filter(None, [path, key]))
171
172## Generator functions
173
174def _dictFromItem(item:dict, table:dict, sdk_v:int, path:str="", skip_invalid:bool=False):
175    ret = {}
176    if not isinstance(item, dict):
177        return ret
178    for k, data in table.items():
179        # try get explicit value from item
180        #if not data.get("doc"): continue
181        if (v := item.get(k)) is None:
182            # try get default value
183            v = data.get('d')
184        # check if there is nested data, eg. in an Action
185        if isinstance(v, dict) and data.get('t') is list:
186            v = _arrayFromDict(v, data.get('l', {}), sdk_v, path=_keyPath(path, k), skip_invalid=skip_invalid)
187        # check that the value is valid and add it to the dict if it is
188        if validateAttribValue(k, v, data, sdk_v, path) or (not skip_invalid and v != None):
189            ret[k] = v
190            # if this is the "sdk" value from TP_PLUGIN_INFO then reset the
191            # passed `sdk_v` param since it was originally set to TPSDK_DEFAULT_VERSION
192            if k == "sdk":
193                sdk_v = v
194    return ret
195
196
197def _arrayFromDict(d:dict, table:dict, sdk_v:int, category:str=None, path:str="", skip_invalid:bool=False):
198    ret = []
199    if not isinstance(d, dict):
200        return ret
201    for key, item in d.items():
202        if not category or not (cat := item.get('category')) or cat == category:
203            ret.append(_dictFromItem(item, table, sdk_v, f"{path}[{key}]", skip_invalid))
204    if path in ["actions","connectors"]:
205        _replaceFormatTokens(ret)
206    return ret
207
208
209def _replaceFormatTokens(items:list):
210    for d in items:
211        if not isinstance(d, dict) or not 'format' in d.keys() or not 'data' in d.keys():
212            continue
213        data_ids = {}
214        for data in d.get('data'):
215            if (did := data.get('id')):
216                data_ids[did.rsplit(".", 1)[-1]] = did
217        if not data_ids:
218            continue
219        fmt = d.get('format')
220        rx = re_compile(r'\$\[(\w+)\]')
221        begin = 0
222        while (m := rx.search(fmt, begin)):
223            idx = m.group(1)
224            if idx in data_ids.keys():
225                val = data_ids.get(idx)
226            elif idx.isdigit() and (i := int(idx) - 1) >= 0 and i < len(data_ids):
227                val = list(data_ids.values())[i]
228            else:
229                begin = m.end()
230                _addMessage(f"WARNING: Could not find replacement for token '{idx}' in 'format' attribute for element `{d.get('id')}`. The data arry does not contain this name/index.")
231                continue
232            # print(m.span(), val)
233            fmt = fmt[:m.start()] + "{$" + val + "$}" + fmt[m.end():]
234            begin = m.start() + len(val) + 4
235        d['format'] = fmt
236
237
238def generateDefinitionFromScript(script:Union[str, TextIO], skip_invalid:bool=False):
239    """
240    Returns an "entry.tp" Python `dict` which is suitable for direct conversion to JSON format.
241
242    `script` should be a valid python script, either a file path (ending in .py), string, or open file handle (like stdin).
243    The script should contain "SDK declaration variables" like	`TP_PLUGIN_INFO`, `TP_PLUGIN_SETTINGS`, etc.
244
245    Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output).
246    Default behavior is to only warn about them.
247
248    Note that the script is interpreted (executed), so any actual "business" logic (like connecting to TP) should be in "__main__".
249    Also note that when using input from a file handle or string, the script's "__file__" attribute is set to the current working
250    directory and the file name "tp_plugin.py".
251
252    May raise an `ImportError` if the plugin script could not be loaded or is missing required variables.
253    Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation).
254    """
255    script_str = ""
256    if hasattr(script, "read"):
257        script_str = script.read()
258    elif not script.endswith(".py"):
259        script_str = script
260
261    try:
262        if script_str:
263            # load from a string
264            spec = importlib.util.spec_from_loader("plugin", loader=None)
265            plugin = importlib.util.module_from_spec(spec)
266            setattr(plugin, "__file__", os.path.join(os.getcwd(), "tp_plugin.py"))
267            exec(script_str, plugin.__dict__)
268        else:
269            # load directly from a file path
270            spec = importlib.util.spec_from_file_location("plugin", script)
271            plugin = importlib.util.module_from_spec(spec)
272            spec.loader.exec_module(plugin)
273        # print(plugin.TP_PLUGIN_INFO)
274    except Exception as e:
275        input_name = "input stream" if script_str else script
276        raise ImportError(f"ERROR while trying to import plugin code from '{input_name}': {repr(e)}")
277    return generateDefinitionFromModule(plugin, skip_invalid)
278
279
280def generateDefinitionFromModule(plugin:ModuleType, skip_invalid:bool=False):
281    """
282    Returns an "entry.tp" Python `dict`, which is suitable for direct conversion to JSON format.
283    `plugin` should be a loaded Python "module" which contains "SDK declaration variables" like
284    `TP_PLUGIN_INFO`, `TP_PLUGIN_SETTINGS`, etc. From within a plugin script this could be called like:
285    `generateDefinitionFromModule(sys.modules[__name__])`.
286
287    Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output).
288    Default behavior is to only warn about them.
289
290    May raise an `ImportError` if the plugin script is missing required variables `TP_PLUGIN_INFO` and `TP_PLUGIN_CATEGORIES`.
291    Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation).
292    """
293    # Load the "standard SDK declaration variables" from plugin script into local scope
294    # INFO and CATEGORY are required, rest are optional.
295    if not (info := getattr(plugin, "TP_PLUGIN_INFO", {})):
296        raise ImportError(f"ERROR: Could not import required TP_PLUGIN_INFO variable from plugin source.")
297    if not (cats := getattr(plugin, "TP_PLUGIN_CATEGORIES", {})):
298        raise ImportError(f"ERROR: Could not import required TP_PLUGIN_CATEGORIES variable from plugin source.")
299    return generateDefinitionFromDeclaration(
300        info, cats,
301        settings = getattr(plugin, "TP_PLUGIN_SETTINGS", {}),
302        actions = getattr(plugin, "TP_PLUGIN_ACTIONS", {}),
303        states = getattr(plugin, "TP_PLUGIN_STATES", {}),
304        events = getattr(plugin, "TP_PLUGIN_EVENTS", {}),
305        connectors = getattr(plugin, "TP_PLUGIN_CONNECTORS", {}),
306        skip_invalid = skip_invalid
307    )
308
309
310def generateDefinitionFromDeclaration(info:dict, categories:dict, skip_invalid:bool=False, **kwargs):
311    """
312    Returns an "entry.tp" Python `dict` which is suitable for direct conversion to JSON format.
313    Arguments should contain SDK declaration dict values, for example as specified for `TP_PLUGIN_INFO`, etc.
314
315    The `info` and `category` values are required, the rest are optional.
316
317    Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output).
318    Default behavior is to warn about them but still include them in the output.
319
320    `**kwargs` can be one or more of:
321    - settings:dict = {},
322    - actions:dict = {},
323    - states:dict = {},
324    - events:dict = {},
325    - connectors:dict = {}
326
327    Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation).
328    """
329    _clearSeenIds()
330    clearMessages()
331    settings = kwargs.get('settings', {})
332    actions = kwargs.get('actions', {})
333    states = kwargs.get('states', {})
334    events = kwargs.get('events', {})
335    connectors = kwargs.get('connectors', {})
336    # print(info, categories, settings, actions, states, events, connectors)
337
338    # Start the root entry.tp object using basic plugin metadata
339    # This will also create an empty `categories` array in the root of the entry.
340    entry = _dictFromItem(info, TPSDK_ATTRIBS_ROOT, TPSDK_DEFAULT_VERSION, "info")
341
342    # Get the target SDK version (was either specified in plugin or is TPSDK_DEFAULT_VERSION)
343    tgt_sdk_v = entry['sdk']
344
345    # Loop over each plugin category and set up actions, states, events, and connectors.
346    for cat, data in categories.items():
347        path = f"category[{cat}]"
348        category = _dictFromItem(data, TPSDK_ATTRIBS_CATEGORY, tgt_sdk_v, path, skip_invalid)
349        category['actions'] = _arrayFromDict(actions, TPSDK_ATTRIBS_ACTION, tgt_sdk_v, cat, "actions", skip_invalid)
350        category['states'] = _arrayFromDict(states, TPSDK_ATTRIBS_STATE, tgt_sdk_v, cat, "states", skip_invalid)
351        category['events'] = _arrayFromDict(events, TPSDK_ATTRIBS_EVENT, tgt_sdk_v, cat, "events", skip_invalid)
352        if tgt_sdk_v >= 4:
353            category['connectors'] = _arrayFromDict(connectors, TPSDK_ATTRIBS_CONNECTOR, tgt_sdk_v, cat, "connectors", skip_invalid)
354        # add the category to entry's categories array
355        entry['categories'].append(category)
356
357    # Add Settings to root
358    if tgt_sdk_v >= 3:
359        entry['settings'].extend(_arrayFromDict(settings, TPSDK_ATTRIBS_SETTINGS, tgt_sdk_v, path = "settings", skip_invalid = skip_invalid))
360
361    return entry
362
363
364## Validation functions
365
366def validateAttribValue(key:str, value, attrib_data:dict, sdk_v:int, path:str=""):
367    """
368    Validates one attribute's value based on provided lookup table and target SDK version.
369    Returns `False` if any validation fails or `value` is `None`, `True` otherwise.
370    Error description message(s) can be retrieved with `getMessages()` and cleared with `clearMessages()`.
371
372    Args:
373        `key` is the attribute name;
374        `value` is what to validate;
375        `attrib_data` is the lookup table data for the given key (eg. `TPSDK_ATTRIBS_INFO[key]` );
376        `sdk_v` is the TP SDK version being used (for validation).
377        `path` is just extra information to print before the key name in warning messages (to show where attribute is in the tree).
378    """
379    global g_seen_ids
380    keypath = _keyPath(path, key)
381    if value is None:
382        if attrib_data.get('r'):
383            _addMessage(f"WARNING: Missing required attribute '{keypath}'.")
384        return False
385    if not isinstance(value, (exp_typ := attrib_data.get('t', str))):
386        _addMessage(f"WARNING: Wrong data type for attribute '{keypath}'. Expected {exp_typ} but got {type(value)}")
387        return False
388    if sdk_v < (min_sdk := attrib_data.get('v', sdk_v)):
389        _addMessage(f"WARNING: Wrong SDK version for attribute '{keypath}'. Minimum is v{min_sdk} but using v{sdk_v}")
390        return False
391    if (choices := attrib_data.get('c')) and value not in choices:
392        _addMessage(f"WARNING: Value error for attribute '{keypath}'. Got '{value}' but expected one of {choices}")
393        return False
394    if key == "id":
395        if not value in _seenIds():
396            _addSeenId(value, keypath)
397        else:
398            _addMessage(f"WARNING: The ID '{value}' in '{keypath}' is not unique. It was previously seen in '{g_seen_ids.get(value)}'")
399            return False
400    return True
401
402def _validateDefinitionDict(d:dict, table:dict, sdk_v:int, path:str=""):
403    # iterate over existing attributes to validate them
404    for k, v in d.items():
405        adata = table.get(k)
406        keypath = _keyPath(path, k)
407        if not adata:
408            _addMessage(f"WARNING: Attribute '{keypath}' is unknown.")
409            continue
410        if not validateAttribValue(k, v, adata, sdk_v, path):
411            continue
412        # print(k, v, type(v))
413        if isinstance(v, list) and (ltable := adata.get('l')):
414            _validateDefinitionArray(v, ltable, sdk_v, keypath)
415    # iterate over table entries to check if all required attribs are present
416    for k, data in table.items():
417        if data.get('r') and k not in d.keys():
418            _addMessage(f"WARNING: Missing required attribute '{_keyPath(path, k)}'.")
419
420def _validateDefinitionArray(a:list, table:dict, sdk_v:int, path:str=""):
421    i = 0
422    for item in a:
423        if isinstance(item, dict):
424            _validateDefinitionDict(item, table, sdk_v, f"{path}[{i:d}]")
425        else:
426            _addMessage(f"WARNING: Unable to handle array member '{item}' in '{path}'.")
427        i += 1
428
429
430def validateDefinitionObject(data:dict):
431    """
432    Validates a TP plugin definition structure from a Python `dict` object.
433    `data` is a de-serialized entry.tp JSON object (eg. `json.load('entry.tp')`)
434    Returns `True` if no problems were found, `False` otherwise.
435    Use `getMessages()` to check for any validation warnings which may be generated.
436    """
437    _clearSeenIds()
438    clearMessages()
439    sdk_v = data.get('sdk', TPSDK_DEFAULT_VERSION)
440    _validateDefinitionDict(data, TPSDK_ATTRIBS_ROOT, sdk_v)
441    return len(g_messages) == 0
442
443def validateDefinitionString(data: dict):
444    """
445    Validates a TP plugin definition structure from JSON string.
446    `data` is an entry.tp JSON string
447    Returns `True` if no problems were found, `False` otherwise.
448    Use `getMessages()` to check for any validation warnings which may be generated.
449    """
450    return validateDefinitionObject(data)
451
452def validateDefinitionFile(file:Union[str, TextIO]):
453    """
454    Validates a TP plugin definition structure from JSON file.
455    `file` is a valid system path to an entry.tp JSON file *or* an already-opened file handle (eg. sys.stdin).
456    Returns `True` if no problems were found, `False` otherwise.
457    Use `getMessages()` to check for any validation warnings which may be generated.
458    """
459    fh = file
460    if isinstance(fh, str):
461        fh = open(file, 'r')
462    ret = validateDefinitionObject(json.load(fh))
463    if fh != file:
464        fh.close()
465    return ret
466
467
468## CLI handlers
469
470def _generateDefinition(script, output_path, indent, skip_invalid:bool=False):
471    input_name = "input stream"
472    if isinstance(script, str):
473        if len(script.split(".")) < 2:
474            script = script + ".py"
475        input_name = script
476    indent = None if indent is None or int(indent) < 0 else indent
477
478    _printToErr(f"Generating plugin definition JSON from '{input_name}'...\n")
479    entry = generateDefinitionFromScript(script, skip_invalid)
480    entry_str = json.dumps(entry, indent=indent) + "\n"
481    valid = True
482    if (messages := getMessages()):
483        valid = False
484        _printMessages(messages)
485        _printToErr("")
486    # output
487    if output_path:
488        # write it to a file
489        with open(output_path, "w") as entry_file:
490            entry_file.write(entry_str)
491        _printToErr(f"Saved generated JSON to '{output_path}'\n")
492    else:
493        # send to stdout
494        print(entry_str)
495    _printToErr(f"Finished generating plugin definition JSON from '{input_name}'.\n")
496    return entry_str, valid
497
498
499def _validateDefinition(entry, as_str=False):
500    name = entry if isinstance(entry, str) and not as_str else "input stream"
501    _printToErr(f"Validating '{name}', any errors or warnings will be printed below...\n")
502    if as_str:
503        res = validateDefinitionString(entry)
504    else:
505        res = validateDefinitionFile(entry)
506    if res:
507        _printToErr("No problems found!")
508    else:
509        _printMessages(getMessages())
510    _printToErr(f"\nFinished validating '{name}'.\n")
511    return res
512
513def generatePythonStruct(entry, name):
514    _printToErr("Generating Python struct from entry json...\n")
515    try:
516        tp_to_py = TpToPy(entry)
517        tp_to_py.writetoFile(name)
518        _printToErr(f"Saved generated Python struct to '{name}'\n")
519        return True
520    except Exception as e:
521        _printToErr(f"Error: {e}")
522        return False
523
524def main(sdk_args=None):
525    from argparse import ArgumentParser
526
527    parser = ArgumentParser(epilog="This script exits with status code -1 (error) if generation or validation produces warning messages about malformed data. "
528                                   "All progress and warning messages are printed to stderr stream.")
529    parser.add_argument("-g", "--generate", action='store_true',
530                        help="Generate a definition file from plugin script data. This is the default action.")
531    parser.add_argument("-v", "--validate", action='store_true',
532                        help="Validate a definition JSON file (entry.tp). If given with `generate` then will validate the generated JSON output.")
533    parser.add_argument("target", metavar="target", nargs="?", default="",
534                        help="Either a plugin script for `generate` or an entry.tp file for `validate`. Paths are relative to current working directory. "
535                             "Defaults to './TPPEntry.py' and './entry.tp' respectively. Use 'stdin' (or '-') to read from input stream instead. "
536                             "Another usage is if you pass in a normal entry.tp without `validate` argument, It can generate "
537                             "Python version of entry.tp struct. It will be saved in `TPPEntry.py` if `output` is not given.")
538    gen_grp = parser.add_argument_group("Generator arguments")
539    gen_grp.add_argument("-o", metavar="<file>",
540                         help="Output file for `generate` action. Default will be a file named 'entry.tp' in the same folder as the input script. "
541                           "Paths are relative to current working directory. Use 'stdout' (or '-') to print the output to the console/stream instead.")
542    gen_grp.add_argument("-s", "--skip-invalid", action='store_true', dest="skip_invalid", default=False,
543                         help="Skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.")
544    gen_grp.add_argument("-i", "--indent", metavar="<n>", type=int, default=2,
545                         help="Indent level (spaces) for generated JSON. Use 0 for only newlines, or -1 for the most compact representation. Default is %(default)s spaces.")
546    gen_grp.add_argument("--noconfirm", action='store_true', default=False,
547                         help="When generating python struct from entry.tp, you can pass this arg to bypass confirm if you want to contiune if any error is given for vaildating entry.tp before generate python struct")
548    opts = parser.parse_args(sdk_args)
549    del parser
550
551    t = _normPath(opts.target or "TPPEntry.py")
552    if opts.target.endswith(".tp") and not opts.validate:
553        valid = _validateDefinition(t)
554        # Incase if file is invaild they will have choice to either contiune. but --noconfirm can override this.
555        # so that way if they use on github action it can still contiune if they wish.
556        if not valid and not opts.noconfirm: input("Found errors. Press Enter to build or Ctrl+C to exit...")
557        successful = generatePythonStruct(t, opts.o or "TPPEntry.py")
558        if not successful: _printToErr("Failed to generate exiting...")
559
560        return successful
561
562    # default action
563    opts.generate = opts.generate or not opts.validate
564
565    _printToErr("")
566
567    if opts.target in ("-","stdin"):
568        opts.target = sys.stdin
569
570    valid = True
571    entry_str = ""
572    if opts.generate:
573        opts.target = _normPath(opts.target or "TPPEntry.py")
574        output_path = None
575        if opts.o:
576            if opts.o not in ("-","stdout"):
577                output_path = opts.o
578        else:
579            out_dir = os.getcwd() if hasattr(opts.target, "read") else os.path.dirname(opts.target)
580            output_path = os.path.join(out_dir, "entry.tp")
581        sys.path.append(os.path.dirname(os.path.realpath(opts.target)))
582        entry_str, valid = _generateDefinition(opts.target, output_path, opts.indent, opts.skip_invalid)
583        if opts.validate and output_path:
584            opts.target = output_path
585
586    if opts.validate:
587        if entry_str:
588            valid = _validateDefinition(entry_str, True)
589        elif opts.target.endswith(".py"): # checks if is python file if It is then It will vaildate the python file by converting it to json first
590            valid = _validateDefinition(generateDefinitionFromScript(opts.target), as_str=True) # little hacky lol
591        else:
592            opts.target = _normPath(opts.target or "entry.tp")
593            valid = _validateDefinition(opts.target)
594
595    return 0 if valid else -1
596
597
598if __name__ == "__main__":
599    sys.exit(main())
def getMessages()
128def getMessages():
129    """ Gets a list of messages which may have been produced during generation/validation.
130    """
131    return g_messages

Gets a list of messages which may have been produced during generation/validation.

def clearMessages()
133def clearMessages():
134    """ Clears the list of validation warning messages.
135    Do this before invoking `validateAttribValue()` directly (outside of the other functions provided here).
136    """
137    global g_messages
138    g_messages.clear()

Clears the list of validation warning messages. Do this before invoking validateAttribValue() directly (outside of the other functions provided here).

def generateDefinitionFromScript(script: Union[str, TextIO], skip_invalid: bool = False)
239def generateDefinitionFromScript(script:Union[str, TextIO], skip_invalid:bool=False):
240    """
241    Returns an "entry.tp" Python `dict` which is suitable for direct conversion to JSON format.
242
243    `script` should be a valid python script, either a file path (ending in .py), string, or open file handle (like stdin).
244    The script should contain "SDK declaration variables" like	`TP_PLUGIN_INFO`, `TP_PLUGIN_SETTINGS`, etc.
245
246    Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output).
247    Default behavior is to only warn about them.
248
249    Note that the script is interpreted (executed), so any actual "business" logic (like connecting to TP) should be in "__main__".
250    Also note that when using input from a file handle or string, the script's "__file__" attribute is set to the current working
251    directory and the file name "tp_plugin.py".
252
253    May raise an `ImportError` if the plugin script could not be loaded or is missing required variables.
254    Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation).
255    """
256    script_str = ""
257    if hasattr(script, "read"):
258        script_str = script.read()
259    elif not script.endswith(".py"):
260        script_str = script
261
262    try:
263        if script_str:
264            # load from a string
265            spec = importlib.util.spec_from_loader("plugin", loader=None)
266            plugin = importlib.util.module_from_spec(spec)
267            setattr(plugin, "__file__", os.path.join(os.getcwd(), "tp_plugin.py"))
268            exec(script_str, plugin.__dict__)
269        else:
270            # load directly from a file path
271            spec = importlib.util.spec_from_file_location("plugin", script)
272            plugin = importlib.util.module_from_spec(spec)
273            spec.loader.exec_module(plugin)
274        # print(plugin.TP_PLUGIN_INFO)
275    except Exception as e:
276        input_name = "input stream" if script_str else script
277        raise ImportError(f"ERROR while trying to import plugin code from '{input_name}': {repr(e)}")
278    return generateDefinitionFromModule(plugin, skip_invalid)

Returns an "entry.tp" Python dict which is suitable for direct conversion to JSON format.

script should be a valid python script, either a file path (ending in .py), string, or open file handle (like stdin). The script should contain "SDK declaration variables" like TP_PLUGIN_INFO, TP_PLUGIN_SETTINGS, etc.

Setting skip_invalid to True will skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.

Note that the script is interpreted (executed), so any actual "business" logic (like connecting to TP) should be in "__main__". Also note that when using input from a file handle or string, the script's "__file__" attribute is set to the current working directory and the file name "tp_plugin.py".

May raise an ImportError if the plugin script could not be loaded or is missing required variables. Use getMessages() to check for any warnings/etc which may be generated (eg. from attribute validation).

def generateDefinitionFromModule(plugin: module, skip_invalid: bool = False)
281def generateDefinitionFromModule(plugin:ModuleType, skip_invalid:bool=False):
282    """
283    Returns an "entry.tp" Python `dict`, which is suitable for direct conversion to JSON format.
284    `plugin` should be a loaded Python "module" which contains "SDK declaration variables" like
285    `TP_PLUGIN_INFO`, `TP_PLUGIN_SETTINGS`, etc. From within a plugin script this could be called like:
286    `generateDefinitionFromModule(sys.modules[__name__])`.
287
288    Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output).
289    Default behavior is to only warn about them.
290
291    May raise an `ImportError` if the plugin script is missing required variables `TP_PLUGIN_INFO` and `TP_PLUGIN_CATEGORIES`.
292    Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation).
293    """
294    # Load the "standard SDK declaration variables" from plugin script into local scope
295    # INFO and CATEGORY are required, rest are optional.
296    if not (info := getattr(plugin, "TP_PLUGIN_INFO", {})):
297        raise ImportError(f"ERROR: Could not import required TP_PLUGIN_INFO variable from plugin source.")
298    if not (cats := getattr(plugin, "TP_PLUGIN_CATEGORIES", {})):
299        raise ImportError(f"ERROR: Could not import required TP_PLUGIN_CATEGORIES variable from plugin source.")
300    return generateDefinitionFromDeclaration(
301        info, cats,
302        settings = getattr(plugin, "TP_PLUGIN_SETTINGS", {}),
303        actions = getattr(plugin, "TP_PLUGIN_ACTIONS", {}),
304        states = getattr(plugin, "TP_PLUGIN_STATES", {}),
305        events = getattr(plugin, "TP_PLUGIN_EVENTS", {}),
306        connectors = getattr(plugin, "TP_PLUGIN_CONNECTORS", {}),
307        skip_invalid = skip_invalid
308    )

Returns an "entry.tp" Python dict, which is suitable for direct conversion to JSON format. plugin should be a loaded Python "module" which contains "SDK declaration variables" like TP_PLUGIN_INFO, TP_PLUGIN_SETTINGS, etc. From within a plugin script this could be called like: generateDefinitionFromModule(sys.modules[__name__]).

Setting skip_invalid to True will skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.

May raise an ImportError if the plugin script is missing required variables TP_PLUGIN_INFO and TP_PLUGIN_CATEGORIES. Use getMessages() to check for any warnings/etc which may be generated (eg. from attribute validation).

def generateDefinitionFromDeclaration(info: dict, categories: dict, skip_invalid: bool = False, **kwargs)
311def generateDefinitionFromDeclaration(info:dict, categories:dict, skip_invalid:bool=False, **kwargs):
312    """
313    Returns an "entry.tp" Python `dict` which is suitable for direct conversion to JSON format.
314    Arguments should contain SDK declaration dict values, for example as specified for `TP_PLUGIN_INFO`, etc.
315
316    The `info` and `category` values are required, the rest are optional.
317
318    Setting `skip_invalid` to `True` will skip attributes with invalid values (they will not be included in generated output).
319    Default behavior is to warn about them but still include them in the output.
320
321    `**kwargs` can be one or more of:
322    - settings:dict = {},
323    - actions:dict = {},
324    - states:dict = {},
325    - events:dict = {},
326    - connectors:dict = {}
327
328    Use `getMessages()` to check for any warnings/etc which may be generated (eg. from attribute validation).
329    """
330    _clearSeenIds()
331    clearMessages()
332    settings = kwargs.get('settings', {})
333    actions = kwargs.get('actions', {})
334    states = kwargs.get('states', {})
335    events = kwargs.get('events', {})
336    connectors = kwargs.get('connectors', {})
337    # print(info, categories, settings, actions, states, events, connectors)
338
339    # Start the root entry.tp object using basic plugin metadata
340    # This will also create an empty `categories` array in the root of the entry.
341    entry = _dictFromItem(info, TPSDK_ATTRIBS_ROOT, TPSDK_DEFAULT_VERSION, "info")
342
343    # Get the target SDK version (was either specified in plugin or is TPSDK_DEFAULT_VERSION)
344    tgt_sdk_v = entry['sdk']
345
346    # Loop over each plugin category and set up actions, states, events, and connectors.
347    for cat, data in categories.items():
348        path = f"category[{cat}]"
349        category = _dictFromItem(data, TPSDK_ATTRIBS_CATEGORY, tgt_sdk_v, path, skip_invalid)
350        category['actions'] = _arrayFromDict(actions, TPSDK_ATTRIBS_ACTION, tgt_sdk_v, cat, "actions", skip_invalid)
351        category['states'] = _arrayFromDict(states, TPSDK_ATTRIBS_STATE, tgt_sdk_v, cat, "states", skip_invalid)
352        category['events'] = _arrayFromDict(events, TPSDK_ATTRIBS_EVENT, tgt_sdk_v, cat, "events", skip_invalid)
353        if tgt_sdk_v >= 4:
354            category['connectors'] = _arrayFromDict(connectors, TPSDK_ATTRIBS_CONNECTOR, tgt_sdk_v, cat, "connectors", skip_invalid)
355        # add the category to entry's categories array
356        entry['categories'].append(category)
357
358    # Add Settings to root
359    if tgt_sdk_v >= 3:
360        entry['settings'].extend(_arrayFromDict(settings, TPSDK_ATTRIBS_SETTINGS, tgt_sdk_v, path = "settings", skip_invalid = skip_invalid))
361
362    return entry

Returns an "entry.tp" Python dict which is suitable for direct conversion to JSON format. Arguments should contain SDK declaration dict values, for example as specified for TP_PLUGIN_INFO, etc.

The info and category values are required, the rest are optional.

Setting skip_invalid to True will skip attributes with invalid values (they will not be included in generated output). Default behavior is to warn about them but still include them in the output.

**kwargs can be one or more of:

  • settings:dict = {},
  • actions:dict = {},
  • states:dict = {},
  • events:dict = {},
  • connectors:dict = {}

Use getMessages() to check for any warnings/etc which may be generated (eg. from attribute validation).

def validateAttribValue(key: str, value, attrib_data: dict, sdk_v: int, path: str = '')
367def validateAttribValue(key:str, value, attrib_data:dict, sdk_v:int, path:str=""):
368    """
369    Validates one attribute's value based on provided lookup table and target SDK version.
370    Returns `False` if any validation fails or `value` is `None`, `True` otherwise.
371    Error description message(s) can be retrieved with `getMessages()` and cleared with `clearMessages()`.
372
373    Args:
374        `key` is the attribute name;
375        `value` is what to validate;
376        `attrib_data` is the lookup table data for the given key (eg. `TPSDK_ATTRIBS_INFO[key]` );
377        `sdk_v` is the TP SDK version being used (for validation).
378        `path` is just extra information to print before the key name in warning messages (to show where attribute is in the tree).
379    """
380    global g_seen_ids
381    keypath = _keyPath(path, key)
382    if value is None:
383        if attrib_data.get('r'):
384            _addMessage(f"WARNING: Missing required attribute '{keypath}'.")
385        return False
386    if not isinstance(value, (exp_typ := attrib_data.get('t', str))):
387        _addMessage(f"WARNING: Wrong data type for attribute '{keypath}'. Expected {exp_typ} but got {type(value)}")
388        return False
389    if sdk_v < (min_sdk := attrib_data.get('v', sdk_v)):
390        _addMessage(f"WARNING: Wrong SDK version for attribute '{keypath}'. Minimum is v{min_sdk} but using v{sdk_v}")
391        return False
392    if (choices := attrib_data.get('c')) and value not in choices:
393        _addMessage(f"WARNING: Value error for attribute '{keypath}'. Got '{value}' but expected one of {choices}")
394        return False
395    if key == "id":
396        if not value in _seenIds():
397            _addSeenId(value, keypath)
398        else:
399            _addMessage(f"WARNING: The ID '{value}' in '{keypath}' is not unique. It was previously seen in '{g_seen_ids.get(value)}'")
400            return False
401    return True

Validates one attribute's value based on provided lookup table and target SDK version. Returns False if any validation fails or value is None, True otherwise. Error description message(s) can be retrieved with getMessages() and cleared with clearMessages().

Args
  • key is the attribute name;
  • value is what to validate;
  • attrib_data is the lookup table data for the given key (eg. TPSDK_ATTRIBS_INFO[key] );
  • sdk_v is the TP SDK version being used (for validation).
  • path is just extra information to print before the key name in warning messages (to show where attribute is in the tree).
def validateDefinitionObject(data: dict)
431def validateDefinitionObject(data:dict):
432    """
433    Validates a TP plugin definition structure from a Python `dict` object.
434    `data` is a de-serialized entry.tp JSON object (eg. `json.load('entry.tp')`)
435    Returns `True` if no problems were found, `False` otherwise.
436    Use `getMessages()` to check for any validation warnings which may be generated.
437    """
438    _clearSeenIds()
439    clearMessages()
440    sdk_v = data.get('sdk', TPSDK_DEFAULT_VERSION)
441    _validateDefinitionDict(data, TPSDK_ATTRIBS_ROOT, sdk_v)
442    return len(g_messages) == 0

Validates a TP plugin definition structure from a Python dict object. data is a de-serialized entry.tp JSON object (eg. json.load('entry.tp')) Returns True if no problems were found, False otherwise. Use getMessages() to check for any validation warnings which may be generated.

def validateDefinitionString(data: dict)
444def validateDefinitionString(data: dict):
445    """
446    Validates a TP plugin definition structure from JSON string.
447    `data` is an entry.tp JSON string
448    Returns `True` if no problems were found, `False` otherwise.
449    Use `getMessages()` to check for any validation warnings which may be generated.
450    """
451    return validateDefinitionObject(data)

Validates a TP plugin definition structure from JSON string. data is an entry.tp JSON string Returns True if no problems were found, False otherwise. Use getMessages() to check for any validation warnings which may be generated.

def validateDefinitionFile(file: Union[str, TextIO])
453def validateDefinitionFile(file:Union[str, TextIO]):
454    """
455    Validates a TP plugin definition structure from JSON file.
456    `file` is a valid system path to an entry.tp JSON file *or* an already-opened file handle (eg. sys.stdin).
457    Returns `True` if no problems were found, `False` otherwise.
458    Use `getMessages()` to check for any validation warnings which may be generated.
459    """
460    fh = file
461    if isinstance(fh, str):
462        fh = open(file, 'r')
463    ret = validateDefinitionObject(json.load(fh))
464    if fh != file:
465        fh.close()
466    return ret

Validates a TP plugin definition structure from JSON file. file is a valid system path to an entry.tp JSON file or an already-opened file handle (eg. sys.stdin). Returns True if no problems were found, False otherwise. Use getMessages() to check for any validation warnings which may be generated.

def generatePythonStruct(entry, name)
514def generatePythonStruct(entry, name):
515    _printToErr("Generating Python struct from entry json...\n")
516    try:
517        tp_to_py = TpToPy(entry)
518        tp_to_py.writetoFile(name)
519        _printToErr(f"Saved generated Python struct to '{name}'\n")
520        return True
521    except Exception as e:
522        _printToErr(f"Error: {e}")
523        return False
def main(sdk_args=None)
525def main(sdk_args=None):
526    from argparse import ArgumentParser
527
528    parser = ArgumentParser(epilog="This script exits with status code -1 (error) if generation or validation produces warning messages about malformed data. "
529                                   "All progress and warning messages are printed to stderr stream.")
530    parser.add_argument("-g", "--generate", action='store_true',
531                        help="Generate a definition file from plugin script data. This is the default action.")
532    parser.add_argument("-v", "--validate", action='store_true',
533                        help="Validate a definition JSON file (entry.tp). If given with `generate` then will validate the generated JSON output.")
534    parser.add_argument("target", metavar="target", nargs="?", default="",
535                        help="Either a plugin script for `generate` or an entry.tp file for `validate`. Paths are relative to current working directory. "
536                             "Defaults to './TPPEntry.py' and './entry.tp' respectively. Use 'stdin' (or '-') to read from input stream instead. "
537                             "Another usage is if you pass in a normal entry.tp without `validate` argument, It can generate "
538                             "Python version of entry.tp struct. It will be saved in `TPPEntry.py` if `output` is not given.")
539    gen_grp = parser.add_argument_group("Generator arguments")
540    gen_grp.add_argument("-o", metavar="<file>",
541                         help="Output file for `generate` action. Default will be a file named 'entry.tp' in the same folder as the input script. "
542                           "Paths are relative to current working directory. Use 'stdout' (or '-') to print the output to the console/stream instead.")
543    gen_grp.add_argument("-s", "--skip-invalid", action='store_true', dest="skip_invalid", default=False,
544                         help="Skip attributes with invalid values (they will not be included in generated output). Default behavior is to only warn about them.")
545    gen_grp.add_argument("-i", "--indent", metavar="<n>", type=int, default=2,
546                         help="Indent level (spaces) for generated JSON. Use 0 for only newlines, or -1 for the most compact representation. Default is %(default)s spaces.")
547    gen_grp.add_argument("--noconfirm", action='store_true', default=False,
548                         help="When generating python struct from entry.tp, you can pass this arg to bypass confirm if you want to contiune if any error is given for vaildating entry.tp before generate python struct")
549    opts = parser.parse_args(sdk_args)
550    del parser
551
552    t = _normPath(opts.target or "TPPEntry.py")
553    if opts.target.endswith(".tp") and not opts.validate:
554        valid = _validateDefinition(t)
555        # Incase if file is invaild they will have choice to either contiune. but --noconfirm can override this.
556        # so that way if they use on github action it can still contiune if they wish.
557        if not valid and not opts.noconfirm: input("Found errors. Press Enter to build or Ctrl+C to exit...")
558        successful = generatePythonStruct(t, opts.o or "TPPEntry.py")
559        if not successful: _printToErr("Failed to generate exiting...")
560
561        return successful
562
563    # default action
564    opts.generate = opts.generate or not opts.validate
565
566    _printToErr("")
567
568    if opts.target in ("-","stdin"):
569        opts.target = sys.stdin
570
571    valid = True
572    entry_str = ""
573    if opts.generate:
574        opts.target = _normPath(opts.target or "TPPEntry.py")
575        output_path = None
576        if opts.o:
577            if opts.o not in ("-","stdout"):
578                output_path = opts.o
579        else:
580            out_dir = os.getcwd() if hasattr(opts.target, "read") else os.path.dirname(opts.target)
581            output_path = os.path.join(out_dir, "entry.tp")
582        sys.path.append(os.path.dirname(os.path.realpath(opts.target)))
583        entry_str, valid = _generateDefinition(opts.target, output_path, opts.indent, opts.skip_invalid)
584        if opts.validate and output_path:
585            opts.target = output_path
586
587    if opts.validate:
588        if entry_str:
589            valid = _validateDefinition(entry_str, True)
590        elif opts.target.endswith(".py"): # checks if is python file if It is then It will vaildate the python file by converting it to json first
591            valid = _validateDefinition(generateDefinitionFromScript(opts.target), as_str=True) # little hacky lol
592        else:
593            opts.target = _normPath(opts.target or "entry.tp")
594            valid = _validateDefinition(opts.target)
595
596    return 0 if valid else -1